Run Multiple Python File Concurrently

Run multiple python file concurrently

Make main() function in every python file and then import all the files in your main file. Then call all main functions.

from . import pop
from . import pop1
# and so on

# and now call all main functions
pop.main()
pop1.main()
# and so on

How to run multiple python files at the same time?

Assuming that your bot files execute something when you run them at the command line, we can load them and execute them by importing them into our python process (instead of the shell). As each python file defines a package we can do this as follows:

import bot_1, bot_2, bot_3

This however will run them one after another, and also prevent you from running the same one twice. To get them to run at once, we can use multiprocessing as you suggest:

import multiprocessing

for bot in ('bot_1', 'bot_2', 'bot_3'):
    p = multiprocessing.Process(target=lambda: __import__(bot))
    p.start()

Processes need a function to run, so we use an anonymous lambda to give it one, and then dynamically import the name.

It's not shown here, but as long as you don't import a module into the parent process, the children will be forced to load it, meaning you can run the same one over and over in the separate process if you need to.

Performance running multiple Python scripts simultaneously

how Python interacts with Windows

Python is an executable, a program. When a program is executed a new process is created.

python myscript.py starts a new python.exe process where the first argument is your script.

when multiple unrelated scripts are run simultaneously

They are multiple processes.

Is the interpreter able to draw resources from the OS (processor/memory/disk) independently?

Yes. Each process may access the OS API however it wishes, to the extend that it is possible.

What are the limitations?

Most likely RAM. The same limitations as any other process might encounter.

how to run multiple python scripts at the same time?

For Ubuntu, there is an answer here that may work. You can run multiple programs through bash. The "&" tells it to run the program in the background.

python program1.py &
python program2.py &

Since it sounds like you are using a remote server that has Ubuntu, I would recommend using tmux instead. It allows you to open up multiple sessions, run a program on each on and keep them running after you have closed your connection. It also allows you to enter back into each session, if you need to enter/read anything from your programs. I found this guide helpful when I had to do something similar a few months ago.

You can run batch files on Ubuntu as well. I'm not as familiar with running batch files on Ubuntu, but something like the following should work for you. You can also add while loops, if statements, etc.. Anything you would normally type into the shell can be put into the batch file to automatically run your programs or navigate directories.

#!/bin/bash
ECHO starting training program
# without the "&", it waits for your training program to finish running
python training_program.py
ECHO training program completed

# Adding the "&" tells the programs to run in the background
# You can also use tmux instead, if you want to navigate the different programs
python program1.py &
python program2.py &
ECHO training programs running in the background

The file should be saved with a ".sh" extension, then make the file executable by running the following through your shell.

chmod +x your_batch_file.sh 

If you are using Windows, you could create a batch file that runs all the programs. Here is an example of the file that you can create with your editor of choice:

# If you don't want to see the outputs/print of your training program, 
#     add @ECHO OFF to the start. If you want to see them, remove the @ECHO OFF
@ECHO OFF

# Without "start" before the script to run your training program,
#     the batch file will wait until the training program finishes
python "path\to\your\training_program.py"
ECHO training program completed

# Adding "start" opens it in a new window, and processes the next line
#     without waiting for the program to finish running
start python "path\to\your\program1.py"
ECHO Running program1
start python "path\to\your\program2.py"
ECHO Running program2

# Adding "PAUSE" makes the script wait for you manually type a key to continue,
#     but it is not required. You can add PAUSE anywhere in the script
PAUSE

"start" runs each program in a new window. After you have configured the text, safe the file with a ".bat" extension. Then all you have to do is click on the file to run the batch file, which will open each program in a separate window.

Similarly, you can just run the following from command prompt and it will open them in separate windows as well.

start python path\to\your\program1.py
start python path\to\your\program2.py

But it sounds like you are performing this more than once, in which case a batch file may be more suitable.

---

Related Topics