3 Replies - 848 Views - Last Post: 14 June 2016 - 05:34 PM Rate Topic: -----

#1 gambrjl   User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 2
  • Joined: 13-June 16

Multiprocessing/Threading (run same python script in parallel)

Posted 13 June 2016 - 07:48 AM

I have a simple python script that grabs some data and does some processing based on an argument sysarg

c:\> create.py argument



What I'd like to do is write a new python script that will kick off 5 or so of the create.py all with different arguments based on a list of arguments. When one of the scripts ends, kick off the next one. Say the argument list is 100 or so

I can kick of 5 of them manually with different arguments and it works fine so just looking to automate the process

I've been reading about threading and here have to say I'm a little confused. I'm probably making it harder than it has to be because threading seems to talk about passing data between the threads which I don't need to do. I just need to kick them off and monitor when they finish and start the next argument

Anyone have a good example that they could point me to?

Is This A Good Question/Topic? 0
  • +

Replies To: Multiprocessing/Threading (run same python script in parallel)

#2 CurlyJoe   User is offline

  • D.I.C Head

Reputation: 22
  • View blog
  • Posts: 97
  • Joined: 19-September 13

Re: Multiprocessing/Threading (run same python script in parallel)

Posted 13 June 2016 - 09:58 AM

Multiprocessing will use multiple cores if you have more than one. I like Doug Hellman's PMOTW write up on multiprocessing https://pymotw.com/2...ing/basics.html The first example kicks off 5 separate processes. For your use, you would periodically check each of the processes in "jobs" from the example to see which have finished, and start a new process.
Was This Post Helpful? 0
  • +
  • -

#3 gambrjl   User is offline

  • New D.I.C Head

Reputation: 0
  • View blog
  • Posts: 2
  • Joined: 13-June 16

Re: Multiprocessing/Threading (run same python script in parallel)

Posted 14 June 2016 - 02:40 PM

Thanks I appreciate it. Ended up using multiprocessing.pool and got what i wanted. If anyone is interested

Opens 3 windows, when one finishes it kicks off the next

import multiprocessing
import os
import subprocess

arguments=['argument1','argument2','argument3','argument4','argument5','argument6']

def worker(args):
	print('Starting %s' % args)
	d=subprocess.Popen(["c:\python34\python.exe", "pyscript.py",args], creationflags=subprocess.CREATE_NEW_CONSOLE)  #Open new cmd window
	d.wait()
	print('Exiting %s' % args)
	
def handler():
	p=multiprocessing.Pool(3)
	p.map(worker, arguments)	

if __name__ == '__main__':
	handler()



Was This Post Helpful? 0
  • +
  • -

#4 CurlyJoe   User is offline

  • D.I.C Head

Reputation: 22
  • View blog
  • Posts: 97
  • Joined: 19-September 13

Re: Multiprocessing/Threading (run same python script in parallel)

Posted 14 June 2016 - 05:34 PM

Showing what works helps any others who get here via a search.
Was This Post Helpful? 0
  • +
  • -

Page 1 of 1