Had a need today to create a multithreaded for loop that calls a specific function – trying to speed up a program I wrote as it’s way too slow in a single threaded context but not spin off so many threads that it crashes the computer.
Not enough simple examples for this online – when I did this in C#, I could always find a result within 10 min on Google.
Here’s this for anyone in the future – picture and code below:
from concurrent.futures import ThreadPoolExecutor
#If wanting to not have python take the computer over and slow it down to a crawl
#executor = ThreadPoolExecutor(max_workers=2)
#If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor.
executor = ThreadPoolExecutor()
futures = []
path = './test_multi.log'
def testFunction(param1):
import time
time.sleep(1)
with open(path, 'a') as the_file:
the_file.write("hithere" + param1 + '\n')
items = []
for i in range(1000):
items.append(str("test " + str(i)))
for item in items:
a = executor.submit(testFunction, item)
futures.append(a)
#Wait for all the jobs to finish before moving on
executor.shutdown(wait=True)
#Loop futures to see any failures - only hits exception block if failure detected
for future in futures:
try:
future.result()
except Exception as e:
print("Detected failed task in threadpool - " + str(e))
import traceback
print(traceback.format_exc())
Yes, more python!
LikeLike