Had a need today to create a multithreaded for loop that calls a specific function – trying to speed up a program I wrote as it’s way too slow in a single threaded context but not spin off so many threads that it crashes the computer.
Not enough simple examples for this online – when I did this in C#, I could always find a result within 10 min on Google.
Here’s this for anyone in the future – picture and code below:

from concurrent.futures import ThreadPoolExecutor #If wanting to not have python take the computer over and slow it down to a crawl #executor = ThreadPoolExecutor(max_workers=2) #If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor. executor = ThreadPoolExecutor() futures = [] path = './test_multi.log' def testFunction(param1): import time time.sleep(1) with open(path, 'a') as the_file: the_file.write("hithere" + param1 + '\n') items = [] for i in range(1000): items.append(str("test " + str(i))) for item in items: a = executor.submit(testFunction, item) futures.append(a) #Wait for all the jobs to finish before moving on executor.shutdown(wait=True) #Loop futures to see any failures - only hits exception block if failure detected for future in futures: try: future.result() except Exception as e: print("Detected failed task in threadpool - " + str(e)) import traceback print(traceback.format_exc())
Yes, more python!
LikeLike