Simple Example for Multithreading in Python

Had a need today to create a multithreaded for loop that calls a specific function – trying to speed up a program I wrote as it’s way too slow in a single threaded context but not spin off so many threads that it crashes the computer.

Not enough simple examples for this online – when I did this in C#, I could always find a result within 10 min on Google.

Here’s this for anyone in the future – picture and code below:

from concurrent.futures import ThreadPoolExecutor

#If wanting to not have python take the computer over and slow it down to a crawl
#executor = ThreadPoolExecutor(max_workers=2)
    
#If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5, assuming that ThreadPoolExecutor is often used to overlap I/O instead of CPU work and the number of workers should be higher than the number of workers for ProcessPoolExecutor.
executor = ThreadPoolExecutor()

futures = []

path = './test_multi.log'

def testFunction(param1):
    import time
    time.sleep(1)
    with open(path, 'a') as the_file:
        the_file.write("hithere" + param1 + '\n')

items = []

for i in range(1000):
    items.append(str("test " + str(i)))

for item in items:
    a = executor.submit(testFunction, item)
    futures.append(a)

#Wait for all the jobs to finish before moving on
executor.shutdown(wait=True)

#Loop futures to see any failures - only hits exception block if failure detected
for future in futures:
    try:
        future.result()
    except Exception as e:
        print("Detected failed task in threadpool - " + str(e))
        import traceback
        print(traceback.format_exc())