Parallel Python

A lecture from ASPP 2024. The slides and exercises are in slides/ and exercises/, respectively.

A really cool example of using asyncio with the dakos program is in extras/.

Lecture outline

1. Processes, threads and THE GIL
2. Hands-on investigations of embarrassingly parallel problems
   A. Multithreading with NumPy
   B. The multiprocessing package
   C. Blending processes and threads
3. Going further

Online resources

• Online textbook on operating systems, processes, threads, etc.
  • Operating Systems: Three Easy Pieces
  • Chapters 4, 13, and 26 are particularly relevant.
  • But the whole book is interesting.
• Report of a talk on Python concurrency types, with focus on sub-interpreters.
  • Python subinterpreters and free-threading
• Comparison of multithreading, multiprocessing, and asyncio.
  • Async Python: The Different Forms of Concurrency
  • Note the code snippets are a little outdated.
• Example of webscraping with asyncio
  • Asynchronous Web Scraping in Python - ZenRows
• Sub-interpreters in 3.12
  • Sub-interpreter web workers