How To Make Parallel Async HTTP Requests in Python

Requests with Threads vs. aiohttp with Semaphore

Photo by Matt Duncan on Unsplash

By reading this piece, you will learn to make multiple asynchronous requests concurrently in Python. This tutorial covers two different methods:

• via requests package with Threads (a native thread for each request)
• via aiohttp client with Semaphore (to limit and pool the number of tasks)

Both implementations are inspired by the explanation in the following blog post. The code samples provided by the author is meant to be run on Jupyter Notebook which comes with its own event loop, eliminating the need to call asyncio.run. In this tutorial, the code will be cleaned and modified for it to work in Python file.

Besides that, we will implement our own server instead of calling an external API. As a result, you can easily scale up or down the number of concurrent requests at your own will.

Let’s proceed to the next section and start installing the necessary modules.

Setup

It is highly recommended to create a new virtual environment before you continue with the installation.

aiohttp