Member-only story
A Beginner's Guide to Connect Celery With MongoDB
Making asynchronous tasks easy using Celery Mongo Database Scheduler
Most real-world applications are built to handle large numbers of users, complex data processing, and computations. All these things are done in the background without hampering the user experience. In this article, I will explain Celery MongoDB Scheduler working, Celery, and Redis connection with MongoDB in a Django project.
Celery comes with the power to handle these responsibilities
According to Wikipedia: Celery is an open-source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling.
Celery has two features:
- Worker: It has child processes that execute tasks.
- Beat: Sends tasks periodically to the broker.
Architecture
Let me explain this. Here we have our tasks stored in a collection in MongoDB, Celery Beat picks tasks periodically from MongoDB and pass to Redis. Redis has task queues where tasks are stored. Worker processes pick tasks from a task queue and execute tasks. The resulting data is once again sent back to MongoDB and this data is stored in a separate collection.
Now let me show you how easily this can be done.
Install All Dependencies
1. Celery
https://docs.celeryproject.org/en/stable/getting-started/introduction.html
pip install celery2. Celerybeat-mongo
To store tasks in MongoDB: To store tasks in MongoDB we are going to use celerybeat-mongo This a great project timely maintained in GitHub. https://github.com/zmap/celerybeat-mongo
pip install celerybeat-mongo3. Redis
Redis is a broker.
pip install redis
