Member-only story
How to Make Sense of Distributed Processing With Python Daemons
Daemonize and scale your Python apps
The term “daemon” was coined by the programmers of MIT’s Project MAC. It comes from Maxwell’s demon — an imaginary being from a thought experiment, who constantly works in the background, sorting molecules. This lead to Unix systems using the term.
In Unix speak, a Daemon is a long-running background process that can perform virtually anything, from executing requests for services to performing any, usually long-running, arbitrary tasks for day-to-day activities on UNIX systems. As opposed to conventional applications, daemons do not run under the direct control of an interactive user. Since daemons do not have a controlling terminal, they run in the background silently. The term daemon originated in Unix and most operating systems use daemons in one form or another. In Unix, the names of daemons conventionally end in “d”. Some examples include inetd, httpd, nfsd, sshd, named, and lpd.
Reasons to Daemonizing Your Python Code
- To run one-off background async tasks without using bloated libraries like
celery, or asynchronous frameworks. - To have long-lived code execution that’s not part of your main program
- To introduce detached distributed computation without the need for subprocesses that depend on your main application life cycle.
Daemon Characteristics
Let's get definitions out of the way first, so that we better understand how the daemon process creation works. Every process within a UNIX system or a windows system has a PID (Process Identification Number — windows has an equivalent). PID is automatically assigned to each process when it’s created. Likewise, whenever we execute any Python code in a Unix system, it creates a special environment for that program with its associated PID. Let's see some examples below:
$ ps -aux | less$ top
