Skip to main content

Python Celery

What's workflow in Celery?

In Celery, workflow is composed of multiple tasks, and a task is deemed to be a universal unit of the workflow, as a function in the program. In Celery, it's recommended to divide a long-running task into multiple short-running tasks. workflow comes out to help ease the orchestrations of the work, such as chain() three tasks.

A demo workflow

Construct a workflow​

Avoid running synchronous subtasks within a task​

Asynchronous tasks with a task​

@app.task(bind=True)
def update_page_info(self, url):
    # fetch_page -> parse_page -> store_page
    chain = fetch_page.s(url) | parse_page.s() | store_page_info.s(url)
    # chain()
    self.replace(chain)

@app.task()
def fetch_page(url):
    return myhttplib.get(url)

@app.task()
def parse_page(page):
    return myparser.parse_document(page)

@app.task(ignore_result=True)
def store_page_info(info, url):
    PageInfo.objects.create(url=url, info=info)

Monitor the workflow​

Resources​

Designing Dynamic Workflows with Celery and Python | by Marin Aglić | Data Engineer Things

The Curious Case of Celery Work-flows

Celery ETA Tasks Demystified. At Instawork, we use Celery to queue… | by Oleg Pesok | Instawork Engineering

Canvas: Designing Work-flows — Celery 5.3.6 documentation