⚓ T300870 Airflow concurrency limits (2024)

Create Task

Maniphest T300870

Edit Task
Edit Related Tasks...
Edit Related Objects...
Mute Notifications
Protect as security issue
Award Token
Flag For Later

Assigned To

Antoine_Quhen

Authored By

	Antoine_Quhen
	Feb 3 2022, 2:23 PM2022-02-03 14:23:46 (UTC+0)

At environmment level (airflow.cfg):

parallelism: This is the maximum number of tasks that can run concurrently within a single Airflow environment. For example, if this setting is set to 32 then no more than 32 tasks can run at once across all DAGs. Think of this as "maximum active tasks anywhere." If you notice that tasks are stuck queued for extended periods of time, this is a value you may want to increase. By default, this is set to 32.

We may set it explicitly to 32.

max_active_runs_per_dag: This determines the maximum number of active DAG Runs (per DAG) that the Airflow Scheduler can create at any given time. In Airflow, a DAG Run represents an instantiation of a DAG in time, much like a task instance represents an instantiation of a task. This parameter is most relevant if Airflow has to catch up from missed DAG runs, also known as backfilling. Consider how you want to handle these scenarios when setting this parameter. By default, it's set to 16.

We should set it to 2. To avoid 1 dag to take the all the resources.

max_active_tasks_per_dag: (formerly dag_concurrency) This determines the maximum number of tasks that can be scheduled at once, per DAG." Use this setting to prevent any one DAG from taking up too many of the available slots from parallelism or your pools, which helps DAGs be good neighbors to one another. By default, this is set to 16.

We should set it to 2. To make sure 1 dag can't take all resources.

At DAG level (in dag definition file):

max_active_runs_per_dag & max_active_tasks_per_dag could be overriden by:

max_active_runs: This is the maximum number of active DAG Runs allowed for the DAG in question. Once this limit is hit, the Scheduler will not create new active DAG Runs. If this setting is not defined, the value of the environment-level setting max_active_runs_per_dag is assumed.
Coudld be set in each dag definition.
concurrency: This is the maximum number of task instances allowed to run concurrently across all active DAG runs for a given DAG. This allows you to set 1 DAG to be able to run 32 tasks at once, while another DAG might only be able to run 16 tasks at once. If this setting is not defined, the value of the environment-level setting max_active_tasks_per_dag is assumed.
Coudld be set in each dag definition.

A clear explanation of the doc is here: https://www.astronomer.io/guides/airflow-scaling-workers

Subject	Repo	Branch	Lines +/-
airflow: change max_active_runs_per_dag back to 1	operations/puppet	production	+1 -1
Set default Airflow concurrency limits for an- airflow instances	operations/puppet	production	+43 -0
Set default Airflow concurrency limits	operations/puppet	production	+36 -6

Customize query in gerrit

Mentions

Mentioned In: T351388: Add a spark global config for better file commit strategy
Mentioned Here: T347076: NEW BUG REPORT Some DAG run attempts fail because File *_temporary/0 does not exist.

Event Timeline

Antoine_Quhen created this task.Feb 3 2022, 2:23 PM2022-02-03 14:23:46 (UTC+0)

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2022, 2:23 PM2022-02-03 14:23:47 (UTC+0)

mforns added a comment.Feb 3 2022, 2:49 PM2022-02-03 14:49:38 (UTC+0)

Comment Actions

This is great @Antoine_Quhen!
I agree with your default value choices.
I think in Airflow2 the concurrency parameter is deprecated in favor of max_active_tasks, see: https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-to-improve-dag-performance