- Report this article
Ranganath S
Ranganath S
Senior Engineering Manager | Data Engineering | Data science
Published Jul 15, 2023
+ Follow
Airflow has been the go to tool for workflow and scheduling , There are various tools in Big Data space, like are prefect, Dagster, Airflow, Azkaban , Apache Dolphin scheduler etc. How ever Airflow and Dagster has been on top of adoption.
Limitations of Airflow
Recommended next reads
You wrote DAG , scheduled it to run lets say every other hour, starting from 10 AM today, every thing is pretty neat till now, you check back at 11 AM and you find that your DAG has triggered, however log indicates there was one run at 10 AM, dont be surprised, if i say this is expected behavior, because,Airflow will run DAG at the end of the scheduling_interval, which means 10 AM DAG gets trigged by 11AM.
So Airflow is a beast on its own, its flexible but its counter-intuitive for adoption and IMO in 2023, its not worth coding your DAGs or workflows when you have better alternatives like Apache Dolphin Scheduler , which offers DAG creationby simple DND, can scale easily without too much of tuning, multi tenancy is available out of box.
Help improve contributions
Mark contributions as unhelpful if you find them irrelevant or not valuable to the article. This feedback is private to you and won’t be shared publicly.
Contribution hidden for you
This feedback is never shared publicly, we’ll use it to show better contributions to everyone.
Like
Celebrate
Support
Love
Insightful
Funny
6
3 Comments
⚓ Shawn Fergus ⚓
World's Tallest Data Enthusiast @Shipyard | Shipyard's Your Secure Data Operations Platform for all Things ETL, Workflow Orchestration, Observability, and Beyond. | Let's Be Friends.
6mo
- Report this comment
👇🏻
No more previous content
No more next content
1Reaction
Kuldeep Pal
Data Engineer - III at Walmart | SDE III, Software Engineer (Backend) | Spark | Big Data | Python | SQL | AWS | GCP | Scala | Kafka | Datawarehouse | Streaming | Airflow 1x | Java Spring Boot
9mo
- Report this comment
Mage is on the way 😂
1Reaction
Santosh Suresh
TWS Administrator
9mo
- Report this comment
very useful information👍
1Reaction
See more comments
To view or add a comment, sign in
Sign in
Stay updated on your professional world
Sign in
By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.
New to LinkedIn? Join now
Insights from the community
- Data Management How can you optimize ETL for Google Cloud Dataflow?
- Data Analytics How can you use Apache Beam for data processing pipelines?
- Data Warehousing How can you use Apache Spark to debug and test ETL processing for Data Warehousing?
- Data Engineering How can you integrate data extraction with data transformation and loading processes?
- Business Intelligence How can you use Google Cloud Dataflow for ETL best practices in Business Intelligence?
- Database Engineering How do you test and debug serverless database functions and triggers?
- Database Development How can you customize ETL learning resources to your needs?
- Business Intelligence What are the best ways to handle missing data during ETL?
- Object-Relational Mapping (ORM) How do you test the scalability and reliability of your ORM-based applications?
- Data Engineering What are the most important testing and debugging practices for batch processing pipelines?
Others also viewed
- Extending your Puppet language dictionary for application orchestration Nicolas Corrarello 8y
- DevOps for Big Data - Autoscale Apache Spark with JupyterHub Ziang Jia 1y
- Testing Strategies for Apache Spark Applications: Ensuring Reliability at Scale InfoMover Technologies 3mo
- ELK Stack + Spring Boot Mukesh C. 2mo
- Software Automation: What I Do Pete Carapetyan 9y
- Apache Airflow Aakash Kumar Dhal 2mo
- Tapping into Docker's Power: Employing Apache Airflow for Seamless Workflow Integration Ankit Shrivastava 2mo
- Airflow: ETL Workflow Management Platform Gopal Kumar Roy 3y
- Apache Airflow Sourav R. 2y
- ETL encapsulation in aws-Lambda Function with Serverless, CloudFormation, APIGateway, Docker, FastAPI to PowerBI API Rafael Vera-Marañón 1mo
Explore topics
- Sales
- Marketing
- Business Administration
- HR Management
- Content Management
- Engineering
- Soft Skills
- See All