E-time | the software company
Apache Airflow version 3.0 represents a significant evolutionary leap compared to previous releases.
This update is designed to meet the growing need to orchestrate complex workflows and establishes itself as a benchmark for the development of enterprise data-driven solutions. Key updates introduced include:
1.Service-Oriented Architecture in Airflow 3.0 and Task Execution API
One of the most significant changes concerns the internal structure of the system: Airflow 3.0 evolves towards a service-oriented architecture supported by the new Task Execution Interface. Thanks to a dedicated API Server, DAG parsing logic is separated from task execution, potentially improving flexibility and security.
This design allows tasks to be executed in multi-cloud and hybrid environments, providing SDKs for tasks in different languages, enabling the definition and execution of pipelines while reducing some infrastructure constraints.
2.Edge Executor in Airflow: distributed orchestration and task execution
To extend orchestration beyond central data centers, Airflow 3.0 introduces the Edge Executor (AIP-69), available through a provider package.
Based on the Task Execution Interface, this executor allows pipelines to run on remote or “edge” devices, enabling distributed management of workflows across geographical locations. Integration with the API Server ensures efficient handling of remote tasks within the main DAGs.
3.DAG Versioning: workflow management in Airflow 3.0
The new architecture lays the foundation for associating DAG runs with a more stable representation of the code and task structure at the time of execution, reducing the impact of changes applied during runtime.
Although it is not yet a fully native versioning system, these improvements help increase pipeline stability and lay the groundwork for more controlled workflow management over time.
4.Revamped User Interface in Airflow 3.0
The Airflow 3.0 user interface (UI) has been completely redesigned using React and relies on new REST APIs built with FastAPI, providing a faster and more intuitive user experience.
The new interface improves the seamless integration of asset-based and task-based workflows, removes navigation constraints, and offers advanced tools such as Grid View and Graph View, making the platform more responsive and accessible.
5. Migrating to Airflow: Compatibility and Tools
To facilitate the upgrade, despite the architectural changes, Airflow 3.0 maintains backward compatibility for existing DAGs thanks to the Python Task SDK.
The transition can be carried out gradually: teams can start working on assets or test execution in Docker environments before moving to production
