Remember the days of wrestling with Python 2? For many of us in the data engineering world, that's a memory that feels both recent and a lifetime ago. Airflow, a tool many of us rely on daily, made a significant leap, requiring a move to Python 3. If you're still on Airflow 1.10, or even contemplating the upgrade path, it's worth chatting about how that transition unfolds, especially from a Python perspective.
Airflow 2.0 marked a clear turning point, ditching Python 2 support entirely. This meant that if your workflows had any lingering Python 2 dependencies, you'd need a strategy. The good news is that Airflow 2.0 itself was built with Python 3.6+ in mind, and subsequent versions have continued to embrace newer Python releases. For those stubborn tasks that absolutely needed Python 2, Airflow offered a lifeline through decorators like @task.virtualenv, @task.docker, or @task.kubernetes. It's a clever way to isolate those legacy bits without holding back the rest of your modern pipeline.
Before diving headfirst into Airflow 2.0, the team introduced a rather thoughtful "bridge release": Airflow 1.10.15. This wasn't just a minor patch; it was designed to smooth the transition. Think of it as a dress rehearsal. It backported many of the architectural and DAG changes from Airflow 2.0, meaning most DAGs written for the new version would actually run on 1.10.15. This gave everyone a crucial window to test their existing setups and start modifying DAGs without the pressure of an immediate, disruptive upgrade. Plus, the CLI commands got an update too, allowing you to get acquainted with the new syntax.
For those of us who live and breathe Kubernetes, 1.10.15 also brought back the pod_template_file capability for the KubernetesExecutor. This was a big deal for customizing pod configurations, and a handy script was even provided to generate a template based on your airflow.cfg settings. It’s these kinds of thoughtful touches that make a complex upgrade feel more manageable.
Once you've landed on 1.10.15 and are running Python 3, the next logical step is to run the upgrade check scripts. Seriously, these are your best friends. They meticulously scan your airflow.cfg and all your DAGs, spitting out a detailed report of exactly what needs attention before you make the final jump to Airflow 2.0. It’s like having a seasoned guide pointing out every potential pothole on the road ahead.
Then comes the part where you start embracing the new world: backport providers. This is where the Python import paths start to shift. Instead of importing an operator directly from airflow.operators, you'll find yourself importing from airflow.providers.<service>.operators. For instance, the DockerOperator moves from from airflow.operators.docker_operator import DockerOperator to from airflow.providers.docker.operators.docker import DockerOperator. You install these providers separately using pip install apache-airflow-backport-providers-<service>. It’s a modular approach that allows you to upgrade components independently, which is always a win.
Finally, as you move into Airflow 2.0 itself, you'll notice some subtle but important changes, like how Jinja templates handle undefined variables. Previously, they'd just render as empty strings, often silently. Now, Airflow 2.0 will throw an error, forcing you to address those undefined variables explicitly. It’s a small change that can save you a lot of headaches down the line by making your templating more robust.
Upgrading Airflow, especially across major versions, can feel like a significant undertaking. But by breaking it down, leveraging the bridge releases, and understanding the shifts in Python and provider management, it becomes a much more approachable, and dare I say, even rewarding, process. It’s about keeping your data pipelines modern, efficient, and reliable.
