Value of Airflow variable gets invalid on restarting docker container
I have installed airflow image on top of docker and thereafter started the container. Everything works fine and I can access the airflow UI from my local system.
I have a long list of dags and associated airflow variables at some other instance of airflow, copy of which is running in my local system. I have impoted all the variables.
Problem:
Whenever I restart the airflow container, all the variables that I have imported during the previous container run get invalid like this.
Please help me understand what am I doing wrong.
Source: Docker Questions
The Root Cause of this Problem is the AirFlow Encryption Mechanism for the Key-value Variables. when you import your variables manually, their is_encrypted attributes, are automatically set to True.
Whenever, you restart the container, new Encryption Key is generated, thus the old ones got Invalid.
You Have 3 Options :
Set the fernet_key explicitly in airflow.cfg
Set the AIRFLOW__CORE__FERNET_KEY Environment Variable in docker-compose.yml
Set the is_encrypted attributes to False(Admin UI, CLI, update sql query , …)
I personally chose the second one, so my docker-compose.yml file, looks like this :
0
The Root Cause of this Problem is the AirFlow Encryption Mechanism for the Key-value Variables. when you import your variables manually, their is_encrypted attributes, are automatically set to True.
Whenever, you restart the container, new Encryption Key is generated, thus the old ones got Invalid.
You Have 3 Options :
Set the fernet_key explicitly in airflow.cfg
Set the AIRFLOW__CORE__FERNET_KEY Environment Variable in docker-compose.yml
Set the is_encrypted attributes to False(Admin UI, CLI, update sql query , …)
I personally chose the second one, so my docker-compose.yml file, looks like this :
environment:
– LOAD_EX=n
– EXECUTOR=Local
– AIRFLOW__CORE__FERNET_KEY=’81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs=’
thanks to wittfabian