-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Task Instance primary key be a UUID #43243
base: main
Are you sure you want to change the base?
Conversation
While working on this apache#43243, I was following https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst and I ran into an error with pre-commit hook. When running the revision command as follows: ``` root@f1f78138ad78:/opt/airflow/airflow# alembic revision -m "New revision" Generating /opt/airflow/airflow/migrations/versions/01b38be821e9_new_revision.py ... done ``` It creates a file as follows: ```python """New revision Revision ID: cd7be1ae8b80 Revises: 05234396c6fc Create Date: 2024-10-22 01:44:17.873864 """ import sqlalchemy as sa from alembic import op # revision identifiers, used by Alembic. revision = 'cd7be1ae8b80' down_revision = '05234396c6fc' branch_labels = None depends_on = None ``` Notice single quotes in `revision` & `down_revision`. Now if I just run that single pre-commit hook (`update-migration-references`), it fails ``` ❯ pre-commit run "update-migration-references" --all-files Update migration ref doc.................................................Failed - hook id: update-migration-references - exit code: 1 Using 'uv' to install Airflow Using airflow version from current sources Updating migration reference for airflow Making sure airflow version updated Making sure there's no mismatching revision numbers Traceback (most recent call last): File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 246, in <module> correct_mismatching_revision_nums(revisions=revisions) File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 230, in correct_mismatching_revision_nums new_content = content.replace(revision_id_match.group(1), revision_match.group(1), 1) AttributeError: 'NoneType' object has no attribute 'group' Error 1 returned If you see strange stacktraces above, run `breeze ci-image build --python 3.9` and try again. ``` That isn't a problem generally as `ruff` will fail before and convert it into double quotes. But rather than doing that, we fix it at source.
A neat trick to avoid the need for pgcrypto extension: https://brandur.org/fragments/secure-bytes-without-pgcrypto Though Oh, Pg 12 is officially all we support anyway, so I think using the |
While working on this apache#43243, I was following https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst and I ran into an error with pre-commit hook. When running the revision command as follows: ``` root@f1f78138ad78:/opt/airflow/airflow# alembic revision -m "New revision" Generating /opt/airflow/airflow/migrations/versions/01b38be821e9_new_revision.py ... done ``` It creates a file as follows: ```python """New revision Revision ID: cd7be1ae8b80 Revises: 05234396c6fc Create Date: 2024-10-22 01:44:17.873864 """ import sqlalchemy as sa from alembic import op # revision identifiers, used by Alembic. revision = 'cd7be1ae8b80' down_revision = '05234396c6fc' branch_labels = None depends_on = None ``` Notice single quotes in `revision` & `down_revision`. Now if I just run that single pre-commit hook (`update-migration-references`), it fails ``` ❯ pre-commit run "update-migration-references" --all-files Update migration ref doc.................................................Failed - hook id: update-migration-references - exit code: 1 Using 'uv' to install Airflow Using airflow version from current sources Updating migration reference for airflow Making sure airflow version updated Making sure there's no mismatching revision numbers Traceback (most recent call last): File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 246, in <module> correct_mismatching_revision_nums(revisions=revisions) File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 230, in correct_mismatching_revision_nums new_content = content.replace(revision_id_match.group(1), revision_match.group(1), 1) AttributeError: 'NoneType' object has no attribute 'group' Error 1 returned If you see strange stacktraces above, run `breeze ci-image build --python 3.9` and try again. ``` That isn't a problem generally as `ruff` will fail before and convert it into double quotes. But rather than doing that, we fix it at source.
While working on this #43243, I was following https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst and I ran into an error with pre-commit hook. When running the revision command as follows: ``` root@f1f78138ad78:/opt/airflow/airflow# alembic revision -m "New revision" Generating /opt/airflow/airflow/migrations/versions/01b38be821e9_new_revision.py ... done ``` It creates a file as follows: ```python """New revision Revision ID: cd7be1ae8b80 Revises: 05234396c6fc Create Date: 2024-10-22 01:44:17.873864 """ import sqlalchemy as sa from alembic import op # revision identifiers, used by Alembic. revision = 'cd7be1ae8b80' down_revision = '05234396c6fc' branch_labels = None depends_on = None ``` Notice single quotes in `revision` & `down_revision`. Now if I just run that single pre-commit hook (`update-migration-references`), it fails ``` ❯ pre-commit run "update-migration-references" --all-files Update migration ref doc.................................................Failed - hook id: update-migration-references - exit code: 1 Using 'uv' to install Airflow Using airflow version from current sources Updating migration reference for airflow Making sure airflow version updated Making sure there's no mismatching revision numbers Traceback (most recent call last): File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 246, in <module> correct_mismatching_revision_nums(revisions=revisions) File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 230, in correct_mismatching_revision_nums new_content = content.replace(revision_id_match.group(1), revision_match.group(1), 1) AttributeError: 'NoneType' object has no attribute 'group' Error 1 returned If you see strange stacktraces above, run `breeze ci-image build --python 3.9` and try again. ``` That isn't a problem generally as `ruff` will fail before and convert it into double quotes. But rather than doing that, we fix it at source.
While working on this apache#43243, I was following https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst and I ran into an error with pre-commit hook. When running the revision command as follows: ``` root@f1f78138ad78:/opt/airflow/airflow# alembic revision -m "New revision" Generating /opt/airflow/airflow/migrations/versions/01b38be821e9_new_revision.py ... done ``` It creates a file as follows: ```python """New revision Revision ID: cd7be1ae8b80 Revises: 05234396c6fc Create Date: 2024-10-22 01:44:17.873864 """ import sqlalchemy as sa from alembic import op # revision identifiers, used by Alembic. revision = 'cd7be1ae8b80' down_revision = '05234396c6fc' branch_labels = None depends_on = None ``` Notice single quotes in `revision` & `down_revision`. Now if I just run that single pre-commit hook (`update-migration-references`), it fails ``` ❯ pre-commit run "update-migration-references" --all-files Update migration ref doc.................................................Failed - hook id: update-migration-references - exit code: 1 Using 'uv' to install Airflow Using airflow version from current sources Updating migration reference for airflow Making sure airflow version updated Making sure there's no mismatching revision numbers Traceback (most recent call last): File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 246, in <module> correct_mismatching_revision_nums(revisions=revisions) File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 230, in correct_mismatching_revision_nums new_content = content.replace(revision_id_match.group(1), revision_match.group(1), 1) AttributeError: 'NoneType' object has no attribute 'group' Error 1 returned If you see strange stacktraces above, run `breeze ci-image build --python 3.9` and try again. ``` That isn't a problem generally as `ruff` will fail before and convert it into double quotes. But rather than doing that, we fix it at source.
While working on this apache#43243, I was following https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst and I ran into an error with pre-commit hook. When running the revision command as follows: ``` root@f1f78138ad78:/opt/airflow/airflow# alembic revision -m "New revision" Generating /opt/airflow/airflow/migrations/versions/01b38be821e9_new_revision.py ... done ``` It creates a file as follows: ```python """New revision Revision ID: cd7be1ae8b80 Revises: 05234396c6fc Create Date: 2024-10-22 01:44:17.873864 """ import sqlalchemy as sa from alembic import op # revision identifiers, used by Alembic. revision = 'cd7be1ae8b80' down_revision = '05234396c6fc' branch_labels = None depends_on = None ``` Notice single quotes in `revision` & `down_revision`. Now if I just run that single pre-commit hook (`update-migration-references`), it fails ``` ❯ pre-commit run "update-migration-references" --all-files Update migration ref doc.................................................Failed - hook id: update-migration-references - exit code: 1 Using 'uv' to install Airflow Using airflow version from current sources Updating migration reference for airflow Making sure airflow version updated Making sure there's no mismatching revision numbers Traceback (most recent call last): File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 246, in <module> correct_mismatching_revision_nums(revisions=revisions) File "/opt/airflow/scripts/in_container/run_migration_reference.py", line 230, in correct_mismatching_revision_nums new_content = content.replace(revision_id_match.group(1), revision_match.group(1), 1) AttributeError: 'NoneType' object has no attribute 'group' Error 1 returned If you see strange stacktraces above, run `breeze ci-image build --python 3.9` and try again. ``` That isn't a problem generally as `ruff` will fail before and convert it into double quotes. But rather than doing that, we fix it at source.
[skip ci] closes apache#43161 part of [AIP-72](https://github.com/orgs/apache/projects/405) | [AIP-72 details here](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK)
[skip ci]
closes #43161 part of AIP-72.
As part of the ongoing work for AIP-72: Task Execution Interface, we are migrating the task_instance table to use a UUID primary key. This change is being made to simplify task instance identification, especially when communicating between the executor and workers.
Currently, the primary key of task_instance is a composite key consisting of
dag_id, task_id, run_id, and map_index
as shown below. This migration introduces a UUID v7 column (id
) as the new primary key.airflow/airflow/models/taskinstance.py
Lines 1815 to 1819 in b4269f3
Why UUID v7?
The UUID v7 format was chosen because of its improved temporal sorting capabilities. For existing records, UUID v7 will be generated using either the queued_dttm, start_date, or the current timestamp.
(From this blog post.)
This change also includes:
TODOs
pgcrypto
TaskInstance
model file to addid
task_instance_history
table?