Make Task Instance primary key be a UUID

[kaxil]

[skip ci]

closes #43161 part of AIP-72.

As part of the ongoing work for AIP-72: Task Execution Interface, we are migrating the task_instance table to use a UUID primary key. This change is being made to simplify task instance identification, especially when communicating between the executor and workers.

Currently, the primary key of task_instance is a composite key consisting of dag_id, task_id, run_id, and map_index as shown below. This migration introduces a UUID v7 column (id) as the new primary key.

airflow/airflow/models/taskinstance.py

Lines 1815 to 1819 in b4269f3

	__tablename__ = "task_instance"
	task_id = Column(StringID(), primary_key=True, nullable=False)
	dag_id = Column(StringID(), primary_key=True, nullable=False)
	run_id = Column(StringID(), primary_key=True, nullable=False)
	map_index = Column(Integer, primary_key=True, nullable=False, server_default=text("-1"))

Why UUID v7?

The UUID v7 format was chosen because of its improved temporal sorting capabilities. For existing records, UUID v7 will be generated using either the queued_dttm, start_date, or the current timestamp.

(From this blog post.)

This change also includes:

• A new UUID v7 function for both PostgreSQL and MySQL.
• Backward-compatible indexes for easier querying by dag_id and run_id.
• Modifications to support the new primary key structure while maintaining foreign key constraints.

TODOs

• Check if the DB user has permissions to add CREATE extension for Postgres as we install pgcrypto
• Convert the raw SQL for dropping the primary key (CASCADE) into an Alembic operation. This will allow capturing foreign key constraints and managing them in a downgrade scenario. MySQL does not support dropping foreign keys using CASCADE with primary keys anyway.
• SQLite Support: Add SQLite support for UUID v7 generation using the uuid6 Python package.
• Add changes to TaskInstance model file to add id
• Is the same change needed for task_instance_history table?
• Batching: Add batching for handling large tables to avoid locking when updating records. Maybe a batch of 1000 rows, TBD!
• Ensure foreign key constraints are re-added during downgrades after the primary key has been changed back to the composite key.

[ashb]

A neat trick to avoid the need for pgcrypto extension: https://brandur.org/fragments/secure-bytes-without-pgcrypto

Though uuid_send() fn it uses doesn't exist on Pg10, it only comes from Pg11 onwards. However I do think it's reasonable to say 12 or 13 is the min required Pg version for Airflow 3 (Postgres only supports v13 onwards).

Oh, Pg 12 is officially all we support anyway, so I think using the uuid_send() function approach could work fine.