Add docs and values to run Airflow locally

[ividito]

Summary:

Docs on getting Airflow set up using Kubernetes in Docker (KinD) to host a local cluster. This can enable local development without relying on MWAA environments, and can be more extensible than the mwaa-local-runner. This can also work for developing SM2A DAGs, while we prepare to migrate to that architecture.

[ciaransweet]

Hopefully small comments but 'blocked' from the rest of the steps because of the docker build

[ciaransweet]

Running just the create command appeared to switch context for me:

kind create cluster --name airflow-cluster
Creating cluster "airflow-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-airflow-cluster"
You can now use your cluster with:

kubectl cluster-info --context kind-airflow-cluster

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

[ciaransweet]

This fails for what looks like issues installing rasterio:

2.305 Collecting rasterio>=1.3.3
2.319   Downloading rasterio-1.3.10.tar.gz (412 kB)
2.334      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 412.9/412.9 kB 29.6 MB/s eta 0:00:00
2.370   Installing build dependencies: started
6.820   Installing build dependencies: finished with status 'done'
6.821   Getting requirements to build wheel: started
7.083   Getting requirements to build wheel: finished with status 'error'
7.086   error: subprocess-exited-with-error
7.086
7.086   × Getting requirements to build wheel did not run successfully.
7.086   │ exit code: 1
7.086   ╰─> [3 lines of output]
7.086       <string>:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
7.086       WARNING:root:Failed to get options via gdal-config: [Errno 13] Permission denied: 'gdal-config'
7.086       ERROR: A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.
7.086       [end of output]
7.086
7.086   note: This error originates from a subprocess, and is likely not a problem with pip.
7.087 error: subprocess-exited-with-error
7.087
7.087 × Getting requirements to build wheel did not run successfully.
7.087 │ exit code: 1
7.087 ╰─> See above for output.
7.087
7.087 note: This error originates from a subprocess, and is likely not a problem with pip.
7.155
7.155 [notice] A new release of pip available: 22.3.1 -> 24.2
7.155 [notice] To update, run: pip install --upgrade pip
------
Dockerfile.airflow:4
--------------------
   2 |     COPY dags/requirements.txt .
   3 |     COPY dags/requirements-constraints.txt .
   4 | >>> RUN pip install --no-cache-dir -r requirements.txt -c requirements-constraints.txt
   5 |     COPY --chown=airflow:root ./dags/ /opt/airflow/dags/
   6 |
--------------------
ERROR: failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c pip install --no-cache-dir -r requirements.txt -c requirements-constraints.txt" did not complete successfully: exit code: 1

[ciaransweet]

Fixed on mac with: docker build --platform linux/amd64 -f ./local-setup/Dockerfile.airflow -t veda-airflow .