Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Airflow DAG setup/scheduling with DeepLynx #489

Open
jack-cavaluzzi opened this issue Oct 21, 2024 · 0 comments
Open

Integrate Airflow DAG setup/scheduling with DeepLynx #489

jack-cavaluzzi opened this issue Oct 21, 2024 · 0 comments
Assignees

Comments

@jack-cavaluzzi
Copy link
Collaborator

Reason

We want users to be able to easily setup/register a given DAG "instance", from their given DeepLynx container. They'll need to provide some configuration and authentication info, and have that stored somewhere and be accessible to both Airflow and DeepLynx.

Design

I think most of that info can be stored in Airflow's Connection object. If so, we'll need to decide the exact backend we use for that. We could go with a standard Airflow approach of storing our Connections in a table in our Airflow metadata database. If we go that route, we could have our Connections table stored in a Postgres DB, and give access to both Airflow and DeepLynx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants