Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inject Python code that will be used in PythonOperator #90

Open
peleyal opened this issue Oct 2, 2020 · 1 comment
Open

Inject Python code that will be used in PythonOperator #90

peleyal opened this issue Oct 2, 2020 · 1 comment

Comments

@peleyal
Copy link
Contributor

peleyal commented Oct 2, 2020

I would like to figure out a way if I can inject some Python code at the beginning of my DAG.
The idea is that I want to use the python_callable field on a PythonOperator to call my code, however my code is going to be like 5-10 lines of code and I cannot really represent it with lambda (or I would like not to, cause it's going to be ugly!)

Currently I see two options but I would like to get your feedback please :) Thanks!

  1. Define a plugin that creates a new operator that inherits from PythonOperator and will do the magic. Then, Boundary Layer will load the YAML file I config in the plugin so the output DAG.py is valid. And, I'll also have to include the implementation itself (not the YAML) in the DAGs folder in airflow itself.
  2. A better option(which I'm not sure if it is supported) will be to have some way to paste code at the beginning of the DAG.py file. This I guess can be done using a pre (or post) processor. However, I'm not sure if Boundary Layer supports something like that.

While exploring these options, I preferred to file a bug and see what you guys think about it.
Thanks!

@peleyal
Copy link
Contributor Author

peleyal commented Oct 2, 2020

@jcraver1021 FYI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant