[AGENT-5360] Create Airflow Operator for Custom Job #86
Conversation
oleksandr-saienko
requested review from
elatt,
dulcardo and
alexromanyuk
as code owners
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
pipelineController framework is deprecated, please migrate CICD for this repository to Harness |
|
FYI this is coming to experimental client - https://github.com/datarobot/public_api_client/pull/2941 |
| params={}, | ||
| ) | ||
| def create_custom_custom_job(): | ||
| create_custom_job_op = CreateCustomJobOperator( |
There was a problem hiding this comment.
Does it make sense to create another kind of create operator which includes files, environment and runtime parameters all in one? This way the DAG could be simplified. Having these more specific operators makes sense, but I think a super-creator operator could be useful.
| response = dr.client.get_client().post( | ||
| "customJobs/", | ||
| data={ | ||
| "name": self.name, | ||
| "description": self.description, | ||
| }, | ||
| ) |
There was a problem hiding this comment.
With Custom Jobs going GA in 10.0 and having support in the Python Client at that time, I think we should postpone this PR until that time.
This repository is public. Do not put here any private DataRobot or customer's data: code, datasets, model artifacts, .etc.
Summary
Added the next operatos:
Rationale