Fix DbtVirtualenvBaseOperator to use correct virtualenv Python path #1252
+32
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR addresses an issue where the
DbtVirtualenvBaseOperator
was executing dbt commands using the system-wide Python path instead of the virtualenv path. The root cause was that the self reference in therun_subprocess
method was bound to a different instance than the one created during initialization, likely due to Airflow's DAG pickling mechanism.To resolve this, we've refactored the
invoke_dbt
andhandle_exception
methods to be properties. This ensures that they dynamically reference the correct method of the current instance at runtime, rather than being bound to a potentially stale instance from initialization.Related Issue(s)
fix #1246
This may be related to issue #958 in version 1.5.0
Breaking Change?
No
Checklist
Additional Notes
I acknowledge that ideally, a test should be added to reproduce the original issue and verify the fix. However, I found it challenging to create an appropriate test, especially considering that this might require an integration test with Airflow to properly simulate Airflow behavior. If there are suggestions for how to effectively test this scenario, I would greatly appreciate the guidance.
I sincerely apologize for introducing this bug in the first place with PR #1200. I understand this has caused inconvenience, and I'm grateful for the opportunity to fix it. I kindly request a thorough review of these changes to ensure we've fully addressed the issue without introducing new problems.