Merge branch 'main' into fix-virtualenv-operator-bug

.pre-commit-config.yaml

@@ -47,7 +47,7 @@ repos:
  - id: remove-tabs
  exclude: ^docs/make.bat$|^docs/Makefile$|^dev/dags/dbt/jaffle_shop/seeds/raw_orders.csv$
  - repo: https://github.com/asottile/pyupgrade
- rev: v3.17.0
+ rev: v3.18.0
  hooks:
  - id: pyupgrade
  args:
@@ -65,7 +65,7 @@ repos:
  - id: black
  args: ["--config", "./pyproject.toml"]
  - repo: https://github.com/asottile/blacken-docs
- rev: 1.18.0
+ rev: 1.19.0
  hooks:
  - id: blacken-docs
  alias: black

README.rst

@@ -83,3 +83,8 @@ Privacy Notice
 ______________
 
 This project follows `Astronomer's Privacy Policy <https://www.astronomer.io/privacy/>`_
+
+.. Tracking pixel for Scarf
+.. raw:: html
+
+ <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=ae43a92a-5a21-4c77-af8b-99c2242adf93" />

cosmos/plugin/templates/dbt_docs_not_set_up.html

@@ -1,6 +1,6 @@
 {% extends base_template %}
 {% block content %}
- <h1 style="display: block; padding-top: 15px; padding-bottom: 15px">⚠️ Your dbt docs are not set up yet! ⚠️</h1>
+ <h1 style="display: block; padding-top: 15px; padding-bottom: 15px">⚠️ Your dbt docs are not set up yet! dbt_docs_dir was not found!⚠️</h1>
 
  <p>
  Read the <a href="https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html">Astronomer Cosmos docs</a> for information on how to set up dbt docs.

cosmos/profiles/clickhouse/user_pass.py

@@ -1,4 +1,4 @@
-"""Maps Airflow Postgres connections using user + password authentication to dbt profiles."""
+"""Maps Airflow Clickhouse connections using user + password authentication to dbt profiles."""
 
 from __future__ import annotations
 
@@ -20,7 +20,7 @@ class ClickhouseUserPasswordProfileMapping(BaseProfileMapping):
 
  required_fields = [
  "host",
- "login",
+ "user",
  "schema",
  "clickhouse",
  ]
@@ -29,7 +29,7 @@ class ClickhouseUserPasswordProfileMapping(BaseProfileMapping):
  ]
  airflow_param_mapping = {
  "host": "host",
- "login": "login",
+ "user": "login",
  "password": "password",
  "port": "port",
  "schema": "schema",

docs/index.rst

@@ -28,19 +28,62 @@
 
 |fury| |ossrank| |downloads| |pre-commit|
 
-Run your dbt Core projects as `Apache Airflow® <https://airflow.apache.org/>`_ DAGs and Task Groups with a few lines of code. Benefits include:
 
-- Run dbt projects against Airflow connections instead of dbt profiles
-- Native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow
-- Run tests immediately after a model is done to catch issues early
-- Utilize Airflow's data-aware scheduling to run models immediately after upstream ingestion
-- Turn each dbt model into a task/task group complete with retries, alerting, etc.
+Welcome to Astronomer Cosmos! Whether you're an experienced data practitioner or just getting started, Cosmos makes it
+simple to manage and orchestrate your dbt workflows using `Apache Airflow® <https://airflow.apache.org/>`_, saving you
+time and effort. By automatically turning dbt workflows into Airflow DAGs, Cosmos allows you to focus on building
+high-quality data models without the hassle of managing complex integrations.
 
+To get started right away, please check out our `Quickstart Guides <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_.
+You can also explore more examples in `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_
+or in the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.
 
-Example Usage
-___________________
+To learn more about about Cosmos, please read on.
 
-You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an example with the `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_:
+
+What Is Astronomer Cosmos?
+___________________________
+
+Astronomer Cosmos is an open-source library that bridges Apache Airflow and dbt, allowing you to easily transform your
+dbt projects into Airflow DAGs and manage everything seamlessly. With Cosmos, you can write your data transformations
+using dbt and then schedule and orchestrate them with Airflow, making the entire process smooth and straightforward.
+
+**Why Cosmos?** Integrating dbt and Airflow can be complex, but Cosmos simplifies it by seamlessly connecting these
+powerful tools—letting you focus on what matters most: delivering impactful data models and results without getting
+bogged down by technical challenges.
+
+
+Why Should You Use Cosmos?
+___________________________
+
+Cosmos makes orchestrating dbt workflows:
+
+- **Effortless**: Transform your dbt projects into Airflow DAGs without writing extra code—Cosmos handles the heavy lifting.
+- **Reliable**: Rely on Airflow's robust scheduling and monitoring features to ensure your dbt workflows run smoothly and efficiently.
+- **Scalable**: Easily scale your workflows to match growing data demands, thanks to Airflow's distributed capabilities.
+
+Whether you're handling intricate data tasks or looking to streamline your processes, Cosmos helps you orchestrate dbt
+with Airflow effortlessly, saving you time and letting you focus on what truly matters—creating impactful insights.
+
+
+Example Usage: Jaffle Shop Project
+__________________________________
+
+Let's explore a practical example to see how Cosmos can convert the dbt workflow into an Airflow DAG.
+
+The `jaffle_shop project <https://github.com/dbt-labs/jaffle_shop>`_ is a sample dbt project that simulates an e-commerce store's data.
+The project includes a series of dbt models that transform raw data into structured tables, such as sales, customers, and products.
+
+Below, you can see what the original dbt workflow looks like in a lineage graph. This graph helps illustrate the
+relationships between different models:
+
+.. image:: /_static/jaffle_shop_dbt_graph.png
+
+Cosmos can take this dbt workflow and convert it into an Airflow DAG, allowing you to leverage Airflow's scheduling and
+orchestration capabilities.
+
+To convert this dbt workflow into an Airflow DAG, create a new DAG definition file, import ``DbtDag`` from the Cosmos library,
+and fill in a few parameters, such as the dbt project directory path and the profile name:
 
 ..
  The following renders in Sphinx but not Github:
@@ -51,32 +94,41 @@ You can render a Cosmos Airflow DAG using the ``DbtDag`` class. Here's an exampl
  :end-before: [END local_example]
 
 
-This will generate an Airflow DAG that looks like this:
+This code snippet will generate an Airflow DAG that looks like this:
 
 .. image:: https://raw.githubusercontent.com/astronomer/astronomer-cosmos/main/docs/_static/jaffle_shop_dag.png
 
-Getting Started
-_______________
+``DbtDag`` is a custom DAG generator that converts dbt projects into Airflow DAGs and accepts Cosmos-specific args like
+``fail_fast`` to immediately fail a dag if dbt fails to process a resource, or ``cancel_query_on_kill`` to cancel any running
+queries if the task is externally killed or manually set to failed in Airflow. ``DbtDag`` also accepts standard DAG arguments such
+as ``max_active_tasks``, ``max_active_runs`` and ``default_args``.
 
-Check out the Quickstart guide on our `docs <https://astronomer.github.io/astronomer-cosmos/getting_started/index.html>`_. See more examples at `/dev/dags <https://github.com/astronomer/astronomer-cosmos/tree/main/dev/dags>`_ and at the `cosmos-demo repo <https://github.com/astronomer/cosmos-demo>`_.
+With Cosmos, transitioning from a dbt workflow to a proper Airflow DAG is seamless, giving you the best of both tools
+for managing and scaling your data workflows.
 
 
 Changelog
 _________
 
 We follow `Semantic Versioning <https://semver.org/>`_ for releases.
-Check `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_
+Refer to `CHANGELOG.rst <https://github.com/astronomer/astronomer-cosmos/blob/main/CHANGELOG.rst>`_
 for the latest changes.
 
-Contributing Guide
+
+Join the Community
 __________________
 
-All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.
+Have questions, need help, or interested in contributing? We welcome all contributions and feedback!
+
+- Join the community on Slack! You can find us in the Airflow Slack workspace `#airflow-dbt <https://apache-airflow.slack.com/archives/C059CC42E9W>`_ channel. If you don't have an account, click `here <https://apache-airflow-slack.herokuapp.com/>`_ to sign up.
+
+- Report bugs, request features, or ask questions by creating an issue in the `GitHub repository <https://github.com/astronomer/astronomer-cosmos/issues/new/choose>`_.
+
+- Want to contribute new features, bug fixes or documentation enhancements? Please refer to our `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.
 
-A detailed overview on how to contribute can be found in the `Contributing Guide <https://astronomer.github.io/astronomer-cosmos/contributing>`_.
-Find out more about `our contributors <https://astronomer.github.io/astronomer-cosmos/contributors>`_.
+- Check out this `link <https://astronomer.github.io/astronomer-cosmos/contributors>`_. to learn more about our current contributors
 
-As contributors and maintainers to this project, you are expected to abide by the
+Note that contributors and maintainers are expected to abide by the
 `Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.
 
 

tests/profiles/clickhouse/test_clickhouse_userpass.py

@@ -83,7 +83,7 @@ def test_profile_args(mock_clickhouse_conn: Connection) -> None:
  assert profile_mapping.profile == {
  "type": "clickhouse",
  "schema": mock_clickhouse_conn.schema,
- "login": mock_clickhouse_conn.login,
+ "user": mock_clickhouse_conn.login,
  "password": "{{ env_var('COSMOS_CONN_GENERIC_PASSWORD') }}",
  "driver": "native",
  "port": 9000,
@@ -102,7 +102,7 @@ def test_mock_profile() -> None:
  assert profile_mapping.mock_profile == {
  "type": "clickhouse",
  "schema": "mock_value",
- "login": "mock_value",
+ "user": "mock_value",
  "driver": "native",
  "port": 9000,
  "host": "mock_value",