Deploying Delta Live Tables Pipelines on AWS with Terraform

About The Project

This repository contains example code for deploying Databricks Delta Live Tables pipelines with Terraform.

Delta Live Tables is a framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data, and Delta Live Tables manages task orchestration, cluster management, monitoring, data quality, and error handling.

Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure. The Terraform "databricks_pipeline" Resource is used to deploy Delta Live Tables.

Built With

• Terraform
• Delta Live Tables
• Python
• PySpark

Getting Started

Prerequisites

• Install Terraform
• Install AWS CLI
• Install Databricks CLI

Installation

1. Clone the repo

$ git clone https://github.com/husqvarnagroup/terraform-dlt-public.git

2. Update the dev and live Terraform var-files in the config folder with your settings. The "backend-config" files contain settings for the Terraform S3 Backend and the "variables" files contain settings for the Delta Live Tables Pipeline.

• config/dev-backend-config.tfvars
• config/dev-variables.tfvars
• config/live-backend-config.tfvars
• config/live-variables.tfvars

Usage

1. Open a terminal and ensure you are logged into the correct AWS Profile:

$ aws sso login --profile <aws_profile_name>

2. Initialize the Terraform backend with the dev or live backend configuration file with terraform init:

$ terraform init -backend-config config/<dev-or-live>-backend-config.tfvars

If changing between dev and live configurations you must include the -reconfigure flag:

$ terraform init -reconfigure -backend-config config/<dev-or-live>-backend-config.tfvars

3. Show the Terraform Plan with the dev or live variables file to confirm everything works:

$ terraform plan -var-file=config/<dev-or-live>-variables.tfvars

4. After confirming everything works, apply the changes with the dev or live variables file:

$ terraform apply -var-file=config/<dev-or-live>-variables.tfvars

5. To delete the previously deployed resources:

$ terraform destroy -var-file=config/<dev-or-live>-variables.tfvars

Delta Live Tables settings

Delta Live Tables Product Edition

Set the correct product edition based on features needed. Delta Live Tables Product Edition

Delta Live Tables Development or Production Mode

Set mode depending on environment. Delta Live Tables Development or Production mode](https://docs.microsoft.com/en-us/azure/databricks/data-engineering/delta-live-tables/delta-live-tables-concepts#--development-and-production-modes)