Merge pull request #28 from fivetran/aug_2023_api_updates

Aug 2023 api updates

.buildkite/scripts/run_models.sh

@@ -19,6 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{iterable__using_campaign_label_history: false, iterable__using_user_unsubscribed_message_type_history: false, iterable__using_campaign_suppression_list_history: false, iterable__using_user_device_history: true}' --full-refresh --target "$db"
+dbt run --vars '{iterable__using_campaign_label_history: false, iterable__using_user_unsubscribed_message_type_history: false, iterable__using_campaign_suppression_list_history: false}' --full-refresh --target "$db"
+dbt run --vars '{ does_table_exist('user_unsubscribed_message_type'): false, does_table_exist('user_unsubscribed_channel'): false}' --full-refresh --target "$db"
 dbt test --target "$db"
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

.gitignore

@@ -3,4 +3,5 @@ target/
 dbt_modules/
 logs/
 .DS_Store
-dbt_packages/
+dbt_packages/
+package-lock.yml

CHANGELOG.md

@@ -1,3 +1,44 @@
+# dbt_iterable_source v0.8.0
+
+## API Updates
+[PR #28](https://github.com/fivetran/dbt_iterable_source/pull/28) includes updates in response to the [Aug 2023 updates](https://fivetran.com/docs/applications/iterable/changelog#august2023) for the Iterable connector.
+
+## 🚨 Breaking Changes 🚨
+- We have removed the `stg_iterable__user_device_history` model as the value is marginal.
+- The following models have been renamed to reflect the new (non history) table names synced by the connector.
+
+| **Previous Model Name** | **Updated Model Name** |
+|---------------------------|---------------------------|
+| `stg_iterable__user_unsub_message_type_history` | `stg_iterable__user_unsub_message_type` |
+| `stg_iterable__user_unsub_message_type_history_tmp` | `stg_iterable__user_unsub_message_type_tmp` |
+| `stg_iterable__user_unsubscribed_channel_history` | `stg_iterable__user_unsubscribed_channel` |
+| `stg_iterable__user_unsubscribed_channel_history_tmp` | `stg_iterable__user_unsubscribed_channel_tmp` |
+
+## Feature Updates
+- Added the fields `updated_by_user_id` and `message_medium` to `stg_iterable__campaign_history`
+- Added the fields `_fivetran_id` and `_fivetran_user_id` to `stg_iterable__event`
+- Added the fields `catalog_collection_count, catalog_lookup_count, city, clicked_url, country, error_code, expires_at, from_phone_number, in_app_body, is_sms_estimation, labels, message_status, mms_send_count, reason, sms_send_count, and _fivetran_user_id` to `stg_iterable__event_extension`
+- Added the fields `list_description` to `stg_iterable__list`
+- Added the fields `message_type_created_at, frequency_cap, rate_limit_per_minute, subscription_policy, and message_type_updated_at` to `stg_iterable__message_type`
+- Added the field `_fivetran_user_id` to `stg_iterable__user_history`
+- Introduced passthrough column functionality for `user_history` and `event_extension` objects. For more information refer to the [README](./README.md#passing-through-additional-fields).
+
+## Field Removals
+- Removed the following fields from `stg_iterable__event_extension`: `deeplink_android, deeplink_ios, image_url, is_ghost_push, link_id, sms_provider_response_code, sms_provider_response_message, sms_provider_response_more_info, sms_provider_response_status, sound`
+
+## Test Updates
+- The unique tests in `stg_iterable__event` and `stg_iterable__event_extension` now uses `unique_event_id`. This is a generated surrogate key from `event_id` and `_fivetran_user_id`. Previously it was just `event_id`, but now the unique keys in the respective source tables involve a combination of `event_id` and `_fivetran_user_id`, if it exists. `_fivetran_user_id` is a key available in the Aug 2023 Iterable API that's generated by taking hash of user_id and/or email fields depending on the project type.
+- The unique test in `stg_iterable__user_history` now tests for a unique combination of `fivetran_id` and `updated_at`. Previously it was just email, however with the Iterable schema updates, that may not exist for each user. This is because based on the channel, not all users have an associated email, particularly if it's marketing done via a project type based on user ID where an email identifier may not be unique. `fivetran_id` is generated by taking hash of user_id and/or email fields depending on the project type. For more information refer to the Iterable [docs](https://support.iterable.com/hc/en-us/articles/9216719179796-Project-Types-and-Unique-Identifiers-).
+
+## Under the Hood:
+- Past August 2023, the `user_unsubscribed_channel_history` and `user_unsubscribed_message_type_history` Iterable objects will no longer be history tables. In addition, the columns have changed. We have checks in place that will automatically persist the respective columns depending on what exists in your schema.
+ - In addition, we have generated keys for these 2 respective staging models that we will use for uniqueness tests. 
+ - For `stg_iterable__user_unsubscribed_channel`, which pulls from the `user_unsubscribed_channel` table, the unique key generated is depends on what exists in your tables (determined by whether you are using the history version). For the history version, it's a key generated from `email`, `channel_id`, and `updated_at`. If not using the history table, it uses `_fivetran_user_id` and `channel_id`. 
+ - For `stg_iterable__user_unsub_message_type`, which pulls from `user_unsubscribed_message_type_history`, the unique key is generated from `email`, `message_type_id`, and `updated_at` if using the history version. If not using the history table, it uses `_fivetran_user_id` and `message_type_id`.
+ - For more information, refer to our [documentation](https://github.com/fivetran/dbt_iterable_source/blob/main/models/stg_iterable.yml).
+- Explicitly casts the `*_id` fields in the staging models as strings. This ensures that downstream, joins will work and the JSON as string operations are not impacted by errors if the connector syncs this field as a JSON datatype.
+
+
 # dbt_iterable_source v0.7.0
 ## 🎉 Feature Update 🎉
 - Databricks compatibility! ([#25](https://github.com/fivetran/dbt_iterable_source/pull/25))

README.md

@@ -27,6 +27,13 @@ To use this dbt package, you must have the following:
 
 - At least one Fivetran Iterable connector syncing data into your destination.
 - A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
+
+#### Unsubscribe tables are no longer history tables
+
+For connectors created past August 2023, the `user_unsubscribed_channel_history` and `user_unsubscribed_message_type_history` Iterable objects will no longer be history tables as part of schema changes following Iterable's API updates. The fields have also changed. There is no lift required, since we have checks in place that will automatically persist the respective fields depending on what exists in your schema (they will still be history tables if you are using the old schema).
+
+*Please be sure you are syncing them as either both history or non-history.*
+
 ## Step 2: Install the package
 Include the following Iterable package version in your `packages.yml` file.
 
@@ -35,7 +42,7 @@ Include the following Iterable package version in your `packages.yml` file.
 ```yaml
 packages:
  - package: fivetran/iterable_source
- version: [">=0.7.0", "<0.8.0"]
+ version: [">=0.8.0", "<0.9.0"]
 ```
 
 ## Step 3: Define database and schema variables
@@ -51,21 +58,43 @@ vars:
 
 Your Iterable connector might not sync every table that this package expects. If your syncs exclude certain tables, it is either because you do not use that functionality in Iterable or have actively excluded some tables from your syncs. In order to enable or disable the relevant tables in the package, you will need to add the following variable(s) to your `dbt_project.yml` file.
 
-By default, all variables are assumed to be `true` (with exception of `iterable__using_user_device_history`, which is set to `false`). 
+By default, all variables are assumed to be `true`. 
 
 ```yml
 vars:
  iterable__using_campaign_label_history: false # default is true
- iterable__using_user_unsubscribed_message_type_history: false # default is true
+ iterable__using_user_unsubscribed_message_type: false  # default is true
  iterable__using_campaign_suppression_list_history: false # default is true 
 
- iterable__using_user_device_history: true # default is false
 ```
 
+
 ## (Optional) Step 5: Additional configurations
 <details><summary>Expand for details</summary>
 <br>
 
+### Passing Through Additional Fields
+
+This package includes fields we judged were standard across Iterable users. However, the Fivetran connector allows for additional columns to be brought through in the `event_extension` and `user_history` objects. Therefore, if you wish to bring them through, leverage our passthrough column variables.
+
+**Notice**: A `dbt run --full-refresh` is required each time these variables are edited.
+
+These variables allow for the passthrough fields to be aliased (alias) and casted (transform_sql) if desired, but not required. Datatype casting is configured via a sql snippet within the transform_sql key. You may add the desired sql while omitting the as field_name at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:
+
+```yml
+# dbt_project.yml
+
+vars:
+ iterable_event_extension_pass_through_columns:
+ - name: "event_extension_field"
+ alias: "renamed_field"
+ transform_sql: "cast(renamed_field as string)"
+ iterable_user_history_pass_through_columns:
+ - name: "user_attribute"
+ alias: "renamed_user_attribute"
+ - name: "user_attribute_2"
+```
+
 ### Changing the Build Schema
 
 By default, this package will build the Iterable staging models within a schema titled (<target_schema> + `_stg_iterable`) in your target database. If this is not where your would like you Iterable staging data to be written to, add the following configuration to your `dbt_project.yml` file:
@@ -93,6 +122,7 @@ By default, this package refers to the new table (`CAMPAIGN_SUPPRESSION_LIST_HIS
 vars:
  iterable_campaign_suppression_list_history_identifier: "campaign_supression_list_history"
 ```
+
 </details>
 
 ## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'iterable_source'
 
-version: '0.7.0'
+version: '0.8.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
@@ -23,6 +23,11 @@ vars:
  message_type: "{{ source('iterable', 'message_type') }}"
  template_history: "{{ source('iterable', 'template_history') }}"
  user_history: "{{ source('iterable', 'user_history') }}"
- user_device_history: "{{ source('iterable', 'user_device_history') }}"
+ user_device: "{{ source('iterable', 'user_device') }}"
+ user_unsubscribed_channel: "{{ source('iterable', 'user_unsubscribed_channel') }}"
+ user_unsubscribed_message_type: "{{ source('iterable', 'user_unsubscribed_message_type') }}"
  user_unsubscribed_channel_history: "{{ source('iterable', 'user_unsubscribed_channel_history') }}"
  user_unsubscribed_message_type_history: "{{ source('iterable', 'user_unsubscribed_message_type_history') }}"
+
+ iterable_event_extension_pass_through_columns: []
+ iterable_user_history_pass_through_columns: []

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
  pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
  dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
  port: 5439
- schema: iterable_source_integration_tests_1
+ schema: iterable_source_integration_tests_3
  threads: 8
  bigquery:
  type: bigquery
  method: service-account-json
  project: 'dbt-package-testing'
- schema: iterable_source_integration_tests_1
+ schema: iterable_source_integration_tests_3
  threads: 8
  keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
  snowflake:
@@ -33,7 +33,7 @@ integration_tests:
  role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
  database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
  warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
- schema: iterable_source_integration_tests_1
+ schema: iterable_source_integration_tests_3
  threads: 8
  postgres:
  type: postgres
@@ -42,13 +42,13 @@ integration_tests:
  pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
  dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
  port: 5432
- schema: iterable_source_integration_tests_1
+ schema: iterable_source_integration_tests_3
  threads: 8
  databricks:
  catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
  host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
  http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
- schema: iterable_source_integration_tests_1
+ schema: iterable_source_integration_tests_3
  threads: 8
  token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
  type: databricks

integration_tests/dbt_project.yml

@@ -1,12 +1,12 @@
 config-version: 2
 name: 'iterable_source_integration_tests'
 
-version: '0.7.0'
+version: '0.8.0'
 profile: 'integration_tests'
 
 vars:
  iterable_source:
- iterable_schema: iterable_source_integration_tests_1
+ iterable_schema: iterable_source_integration_tests_3
  iterable_campaign_history_identifier: "campaign_history_data"
  iterable_campaign_label_history_identifier: "campaign_label_history_data"
  iterable_campaign_list_history_identifier: "campaign_list_history_data"
@@ -18,7 +18,9 @@ vars:
  iterable_message_type_identifier: "message_type_data"
  iterable_template_history_identifier: "template_history_data"
  iterable_user_history_identifier: "user_history_data"
- iterable_user_device_history_identifier: "user_device_history_data"
+ iterable_user_device_identifier: "user_device_data"
+ iterable_user_unsubscribed_channel_identifier: "user_unsubscribed_channel_data"
+ iterable_user_unsubscribed_message_type_identifier: "user_unsubscribed_message_type_data"
  iterable_user_unsubscribed_channel_history_identifier: "user_unsubscribed_channel_history_data"
  iterable_user_unsubscribed_message_type_history_identifier: "user_unsubscribed_message_type_history_data"
  is_old_spelling: false
@@ -30,10 +32,12 @@ seeds:
  _fivetran_synced: timestamp
  event_data:
  +column_types:
+ _fivetran_user_id: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string'}}"
  transactional_data: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string'}}"
  created_at: timestamp
  event_extension_data:
  +column_types:
+ _fivetran_user_id: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string'}}"
  payload: "{{ 'varchar(100)' if target.type in ('redshift', 'postgres') else 'string'}}"
 
 dispatch:

integration_tests/requirements.txt

@@ -4,4 +4,4 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
+dbt-databricks>=1.6.0,<2.0.0

integration_tests/seeds/campaign_history_data.csv

@@ -1,3 +1,3 @@
-id,updated_at,template_id,recurring_campaign_id,campaign_state,created_at,created_by_user_id,ended_at,name,send_size,start_at,type,workflow_id,_fivetran_synced
-2452145,2021-06-11 13:31:13.748,3357343,,Ready,2021-06-11 13:31:13.748,[email protected],,20180618_fun_campaign campaign from Workflow 157274,,,Triggered,157274,2021-06-11 13:32:03.771
-2452149,2021-06-11 13:31:20.601,3357342,,Ready,2021-06-11 13:31:20.601,[email protected],,20180620_other_thing campaign from Workflow 157274,,,Triggered,157274,2021-06-11 13:32:03.771
+id,updated_at,template_id,recurring_campaign_id,campaign_state,created_at,created_by_user_id,ended_at,name,send_size,start_at,type,workflow_id,_fivetran_synced," updated_by_user_id",message_medium
+2452145,2021-06-11 13:31:13.748,3357343,,Ready,2021-06-11 13:31:13.748,[email protected],,20180618_fun_campaign campaign from Workflow 157274,,,Triggered,157274,2021-06-11 13:32:03.771,12334,Email
+2452149,2021-06-11 13:31:20.601,3357342,,Ready,2021-06-11 13:31:20.601,[email protected],,20180620_other_thing campaign from Workflow 157274,,,Triggered,157274,2021-06-11 13:32:03.771,35644,Email

integration_tests/seeds/event_data.csv

@@ -1,3 +1,3 @@
-_fivetran_id,email,campaign_id,message_id,content_id,created_at,event_name,ip,message_bus_id,message_type_id,recipient_state,status,unsub_source,user_agent,user_agent_device,transactional_data,additional_properties,_fivetran_synced
-HFNB4mJWJeXKk7AwJAEPCCmvpkM=,[email protected],,,,2020-06-12 14:42:41.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[575561]"",""hasThisThing"": ""Yes"",""hasOtherThing"": ""Yes"",""hasOverdueThing"": ""Yes"",""didAThing"": ""Yes"",""isAFriend"": ""Yes"",""isCool"": ""Yes"",""isMean"": ""No"",""profileUpdatedAt"": ""2020-06-12 14:39:11 +00:00"",""signupSource"": ""Import""}",2020-08-04 12:04:05.997
-8nxrQcLoFB96e+oOBqzBDsXnMBU=,[email protected],,,,2020-03-26 12:30:43.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079]"",""profileUpdatedAt"": ""2020-03-26 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-04 14:10:39.711
+_fivetran_id,email,campaign_id,message_id,content_id,created_at,event_name,ip,message_bus_id,message_type_id,recipient_state,status,unsub_source,user_agent,user_agent_device,transactional_data,additional_properties,_fivetran_synced, _fivetran_user_id
+HFNB4mJWJeXKk7AwJAEPCCmvpkM=,[email protected],,,,2020-06-12 14:42:41.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[575561]"",""hasThisThing"": ""Yes"",""hasOtherThing"": ""Yes"",""hasOverdueThing"": ""Yes"",""didAThing"": ""Yes"",""isAFriend"": ""Yes"",""isCool"": ""Yes"",""isMean"": ""No"",""profileUpdatedAt"": ""2020-06-12 14:39:11 +00:00"",""signupSource"": ""Import""}",2020-08-04 12:04:05.997,987654
+8nxrQcLoFB96e+oOBqzBDsXnMBU=,[email protected],,,,2020-03-26 12:30:43.000,emailSubscribe,,,,,,,,,null,"{""channelIds"": ""[]"",""emailListIds"": ""[501079]"",""profileUpdatedAt"": ""2020-03-26 12:30:42 +00:00"",""signupSource"": ""Import""}",2020-08-04 14:10:39.711,12345