v0.9.0 dbt_iterable_source

PR #31 contains the following updates:

🚨 Breaking Changes 🚨

• Introduces variable iterable__using_event_extension to disable the stg_iterable__event_extension model. This allows the downstream models to run even if the source event_extension table does not exist. For more information on how to configure refer to the README.

Full Changelog: v0.8.1...v0.9.0

v0.8.1 dbt_iterable_source

PR #30 introduces the following updates:

Bug Fixes

• IMPORTANT: This only impacts customers who are using Fivetran connectors that were setup before August 2023, when user_unsubscribed_message_type_history and user_unsubscribed_channel_history were the tables being used for tracking unsubscribe events.
• Updated the stg_iterable__user_unsub_message_type model to partition the latest_batch_index value on message_type_id along with email. This brings in all unsubscribes for the unique_user_key across all message types for an email.
• Updated stg_iterable__user_unsubscribed_channel and stg_iterable__user_unsubscribed_channel to partition the latest_batch_index value on channel_id along with email. This brings in all unsubscribes for the unique_user_key across all channels for an email.
• These updates ensure that iterable__user_unsubscriptions in the dbt_iterable package brings in all the latest unsubscribing behavior for a user by channel and by message type, and not just the last unsubscription for a user based on the updated_at value.

Under the Hood

• Updated seed files to effectively run and test that the new partitions worked as expected.
• Updated pull request and issue templates.
• Included auto-releaser GitHub Actions workflow to automate future releases.

Full Changelog: v0.8.0...v0.8.1

v0.8.0 dbt_iterable_source

API Updates

PR #28 includes updates in response to the Aug 2023 updates for the Iterable connector.

🚨 Breaking Changes 🚨

• We have removed the stg_iterable__user_device_history model as the value is marginal.
• The following models have been renamed to reflect the new (non history) table names synced by the connector.

Previous Model Name	Updated Model Name
stg_iterable__user_unsub_message_type_history	stg_iterable__user_unsub_message_type
stg_iterable__user_unsub_message_type_history_tmp	stg_iterable__user_unsub_message_type_tmp
stg_iterable__user_unsubscribed_channel_history	stg_iterable__user_unsubscribed_channel
stg_iterable__user_unsubscribed_channel_history_tmp	stg_iterable__user_unsubscribed_channel_tmp

Feature Updates

• Added the fields updated_by_user_id and message_medium to stg_iterable__campaign_history
• Added the fields _fivetran_id and _fivetran_user_id to stg_iterable__event
• Added the fields catalog_collection_count, catalog_lookup_count, city, clicked_url, country, error_code, expires_at, from_phone_number, in_app_body, is_sms_estimation, labels, message_status, mms_send_count, reason, sms_send_count, and _fivetran_user_id to stg_iterable__event_extension
• Added the fields list_description to stg_iterable__list
• Added the fields message_type_created_at, frequency_cap, rate_limit_per_minute, subscription_policy, and message_type_updated_at to stg_iterable__message_type
• Added the field _fivetran_user_id to stg_iterable__user_history
• Introduced passthrough column functionality for user_history and event_extension objects. For more information refer to the README.

Field Removals

• Removed the following fields from stg_iterable__event_extension: deeplink_android, deeplink_ios, image_url, is_ghost_push, link_id, sms_provider_response_code, sms_provider_response_message, sms_provider_response_more_info, sms_provider_response_status, sound

Test Updates

• The unique tests in stg_iterable__event and stg_iterable__event_extension now uses unique_event_id. This is a generated surrogate key from event_id and _fivetran_user_id. Previously it was just event_id, but now the unique keys in the respective source tables involve a combination of event_id and _fivetran_user_id, if it exists. _fivetran_user_id is a key available in the Aug 2023 Iterable API that's generated by taking hash of user_id and/or email fields depending on the project type.
• The unique test in stg_iterable__user_history now tests for a unique combination of fivetran_id and updated_at. Previously it was just email, however with the Iterable schema updates, that may not exist for each user. This is because based on the channel, not all users have an associated email, particularly if it's marketing done via a project type based on user ID where an email identifier may not be unique. fivetran_id is generated by taking hash of user_id and/or email fields depending on the project type. For more information refer to the Iterable docs.

Under the Hood:

• Past August 2023, the user_unsubscribed_channel_history and user_unsubscribed_message_type_history Iterable objects will no longer be history tables. In addition, the columns have changed. We have checks in place that will automatically persist the respective columns depending on what exists in your schema.
  • In addition, we have generated keys for these 2 respective staging models that we will use for uniqueness tests.
  • For stg_iterable__user_unsubscribed_channel, which pulls from the user_unsubscribed_channel table, the unique key generated is depends on what exists in your tables (determined by whether you are using the history version). For the history version, it's a key generated from email, channel_id, and updated_at. If not using the history table, it uses _fivetran_user_id and channel_id.
  • For stg_iterable__user_unsub_message_type, which pulls from user_unsubscribed_message_type_history, the unique key is generated from email, message_type_id, and updated_at if using the history version. If not using the history table, it uses _fivetran_user_id and message_type_id.
  • For more information, refer to our documentation.
• Explicitly casts the *_id fields in the staging models as strings. This ensures that downstream, joins will work and the JSON as string operations are not impacted by errors if the connector syncs this field as a JSON datatype.

Full Changelog: v0.7.0...v0.8.0

v0.7.0 dbt_iterable_source

This release of dbt_iterable_source includes:

🎉 Feature Update 🎉

• Databricks compatibility! (#25)

🚘 Under the Hood 🚘

• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job. (#24)
• Updated the pull request templates. (#24)

Full Changelog: v0.6.0...v0.7.0

v0.6.0 dbt_iterable_source

This release of the dbt_iterable_source package includes the following updates:

🚨 Breaking Changes 🚨

• Added iterable_[source_table_name]_identifier variables to allow easier flexibility of the package to refer to source tables with different names.
• Note! For the table campaign_suppression_list_history, the identifier variable has been updated from iterable__campaign_suppression_list_history_table to iterable_campaign_suppression_list_history_identifier to align with the current naming convention. If you are using the former variable in your dbt_project.yml, you will need to update it for the package to run.
  (#23)

🎉 Features

• Updated README with identifier instructions and format update. (#23)

dbt_iterable_source v0.5.0

🚨 Breaking Changes 🚨:

PR #18 includes the following breaking changes:

• Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically {{ dbt_utils.<macro> }} have been updated to {{ dbt.<macro> }} for the below macros:
  • any_value
  • bool_or
  • cast_bool_to_text
  • concat
  • date_trunc
  • dateadd
  • datediff
  • escape_single_quotes
  • except
  • hash
  • intersect
  • last_day
  • length
  • listagg
  • position
  • replace
  • right
  • safe_cast
  • split_part
  • string_literal
  • type_bigint
  • type_float
  • type_int
  • type_numeric
  • type_string
  • type_timestamp
  • array_append
  • array_concat
  • array_construct
• For current_timestamp and current_timestamp_in_utc macros, the dispatch AND the macro names have been updated to the below, respectively:
  • dbt.current_timestamp_backcompat
  • dbt.current_timestamp_in_utc_backcompat
• Dependencies on fivetran/fivetran_utils have been upgraded, previously [">=0.3.0", "<0.4.0"] now [">=0.4.0", "<0.5.0"].

dbt_iterable_source v0.4.1

🎉 Documentation and Feature Updates

• Added variable iterable__using_campaign_suppression_list_history to disable stg_iterable__campaign_suppression_list_history if the respective source table is not being used. For how to configure refer to the README. (#21)

dbt_iterable_source v0.4.0

🎉 dbt v1.0.0 Compatibility 🎉

🚨 Breaking Changes 🚨

• Adjusts the require-dbt-version to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.
  • For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
  • For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
• Upgrades the package dependency to refer to the latest dbt_fivetran_utils. The latest dbt_fivetran_utils package also has a dependency on dbt_utils [">=0.8.0", "<0.9.0"].
  • Please note, if you are installing a version of dbt_utils in your packages.yml that is not in the range above then you will encounter a package dependency error.

Breaking Changes: New Campaign Suppression Variable

Happy Friday! 🕺

This release streamlines the process for using the deprecated spelling of campaign_list_suppression_history. Instead of changing both the campaign_suppression_list_history and using_old_spelling variables, users can now just provide the spelling they would like to use to the new iterable__campaign_suppression_list_history_table variable. Details can be found in the README.

dbt 0.20.0 Compatibility

🎉 dbt 0.20.0 Compatibility 🎉

🚨 This is a breaking change! 🚨 dbt v0.20.0 or greater is required for this release. If you are not ready to upgrade, consider using a previous release of this package.

Additional considerations when upgrading to this package:

• This package utilizes the latest v0.7.0 release of the dbt-labls/dbt_utils package. If your project also utilizes a version of the fishtown-analytics/dbt_utils package then you will receive a duplicate package error. If this is the case you will need to consider upgrading your other packages to be compatible with this update or use a previous release of this package.
• Similar to the above point, all previous Fivetran dbt packages utilize the fishtown-analytics/dbt_utils package and you will need to upgrade all Fivetran dbt packages to the latest dbt 0.20.0 compatibility releases in order for your packages to run without package conflicts.