-
Notifications
You must be signed in to change notification settings - Fork 5
/
CHANGELOG
255 lines (196 loc) · 13 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
snowplow-unified 0.5.0 (2024-10-14)
---------------------------------------
## Summary
This release brings significant updates to support Apache Spark with the Iceberg file format, enhances browser context support, improves documentation, and refines language mapping and column tests.
## Features
- Add support for Apache Spark with Iceberg file format
- Introduce support for browser_context_2
- Implement unified language mapping for Android and iOS to produce a common language name
- Enhance documentation for previously undocumented fields in derived tables
## Under the hood
- Modify integration tests to support Spark
- Adjust column tests for conditional scenarios
## Upgrading
Update the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.4.5 (2024-07-30)
---------------------------------------
## Summary
This release extends support for those users who would like to use the Snowplow incremental base macros but are using specific older versions of the AMP tracker or the Pixel tracker which do not send dvce_created / dvce_sent_tstamp fields.
## Features
- Add support for allow_null_dvce_tstamps var
## Upgrading
Update the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.4.4 (2024-06-26)
---------------------------------------
## Summary
This release adds early-release support for running the models on a Snowflake lake loader Iceberg table. Please note this feature is actively in testing and not all features of the package may work as expected. Please [let us know](https://github.com/snowplow/dbt-snowplow-unified/issues/new/choose) if you find any issues.
## Features
- Add early release support for Snowflake Iceberg events tables via lake loader
# Optimizations
- Remove unused columns and attempt to improve window functions in user_sessions_this_run
## Under the hood
- Alter GH action to remove fail-fast on integration tests
## Upgrading
Update the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.4.3 (2024-06-24)
---------------------------------------
## Summary
This release brings a couple of optimizations and fixes: it fixes a syntax error when `snowplow__use_refr_if_mkt_null` is enabled, optimizes the test `snowplow_tests_view_in_session_values`, restructures the way `user_sessions_this_run` table is created to possibly help with query optimization to reduce processing time, and adds further initialization checks which are now tied to the `snowplow__enable_initial_checks` variable.
## Features
- Add more robust init tests
# Optimizations
- Move tests view in session values to this run (Close #56)
- Restructure user sessions this run
## Fixes
- Fix syntax when snowplow__use_refr_if_mkt_null enabled
## Upgrading
Update the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.4.2 (2024-06-12)
---------------------------------------
## Summary
This release introduces enhancements to the initialization checks and refinements in data handling, particularly in Databricks/Spark environments. These updates ensure smoother operations and better data integrity checks.
## Features
- Introduced a new validation test to halt the process early if all required seeds are not present in the data warehouse. This ensures all necessary data components are available before proceeding.
## Fixes
- Revised the field casting approach within Databricks/Spark to enhance data handling and compatibility.
- Removed the non-null constraint test on `user_identifier` to prevent unnecessary validation errors and allow for more flexible data integration.
## Upgrading
Update the snowplow-unified version in your `packages.yml` file to take advantage of these improvements.
snowplow-unified 0.4.1 (2024-05-13)
---------------------------------------
## Summary
This feature updates our `default_channel_group` definition to support the use of `refr_` fields in the case that the `mkt_` fields are null. This is a requested feature in particular for landing pages with redirects. This functionality is turned off by default.
## Features
- New `snowplow__use_refr_if_mkt_null` variable to use `refr_` fields if `mkt_` ones are null in default channel group classification
## Fixes
- Fix an issue in the channel group classification where direct channels were sometimes ignored due to string checks
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.4.0 (2024-03-25)
---------------------------------------
## Summary
This release adds a surrogate key to the conversions table, in case of an event being valid against multiple conversion types, and fixes an issue with bigquery if `snowplow` was in your project name.
## 🚨 Breaking Changes 🚨
- Adds a new surrogate key to the optional conversions table, to allow for the same event to be part of multiple conversions
## Fixes
- Fix an issue where having `snowplow` in your project name caused issue when using a bigquery target
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file. If you already make use of the conversions table and wish to not do a full refresh, you can add the new column by following the migration guide [here](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/#upgrading-to-040).
snowplow-unified 0.3.1 (2024-03-11)
---------------------------------------
## Summary
This release fixes an issue where it was not possible to full refresh a single table using the `models_to_remove` variable, as well as removing a non-valid tests on the conversions table.
## Fixes
- Fix missing argument in `snowplow_utils.snowplow_delete_from_manifest` call
- Remove null test on `user_id` in conversions table
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.3.0 (2024-02-26)
---------------------------------------
## Summary
This release adds one major new feature, which is custom aggregations on the views, sessions, and users tables. You can read more about it in our docs [here](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/custom-aggregations/). We also added the ability to manage grants to all tables in the package via the `snowplow__grant_select_to` variable.
Under the hood we did a lot of small tweaks and improvements including prefixing all macro calls for easier custom models, moved cluster by fields to macros, ensure that the manifest tables are only full refreshed when `snowplow__allow_refresh` is set to true AND there is a full refresh flag on the run, and added a few context fields to the derived tables that were being discarded previously.
## 🚨 Breaking Changes 🚨
- We have changed the behavior of the `allow_refresh` macro so now if `snowplow__allow_refresh` is set to `true` it will only refresh the manifest models if the `--full-refresh` flag is also set. If you require the old behavior where it would refresh the manifest models on an incremental run when `snowplow__allow_refresh` was set to `true`, please overwrite this macro. See the [Overriding Macros](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/macros-and-keys/#overriding-macros) guide for more details.
- Renamed `snowplow__page_view_passthroughs` to `snowplow__view_passthroughs` to be consistent with the rest of the package
- Minimum `snowplow-utils` version is now 0.16.2
## Features
- Add new passthrough aggregations to the views, sessions, and users table, enabled using `snowplow__view/session/user_aggregations`
- Reorder and add some additional context fields to derived tables (non-breaking change)
- Add `snowplow__custom_sql` to allow adding custom sql to the `snowplow_unified_base_events_this_run` and `snowplow_unified_events_this_run` models
- Add macro to define cluster-by for tables to allow users to overwrite this if required
- Add check for `--full-refresh` flag before allowing refresh of manifest models when `snowplow__allow_refresh` is set to `true`.
- Add ability to grant select to a list of users, principals or roles on tables created by the package using `snowplow__grant_select_to` for all warehouses except BigQuery
- Add auto-grant of `usage` on schemas to `snowplow__grant_select_to`, can be disabled using `snowplow__grant_schema_usage` (see docs at https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/package-features/table-grants/)
## Fixes
- Fix a bug where if you ran the package in a period with no data, and had list all events enabled, the package would error rather than complete
- Fix incorrect tagging in app errors module tables
## Under the hood
- Prefix all macro calls with package name for easier customization
- Use macros for grouped fields (e.g. contexts) where possible
- Bump actions version numbers
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file, paying attention to the breaking changes above..
snowplow-unified 0.2.0 (2024-01-30)
---------------------------------------
## Summary
This release adds the ability to calculate mobile screen engagement using the screen summary context. There is also a new optional module for a conversions table. Other changes are the ability to stitch the users table during session stitching and heatset is a recognised platform now.
## 🚨 Breaking Changes 🚨
Existing users on Snowflake / Databricks / Redshift will need to make changes to some of their derived tables. For a full sql script on how to achieve this, check out the relevant [migration guide](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/). The other option is to do a [complete refresh](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.
## Features
- Add mobile screen engagement calculation using the screen summary context (#16)
- Adds user stitching to the users table (enabled with `snowplow__session_stitching`)
- Adds "headset" to the list of recognized platforms
- Add optional conversions module
## Fixes
- Consider screen view ID from the screen view context (#14)
- Fix link to incorrect FAQ in README
- Remove test for not null screen ID and name in app errors table
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.1.2 (2023-11-23)
---------------------------------------
## Summary
This is a patch release to fix the default browser context variable for warehouses other than redshift/postgres.
## Fixes
- Fix browser context
## Upgrading
Bump the snowplow-unified version in your `packages.yml` file.
snowplow-unified 0.1.1 (2023-11-21)
---------------------------------------
## Summary
This release is for supporting multiple versions of the session context schema for Bigquery (mobile) users not just contexts_com_snowplowanalytics_snowplow_client_session_1_0_0 out-of-the-box.
## Features
- Support latest session context schema
### Github
```yml
packages:
- git: "https://github.com/snowplow/dbt-snowplow-unified.git"
revision: 0.1.1
```
### dbt hub
```yml
packages:
- package: snowplow/snowplow_unified
version: [">=0.1.0", "<0.2.0"]
```
snowplow-unified 0.1.0 (2023-11-14)
---------------------------------------
## Summary
This is the first official release of the Snowplow Unified package, which contains a fully incremental model that transforms raw web and mobile event data generated by the Snowplow JavaScript and mobile trackers into a set of derived tables: views, sessions and users.
## Features
- dbt Package that processes web and mobile events simultaneously
- Support for Snowflake / BigQuery / Databricks / Redshift / Postgres
- optional modules such as consent, app errors and web performance (core web vitals)
Please note that this data model is under the Snowplow Personal & Academic License (SPAL). For further details please refer to our [documenation site](https://docs.snowplow.io/docs/contributing/personal-and-academic-license-faq/).
## Installation
To install the package, add the following to the `packages.yml` in your project:
### Github
```yml
packages:
- git: "https://github.com/snowplow/dbt-snowplow-unified.git"
revision: 0.1.0
```
### dbt hub
Please note that it may be a few days before the package is available on dbt hub after the initial release.
```yml
packages:
- package: snowplow/snowplow_unified
version: [">=0.1.0", "<0.2.0"]
```
snowplow-unified 0.0.1 (2023-10-24)
---------------------------------------
## Summary
This is the first release of the Snowplow Unified package, which contains a fully incremental model that transforms raw web and mobile event data generated by the Snowplow JavaScript and mobile trackers into a set of derived tables: views, sessions and users.
## Features
- dbt Package that processes web and mobile events simultaneously
- Support for Snowflake / BigQuery / Databricks / Redshift / Postgres
- optional modules such as consent, app errors and web performance (core web vitals)
## Installation
To install the package, add the following to the `packages.yml` in your project:
### Github
```yml
packages:
- git: "https://github.com/snowplow/dbt-snowplow-unified.git"
revision: 0.0.1
```