1.0.0

First stable release with independent RDB Loader and new Stream Shredder beta component.

• Discourse post
• Upgrade guide

New features

• Common: add events count to the shredding complete message (#376)
• Stream Shredder: add (#354)
• RDB Loader: add 2nd gen load manifest table (#366)

Dependency bumps

• Common: bump base-debian to 0.2.2 (#380)
• Common: bump pureconfig from 0.14.0 to 0.14.1 (#329)
• Common: bump aws-java-sdk from 1.11.916 to 1.11.990 (#370)
• Common: bump decline from 1.3.0 to 1.4.0 (#333)
• Common: bump snowplow-scala-tracker to 1.0.0 (#372)
• Common: bump kind-projector to 0.11.3 (#371)
• Common: bump sbt from 1.4.4 to 1.4.9 (#344)

API changes

• Common: merge good and bad output folders (#358)

Under the hood

• Common: use a single tag to publish all assets (#379)
• Common: extract shredding logic into common module (#355)
• RDB Loader: migrate to doobie (#367)
• RDB Shredder: shade cats in sbt-assembly (#382)

Bugfixes and improvements

• Common: remove compRows configuration parameter (#390)
• RDB Loader: skip empty folders (#357)
• RDB Shredder: check for incomplete folders asynchronously (#385)

0.18.2

Maintainance release.

• RDB Loader: fix consistency check truncation (#314)
• RDB Loader: bring pre-R34 logging format back (#301)
• RDB Loader: print list of shredded types if consistency_check is skipped (#303)
• RDB Loader: print version before loading (#302)

R35

Pubilc beta with deprecation of EmrEtlRunner and major changes in architecture.

• Upgrade guide
• Discourse announcement

Architecture/API changes

• RDB Loader: remove EmrEtlRunner support (#251)
• RDB Loader: add discovery through SQS stream (#234)
• RDB Loader: switch to SQS-only loading (#262)
• RDB Loader: swtich to a HOCON config (#250)
• RDB Shredder: swtich to a HOCON config (#256)
• RDB Shredder: add min and max timestamps of events for the batch to the SQS message with shredded types (#275)
• RDB Shredder: add archive discovery (#263)
• RDB Shredder: send shredding info to SQS when it's done (#200)
• Common: get rid of atomic-events folder (#183)

New features

• RDB Loader: integrate Sentry (#235)

Under the hood

• RDB Loader: build Docker image (#247)
• Common: extend copyright notice to 2021 (#287)
• Bump decline from 0.6.2 to 1.3.0 (#223)
• Bump aws-java-sdk from 1.11.319 to 1.11.916 (#244)
• Bump scalacheck from 1.14.0 to 1.14.3 (#243)
• Bump circe-yaml from 0.9.0 to 0.13.1 (#226)
• Bump snowplow-events-manifest from 0.2.0 to 0.3.0 (#224)
• Bump fs2-core from 2.4.4 to 2.4.6 (#222)
• Bump redshift-jdbc42-no-awssdk from 1.2.36.1060 to 1.2.51.1078 (#208)
• Bump kind-projector from 0.9.6 to 0.9.10 (#217)

R34

Bugfix release.

Bug fixes

• RDB Shredder: escape newlines and tabs (#238)

R33

Bugfix and performance improvements.

• Discourse announcement

Bugfixes

• Common: fix ambigrous empty string (#171)
• Common: create a separate modules root (#195)
• RDB Shredder: optimize DAG (#181)
• RDB Shredder: use single etl_tstamp for cross-batch deduplication (#180)
• Loader: mitigate exhausted input error (#203)
• Loader: fix ignored schema in metadata queries (#178)

Under the hood

• Common: integrate coveralls (#220)
• Common: integrate Snyk (#188)
• Common: integrate Scala Steward (#206)
• Common: switch to Github Actions (#189)
• Common: disable test in assembly (#202)
• Loader: switch from Free monad to cats-effect IO (#184)

Removed features

• Common: remove Processing Manifest (#186)
• RDB Loader: remove Postgres support (#191)
• Loader: drop legacy S3 paths (#198)

Dependency updates

• Common: bump snowplow-scala-analytics-sdk to 2.1.0 (#201)
• Common: bump snowplow-badrows to 2.1.0 (#199)
• Common: bump specs2-core to 4.10.5 (#197)
• Common: bump schema-ddl to 0.12.0 (#192)
• Common: bump sbt to 1.4.4 (#196)
• Common: bump Scala to 2.12 (#127)
• RDB Shredder: bump Spark to 3.0.1 (#101)
• RDB Loader: bump jsch to 0.1.55 (#177)

Release 32

Redshift automigrations.

• Discourse announcement
• Upgrade guide

New features

• Common: allow tabular blacklisting (#156)
• RDB Shredder: add tabular data output (#151)
• RDB Shredder: use snowplow-badrows (#169)
• RDB Loader: add tabular output loading (#152)

Improvements and bugfixes

• RDB Shredder: fix false positive cross-batch deduplication for RT pipelines (#173)
• Common: force storage target ids to be valid UUID (#168)

Under the hood

• Common: remove VERSION file (#164)
• Common: bump iglu-scala-client to 0.6.2 (#167)
• Common: add Local Iglu Server to CI/CD process (#157)
• RDB Shredder: bump to 0.16.0 (#154)
• RDB Loader: bump redshift-jdbc to 1.2.36 (#61)
• RDB Loader: bump to 0.17.0 (#153)

Release 31

New bad rows format and orphan events bug fix.

• Blog post

Improvements and bugfixes

• RDB Shredder: persist synthetic duplicates on disk (#142)
• RDB Shredder: truncate string values exceeding column's limits (#143)
• RDB Loader: fix non-exhaustive match in data discovery (#117)
• RDB Loader: retry on refused connection (#8)

Under the hood

• Common: add Snowplow Maven resolver (#147)
• Common: extend copyright notice to 2019 (#144)
• Common: bump sbt-assembly to 1.4.9 (#135)
• Common: bump SBT to 1.2.8 (#134)
• Common: bump Travis oraclejdk version to 11 (#146)
• Common: bump Scala to 2.11.12 (#136)
• Common: switch to OpenJDK (#159)
• Common: bump release-manager to 0.4.1 (#132)
• RDB Shredder: bump to 0.15.0 (#139)
• RDB Shredder: factor out scala-common-enrich (#138)
• RDB Shredder: replace real DynamoDB with local version in CrossBatchDeduplicationSpec (#149)
• RDB Shredder: bump Spark to 2.3.2 (#162)
• RDB Loader: bump to 0.16.0 (#140)
• RDB Loader: bump iglu-scala-client to 0.6.0 (#141)
• RDB Loader: bump snowplow-scala-tracker to 0.6.1 (#148)

Release 30

Multiple usability and stability improvements.

• Blog post

Common

• Add VERSION file (#82)
• Bump SBT to 1.1.6 (#89)
• Use processing manifest (#81)
• Fix README links (#109)
• Extend copyright notice to 2018 (#91)

RDB Shredder

• Bump to 0.14.0 (#96)
• Use RDB Loader AWS SDK (#94)
• Remove auto-creation of event manifests table (#62)

RDB Loader

• Bump to 0.15.0 (#100)
• Improve log output (#23)
• Escape input for sanitize function (#87)
• Remove unused configuration properties (#95)
• Add ability to skip all load manifest interactions (#97)
• Make SSL configuration compatible with native JDBC settings (#73)
• Improve "no data discovered" error message (#69)
• Tolerate deleted folder artifacts when consistency_check is skipped (#80)
• Fix manifest population assumes load is most recent (#70)
• Fail load if entry exists in manifest (#14)
• Bump AWS SDK to 1.11.319 (#79)
• Bump snowplow-scala-tracker to 0.5.0 (#57)
• Bump specs2 to 4.0.4 (#90)
• Bump cats to 1.1.0 (#77)
• Fix typo in load_succeeded schema (#108)

Release 29

Hotfilx RDB Shredder release

• Blog post

Bug fixes

• RDB Shredder: align PostgresConstraints with atomic.events 0.10.0 (#103)

Other

• RDB Shredder: bump to 0.13.1 (#105)
• RDB Shredder: bump scala-common-enrich to 0.32.0 (#99)

Release 28

First release with umbrella-version, adding SSH-tunnels and some minor improvements.

• Blog post

New features

• RDB Loader: add support for SSH tunnels (#22)
• RDB Loader: allow JDBC credentials to be stored in EC2 parameter store (#19)

Testing and deployment

• Common: add CI/CD (#55)
• Common: remove AWS Java SDK shading (#54)
• RDB Shredder: add Snowplow and Clojars resolvers (#56)

UX-improvements

• RDB Loader: make logkey optional (#35)
• RDB Loader: make loading shredded data always required (#29)
• RDB Shredder: overwrite output datasets (#41)

Under the hood

• RDB Shredder: bump Spark to 2.2.0 (#52)
• RDB Shredder: bump to 0.13.0 (#49)
• RDB Shredder: bump scala-common-enrich to 0.27.0 (#39)
• RDB Loader: bump sbt-assembly to 0.14.5 (#51)
• RDB Loader: bump SBT to 0.13.16 (#50)
• RDB Loader: bump AWS SDK to 1.11.208 (#48)
• RDB Loader: bump redshift-jdbc to 1.2.8.1005 (#40)

Bug fixes

• RDB Loader: execute manifest insert in same transaction as load (#36)
• RDB Loader: remove tracking from dry run (#42)