-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
materialize-redshift: compress checkpoints
Adds compression for checkpoints that are stored in the checkpoints table, which should significantly increase the maximum size of checkpoint the 1 MiB VARBYTE column can hold. VARBYTE columns can also be made larger (with added query overhead), so if this still ends up not being enough we can add automatic enlargement of the VARBYTE columns, similar to what we have for VARCHAR columns in Redshift already. The changes in this commit really lean into Redshift handling metadata in its own weird way. Previously the general "install fence" routine was modified to all a decoding callback which was only used by Redshift. Rather than add an additional encoding callback that only Redshift would use, we'll just have Redshift do its own thing entirely and simplify the general case for all other materializations. Having more direct control over the query construction for Redshift has the added benefit of not needing to do a separate Hex decode within the connector, since we can query the known encoding directly.
- Loading branch information
1 parent
206e4d9
commit 946e2d1
Showing
10 changed files
with
254 additions
and
98 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.