Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: save CKAN metadata; output second ZIP #10

Merged
merged 12 commits into from
Sep 11, 2024
Merged

feat: save CKAN metadata; output second ZIP #10

merged 12 commits into from
Sep 11, 2024

Conversation

simon-20
Copy link
Contributor

  • Fix bug where Last-Modified headers weren't converted to datetime
  • Altered provisioning script to turn CDN caching off
  • Put wait time between runs into environment variable
  • Handle unexpected missing Azure downloads gracefully
  • Save CKAN publisher, dataset metadata
  • Output second ZIP file, legacy format, for backwards compatibility

Resolves #4

@simon-20
chore: update yoyo migrations; add VS Code helper
a141f4c
@simon-20
chore: add and tidy config vars
465f853
@simon-20
feat: turn caching on CDN off
10f865b
@simon-20
chore: env var for wait time; urblib warnings off
398cd8e
@simon-20
test: update test env
7e4a898
@simon-20
test: tests for utility functions
2d47299
@simon-20
test: tests for fetching, storing CKAN metadata
3377c9d
@simon-20
test: tests for new ZIP file creation
63040d7
@simon-20
chore: utility functions for new features
dc39f9c
@simon-20
feat: store CKAN metadata
bbdabd6 
* Database migrations
* Download CKAN publisher, dataset metadata
* Save to DB
* Redone indexing code to avoid including legacy metadata in JSON index
@simon-20
feat: second ZIP file format
aa188d5 
* Added code to make outputting of new ZIP file formats easier
* Implemented a second ZIP format, which is stripped down version of legacy Data Dump
@simon-20
chore: update release targets ready for release
d8dd0c6
tillywoodfield
tillywoodfield approved these changes Sep 10, 2024
View reviewed changes
Copy link

@tillywoodfield tillywoodfield left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super familiar with the bulk data service logic prior to this, but everything here looks reasonable to me :)

@simon-20 simon-20 merged commit 1aac6df into develop Sep 11, 2024
1 check passed
@simon-20 simon-20 deleted the multi_zip__cdn_change branch September 18, 2024 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tillywoodfield tillywoodfield tillywoodfield approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

App crashes when it can't download a file it expects to exist from blob storage
2 participants
@simon-20 @tillywoodfield