Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Harvest dataservices #3029

Merged
merged 72 commits into from
Jun 14, 2024
Merged

Harvest dataservices #3029

merged 72 commits into from
Jun 14, 2024

Commits on Apr 30, 2024

  1. Switch DCAT backend to not use one job for each dataset

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    9b35b43 View commit details
    Browse the repository at this point in the history

  2. Fix missing owner/org in new datasets

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    cb622a8 View commit details
    Browse the repository at this point in the history

  3. Remove prints

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    cdd5f01 View commit details
    Browse the repository at this point in the history

  4. Refactor using two functions

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    35ad714 View commit details
    Browse the repository at this point in the history

  5. Add back should_stop

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    76d99d8 View commit details
    Browse the repository at this point in the history

  6. Add back autoarchive and done with failed items

    @ThibaudDauce
    ThibaudDauce committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    e90ddcc View commit details
    Browse the repository at this point in the history

  7. Merge branch 'master' into sync_harvest_backend

    @ThibaudDauce
    ThibaudDauce authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    203b394 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. Always returns the graphs for debug

    @ThibaudDauce
    ThibaudDauce committed May 2, 2024
    Configuration menu
    Copy the full SHA
    5dd3c04 View commit details
    Browse the repository at this point in the history

  2. Add test for stopping due to HARVEST_MAX_ITEMS

    @ThibaudDauce
    ThibaudDauce committed May 2, 2024
    Configuration menu
    Copy the full SHA
    d8eaf45 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Update test backends

    @ThibaudDauce
    ThibaudDauce committed May 14, 2024
    Configuration menu
    Copy the full SHA
    ebf2af7 View commit details
    Browse the repository at this point in the history

  2. Fix some tests

    @ThibaudDauce
    ThibaudDauce committed May 14, 2024
    Configuration menu
    Copy the full SHA
    a2701af View commit details
    Browse the repository at this point in the history

  3. Revert ID change for FakeBackend

    @ThibaudDauce
    ThibaudDauce committed May 14, 2024
    Configuration menu
    Copy the full SHA
    a163b70 View commit details
    Browse the repository at this point in the history

  4. Simplify SyncBackend

    @ThibaudDauce
    ThibaudDauce committed May 14, 2024
    Configuration menu
    Copy the full SHA
    3693bd2 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Merge branch 'master' into sync_harvest_backend

    @maudetes
    maudetes authored May 21, 2024
    Configuration menu
    Copy the full SHA
    d5c0e29 View commit details
    Browse the repository at this point in the history

  2. fix wrong remote_id

    @ThibaudDauce
    ThibaudDauce committed May 21, 2024
    Configuration menu
    Copy the full SHA
    a71b46e View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Merge branch 'master' into sync_harvest_backend

    @ThibaudDauce
    ThibaudDauce authored May 23, 2024
    Configuration menu
    Copy the full SHA
    914b69d View commit details
    Browse the repository at this point in the history

  2. Remove dead code

    @ThibaudDauce
    ThibaudDauce committed May 23, 2024
    Configuration menu
    Copy the full SHA
    fc98e28 View commit details
    Browse the repository at this point in the history

  3. update comment

    @ThibaudDauce
    ThibaudDauce committed May 23, 2024
    Configuration menu
    Copy the full SHA
    20ce7e4 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Merge branch 'master' into sync_harvest_backend

    @ThibaudDauce
    ThibaudDauce authored May 27, 2024
    Configuration menu
    Copy the full SHA
    c3c4c27 View commit details
    Browse the repository at this point in the history

  2. Move docstring

    @ThibaudDauce
    ThibaudDauce committed May 27, 2024
    Configuration menu
    Copy the full SHA
    b9b41e2 View commit details
    Browse the repository at this point in the history

  3. Switch is_done() do its own function

    @ThibaudDauce
    ThibaudDauce committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0a6dfa5 View commit details
    Browse the repository at this point in the history

  4. Rename process_datasets method

    @ThibaudDauce
    ThibaudDauce committed May 27, 2024
    Configuration menu
    Copy the full SHA
    523c754 View commit details
    Browse the repository at this point in the history

  5. yield instead of callback

    @ThibaudDauce
    ThibaudDauce committed May 27, 2024
    Configuration menu
    Copy the full SHA
    064f8eb View commit details
    Browse the repository at this point in the history

  6. fix other backends

    @ThibaudDauce
    ThibaudDauce committed May 27, 2024
    Configuration menu
    Copy the full SHA
    263c3a3 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Merge branch 'master' into sync_harvest_backend

    @ThibaudDauce
    ThibaudDauce committed May 29, 2024
    Configuration menu
    Copy the full SHA
    6b6d11e View commit details
    Browse the repository at this point in the history

  2. Update changelog

    @ThibaudDauce
    ThibaudDauce committed May 29, 2024
    Configuration menu
    Copy the full SHA
    9124fc0 View commit details
    Browse the repository at this point in the history

  3. Remove unused process method

    @ThibaudDauce
    ThibaudDauce committed May 29, 2024
    Configuration menu
    Copy the full SHA
    e9b9af1 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Skip dataset without remote_id

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    318603d View commit details
    Browse the repository at this point in the history

  2. Harvest dataservices

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    e1b1bc0 View commit details
    Browse the repository at this point in the history

  3. Add more data inside dataservices

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    d536874 View commit details
    Browse the repository at this point in the history

  4. Revert hvd labels from rebase

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ce30a60 View commit details
    Browse the repository at this point in the history

  5. Fix tests

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    3e4d901 View commit details
    Browse the repository at this point in the history

  6. Remove duplicate code

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    ae02115 View commit details
    Browse the repository at this point in the history

  7. wip add printing graphs

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    345f05b View commit details
    Browse the repository at this point in the history

  8. Remove duplicate harvest update info

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    735f6d5 View commit details
    Browse the repository at this point in the history

  9. Add link between dataservices and datasets

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    b83d697 View commit details
    Browse the repository at this point in the history

  10. link dataset with dataservices

    @ThibaudDauce
    ThibaudDauce committed May 30, 2024
    Configuration menu
    Copy the full SHA
    29cede6 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Use forked XSLT

    @ThibaudDauce
    ThibaudDauce committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    2c7a24d View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    23d8111 View commit details
    Browse the repository at this point in the history

  2. fix merge issues

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    aa125aa View commit details
    Browse the repository at this point in the history

  3. Fix tests

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    d667b7b View commit details
    Browse the repository at this point in the history

  4. Improve some skips

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    e6816c9 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f630a25 View commit details
    Browse the repository at this point in the history

  6. Remove merge tag

    @ThibaudDauce
    ThibaudDauce committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    4eec493 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Separate metadata

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    f5c6917 View commit details
    Browse the repository at this point in the history

  2. Add comment

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    c9b9940 View commit details
    Browse the repository at this point in the history

  3. Add dataservice card to harvester admin

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    69ec784 View commit details
    Browse the repository at this point in the history

  4. Revert XSLT modifications

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    3403944 View commit details
    Browse the repository at this point in the history

  5. Remove some blanks lines

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    7c5b0ca View commit details
    Browse the repository at this point in the history

  6. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce authored Jun 5, 2024
    Configuration menu
    Copy the full SHA
    f5a5108 View commit details
    Browse the repository at this point in the history

  7. Update changelog

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    8e3e5ef View commit details
    Browse the repository at this point in the history

  8. Do not duplicate datasets on each harvesting

    @ThibaudDauce
    ThibaudDauce committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    9a9a43b View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    e94a23f View commit details
    Browse the repository at this point in the history

  2. Cleanup imports

    @ThibaudDauce
    ThibaudDauce committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    5fee125 View commit details
    Browse the repository at this point in the history

  3. Fix changelog

    @ThibaudDauce
    ThibaudDauce committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    31bfda7 View commit details
    Browse the repository at this point in the history

  4. Fix merge

    @ThibaudDauce
    ThibaudDauce committed Jun 10, 2024
    Configuration menu
    Copy the full SHA
    f477ef6 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce authored Jun 10, 2024
    Configuration menu
    Copy the full SHA
    8f910e6 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2024

  1. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce authored Jun 11, 2024
    Configuration menu
    Copy the full SHA
    aba27e4 View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: maudetes <[email protected]>
    @ThibaudDauce @maudetes
    ThibaudDauce and maudetes authored Jun 11, 2024
    Configuration menu
    Copy the full SHA
    9f9a7f8 View commit details
    Browse the repository at this point in the history

  3. add harvest metadata to API

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    2a0da24 View commit details
    Browse the repository at this point in the history

  4. harvest metadata as readonly

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    dd54158 View commit details
    Browse the repository at this point in the history

  5. Rename last_harvested_at and add harvest.created_at

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    30ba48d View commit details
    Browse the repository at this point in the history

  6. Fix wrong attribute

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b083c37 View commit details
    Browse the repository at this point in the history

  7. Do not empty the datasets list if no datasets found in harvesting

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    fbe6f22 View commit details
    Browse the repository at this point in the history

  8. fix casing

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    7863547 View commit details
    Browse the repository at this point in the history

  9. Fix tests

    @ThibaudDauce
    ThibaudDauce committed Jun 11, 2024
    Configuration menu
    Copy the full SHA
    b401700 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Save node id if it's an URL

    @ThibaudDauce
    ThibaudDauce committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    1d6714a View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Add landing page as remote_url

    @ThibaudDauce
    ThibaudDauce committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    cc6b5e8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into harvest_dataservices

    @ThibaudDauce
    ThibaudDauce authored Jun 13, 2024
    Configuration menu
    Copy the full SHA
    dc1a093 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Rename rdf_node_id_as_url to follow the same name as dataset :-(

    @ThibaudDauce
    ThibaudDauce committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    d484199 View commit details
    Browse the repository at this point in the history

  2. Remove dynamic to harvest dataservice metadata

    @ThibaudDauce
    ThibaudDauce committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    fa816cc View commit details
    Browse the repository at this point in the history

  3. Update udata/core/dataservices/models.py 

    Co-authored-by: maudetes <[email protected]>
    @ThibaudDauce @maudetes
    ThibaudDauce and maudetes authored Jun 14, 2024
    Configuration menu
    Copy the full SHA
    546c1be View commit details
    Browse the repository at this point in the history