Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-hubspot-native: add auto backfills for delayed streams #2069

Merged
merged 1 commit into from
Oct 18, 2024

Conversation

Alex-Bair
Copy link
Contributor

@Alex-Bair Alex-Bair commented Oct 18, 2024

Description:

Delayed incremental streams should be wrapped in a try/except block to trigger automatic backfills if we encounter a MustBackfillBinding error.

I kept the recent incremental streams wrapped in the existing try/except that serves the same purpose since it's possible for MustBackfillBinding exceptions to be raised if a task is disabled long enough for the cursor to fall far enough behind the recent changes available from HubSpot.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

Tested on a local stack. Confirmed that auto backfills are triggered if the recent incremental engagements stream falls far enough behind by shifting the cutoff date back ~60 days. I didn't confirm that the same happens for delayed incremental streams since finding the right cutoff shift where the delayed stream raises a MustBackfillBinding exception but not the recent stream would some sifting through API responses. Since the try/except wrapper for both recent & delayed streams are the same, and the wrapper works for the recent stream, I'm fairly confident it'll work for the delayed stream.


This change is Reviewable

…l streams

Delayed incremental streams should be wrapped in a `try/except` block to
trigger automatic backfills if we encounter a `MustBackfillBinding`
error.

I kept the recent incremental streams wrapped in the `try/except` that
serves the same purpose since it's possible for `MustBackfillBinding`
exceptions to be raised if a task is disabled long enough for the
cursor to fall far enough behind the recent changes available from
HubSpot.
@Alex-Bair Alex-Bair added the change:unplanned Unplanned change, useful for things like doc updates label Oct 18, 2024
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Alex-Bair Alex-Bair merged commit 9aa980f into main Oct 18, 2024
72 of 77 checks passed
@Alex-Bair Alex-Bair deleted the bair/hubspot-native-engagements-limit-backfill branch October 18, 2024 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
change:unplanned Unplanned change, useful for things like doc updates
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants