Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(events tracking): add abstract class and logging implementation #80117

Draft
wants to merge 17 commits into
base: master
Choose a base branch
from
Draft

feat(events tracking): add abstract class and logging implementation #80117

wants to merge 17 commits into from
+116 −2

Conversation

victoria-yining-huang
Copy link
Contributor

@victoria-yining-huang victoria-yining-huang commented Oct 31, 2024 •
edited
Loading

design doc

need to track the completion of each stage, to 1) compute events conversion rates 2) enable debugging visibility into where events are being dropped

the usage will be heavily sampled to not blow up traffic

@github-actions github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 31, 2024
fpacifici
fpacifici reviewed Nov 1, 2024
View reviewed changes
src/sentry/utils/event_tracker.py Outdated Show resolved Hide resolved
src/sentry/utils/event_tracker.py Outdated
START = "start"
END = "end"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider making the enum an integer. That would take a lot less memory in redis.

The galaxy brain version would be to create a bitmap. One bit per phase. Each event only needs 12 bits to encode all phases. https://redis.io/docs/latest/develop/data-types/bitmaps/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not writing to redis with the logging implementation but i get the idea

lynnagara
lynnagara reviewed Nov 1, 2024
View reviewed changes
src/sentry/utils/event_tracker.py Outdated
"""
Records how far an event has made it through the ingestion pipeline.
"""
self.logger.info(f"EventTracker recorded event {event_id} - {status.value}")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you are building log analysis tools on top of this structured logging might make life easier

@victoria-yining-huang victoria-yining-huang requested a review from a team as a code owner November 1, 2024 23:20
@codecov Codecov
Copy link

codecov bot commented Nov 1, 2024 •
edited
Loading

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
22959 1 22958 214
View the top 1 failed tests by shortest run time 
tests.sentry.ingest.ingest_consumer.test_ingest_consumer_processing test_transactions_save_event_transaction_is_tracked
Stack Traces | 0.058s run time 
#x1B[1m#x1B[.../ingest/ingest_consumer/test_ingest_consumer_processing.py#x1B[0m:188: in test_transactions_save_event_transaction_is_tracked
    payload = get_normalized_event(event, default_project)
#x1B[1m#x1B[.../ingest/ingest_consumer/test_ingest_consumer_processing.py#x1B[0m:55: in get_normalized_event
    mgr = EventManager(data, project=project)
#x1B[1m#x1B[31msrc/sentry/event_manager.py#x1B[0m:383: in __init__
    grouping_config = get_grouping_config_dict_for_project(self._project)
#x1B[1m#x1B[31m.venv/lib/python3.12.../site-packages/sentry_sdk/tracing_utils.py#x1B[0m:698: in func_with_tracing
    return func(*args, **kwargs)
#x1B[1m#x1B[.../sentry/grouping/api.py#x1B[0m:148: in get_grouping_config_dict_for_project
    return loader.get_config_dict(project)
#x1B[1m#x1B[.../sentry/grouping/api.py#x1B[0m:73: in get_config_dict
    "enhancements": self._get_enhancements(project),
#x1B[1m#x1B[.../sentry/grouping/api.py#x1B[0m:80: in _get_enhancements
    enhancements_base = CONFIGURATIONS[config_id].enhancements_base
#x1B[1m#x1B[31mE   KeyError: <MagicMock name='is_sampled_to_track.get_option()' id='140534966318528'>#x1B[0m

To view more test analytics, go to the Test Analytics Dashboard
Got feedback? Let us know on Github

@victoria-yining-huang victoria-yining-huang marked this pull request as draft November 5, 2024 18:37
evanpurkhiser
evanpurkhiser reviewed Nov 5, 2024
View reviewed changes
src/sentry/utils/event_tracker.py Outdated
Records how far an event has made it through the ingestion pipeline.
"""
if is_tracked:
extra = {"event_id": event_id, "status": status}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to do EventStageStatus.value to get the string value from the enum.

Alternatively you can use StrEnum which is probably a better approach

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lynnagara lynnagara lynnagara left review comments

@fpacifici fpacifici fpacifici left review comments

@evanpurkhiser evanpurkhiser evanpurkhiser left review comments

@markstory markstory markstory left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Scope: Backend Automatically applied to PRs that change backend components Scope: Frontend Automatically applied to PRs that change frontend components
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@victoria-yining-huang @markstory @evanpurkhiser @fpacifici @lynnagara