Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extracting less data by Meltano as compared to data in Facebook UI #207

Open
MonikaRM opened this issue Jan 19, 2023 · 0 comments
Open

Extracting less data by Meltano as compared to data in Facebook UI #207

MonikaRM opened this issue Jan 19, 2023 · 0 comments

Comments

@MonikaRM
Copy link

Hello Team,
We are working on extracting data from Facebook API using the tap-facebook extractor. We were able to extract and load the data but when we are comparing the spend with the Facebook data, (we have a python file that is extracting the data directly from Facebook UI using the Facebook API) it's not matching with the data we have in the UI.
We are missing lot of data and would love some assistance. If someone has faced or can give suggestions on how to resolve this issue that would be great.
Context:
Before (up until mid-December) our queries were showing ~10,000 rows loaded per day and now we're down to ~250 rows loaded per day.
Going off of the Meltano status, all data should be loaded but we're not seeing that when we query the data.
We have used batch sizes: 50, 100, 1000, 3000, 4000, 6000, 8000, 10000 using the variable: max_batch_rows
The pipeline is currently the pipeline is running at an hourly basis.
Total rows
Facebook UI: 5073620
Meltano: 382000
We are using _sdc_batched_at to count the daily number of total rows.
Could this be causing an issue since _sdc_batched_at gets updated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant