Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load csv tables with pandas if pyarrow fails #450

Merged
merged 5 commits into from
Jul 12, 2024
Merged

Load csv tables with pandas if pyarrow fails #450

merged 5 commits into from
Jul 12, 2024

Conversation

hagenw
Copy link
Member

@hagenw hagenw commented Jul 11, 2024

Closes #449

Unfortunately, loading csv files with pyarrow.csv.read_csv() as introduced in #419 is not as tolerant to malformed csv files as pandas.read_csv(). I have identified so far three cases in which loading of a csv file might fail (two of them are listed in #449):

  1. Loading a csv file can fail, if the csv file contains more columns, as mentioned in the header of a database
  2. Loading a csv file can also fail, if it is very long and contains a lot of special characters, like ", "", ,. I did not added a test for it, because it turns out that the syntax of the csv file is correct, and it works when splitting the file into smaller ones.
  3. Loading of a csv file can fail, if it contain some offsets in there date values, e.g. +00:00, which is the case for some of our older datasets.

As pyarrow.csv.read_csv() cannot be easily extended to handle those cases, I use now a try-except statement, that falls back to loading the file with pandas.read_csv(). This is very unfortunate as it means when implementing a new feature (e.g. streaming) it needs to be implemented for both cases. But I don't know a better solution at the moment.

In principle, we could solve 1. and 3. by updating the databases, but you will still no longer be able to load old versions then, which is not acceptable. How we could solve 2. otherwise, I don't know.

@hagenw hagenw marked this pull request as draft July 11, 2024 11:40
Copy link

codecov bot commented Jul 11, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.0%. Comparing base (e3fd511) to head (6f53c6b).

Additional details and impacted files
Files Coverage Δ
audformat/core/table.py 100.0% <100.0%> (ø)

@hagenw hagenw marked this pull request as ready for review July 11, 2024 14:13
tests/test_table.py Outdated Show resolved Hide resolved
@hagenw hagenw merged commit 60cd1ed into main Jul 12, 2024
10 checks passed
@hagenw hagenw deleted the fix-csv-loading branch July 12, 2024 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parsing of CSV files with pyarrow.csv.read_csv() can fail
2 participants