Apply filters to a Hugging Face dataset to avoid repeating all varian…

…ts. (#719) The only issue for now is that `regex` is a regex while `includes` is a glob... So I use heuristics to convert from one to another. I think it's not a problem for hugging face datasets as we control the form they have. But it can be challenging to have a generic conversion. The best would be to use either regular expressions or glob patterns everywhere.
mlcommons · Jul 19, 2024 · 6fc0adb · 6fc0adb
1 parent eaf6d61
commit 6fc0adb
Show file tree

Hide file tree

Showing 7 changed files with 35 additions and 404 deletions.
diff --git a/datasets/0.8/huggingface-c4/metadata.json b/datasets/0.8/huggingface-c4/metadata.json
diff --git a/datasets/0.8/huggingface-c4/output/en.jsonl b/datasets/0.8/huggingface-c4/output/en.jsonl