Add predicate pushdown

[mike-luabase]

No description provided.

[mike-luabase]

@samansmink made the updates here. I tried getting rid of the data dir from the commit, but wasn't sure the best way to do that.

[mike-luabase]

Enhance query performance by filtering data at the metadata level, reducing the amount of data read during scans.

Key Changes

• Extended IcebergManifestEntry:

  • Added lower_bounds and upper_bounds maps to store column statistics.

• Utility Function:

  • Implemented IcebergUtils::GetFullPath to resolve file paths accurately.

• Metadata Retrieval:

  • Added GetEntries template in IcebergTable to fetch relevant manifest entries, excluding deleted ones.

• Predicate Evaluation:

  • Created EvaluatePredicateAgainstStatistics to assess if data files satisfy query predicates based on their statistics.
  • For each predicate:
    • Identifies the column involved.
    • Checks if the column has defined lower and upper bounds.
    • Based on the comparison type (e.g., =, >, <), determines if the predicate can be satisfied given the file's bounds.
    • If any predicate fails, the file is excluded from the scan.

• Scan Expression Modification:

  • Updated MakeScanExpression to filter data_file_entries using predicates before scanning.

• Binding Function Enhancements:

  • Enhanced IcebergScanBindReplace with additional logging and prepared data files based on predicate results.

This implementation optimizes Iceberg table scans by leveraging metadata for early data filtering, significantly improving query efficiency and resource usage.

[Mytherin]

Thanks for the PR! Some comments from my side. Could you also look at the failing CI?

[Mytherin]

Do we need to delete all of these files?

[Mytherin]

This code is generated - should this be modified?

[Mytherin]

All these format changes make the code hard to review - can we leave the old format in place?

[mike-luabase]

@Mytherin thanks for the review! I cleaned up the issues here: #78

will close this one.

[mike-luabase]

closing, see above.