Skip to content

Commit

Permalink
Arrow support is now stable and ready for production (#43)
Browse files Browse the repository at this point in the history
* Remove the experimental warning from the external formats page
* Remove the experimental settings section from the hyper_process page as Arrow
 was the only setting mentioned there.
* Add to the release notes
* Remove broken link from release notes
  • Loading branch information
PerFuchs authored Jul 20, 2023
1 parent fdf100a commit c578a03
Show file tree
Hide file tree
Showing 3 changed files with 5 additions and 16 deletions.
8 changes: 2 additions & 6 deletions website/docs/hyper-api/hyper_process.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ unsupported version 2 (max supported version: 1). To open this database,
please update your product. (error code 0AS01)"
:::

### Experimental Settings {#experimentalsettings}
<!-- ### Experimental Settings {#experimentalsettings}
:::warning
This section describes pre-release features that are not supported and
Expand All @@ -267,8 +267,4 @@ We also encourage you to share your feedback about experimental
features in our [slack space](https://join.slack.com/t/tableau-datadev/shared_invite/zt-1q4rrimsh-lHHKzrhid1MR4aMOkrnAFQ).
:::
These settings control experimental features of Hyper:

`experimental_external_format_arrow`

: Enables scanning of [Arrow files and streams](/docs/sql/external/formats#external-format-arrow).
These settings control experimental features of Hyper: -->
6 changes: 3 additions & 3 deletions website/docs/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ In case you are wondering why all our releases start with `0.0`, read [this FAQ
* The functions `EXTRACT`, `date_part` and `date_trunc` have the following fiscal calendar fields: `fiscal_week`, `fiscal_month`, `fiscal_quarter`, and `fiscal_year`.
* Fiscal calendar options for the newly-added fields were added. See [Fiscal Calendar Options](/docs/sql/scalar_func/datetime#fiscal-calendar-options).
* Updated OpenSSL version from 1.1.1t to 1.1.1u.
* Support for reading and writing [Arrow](/docs/sql/external/formats#external-format-arrow) is now stable and ready for use in production

### v0.0.17360 [July 5, 2023]

Expand Down Expand Up @@ -166,9 +167,8 @@ In case you are wondering why all our releases start with `0.0`, read [this FAQ

Improved external file format support (CSV & Apache Parquet): Now you can use Hyper as a SQL
query engine directly on top of open formats and data lakes.
* Hyper now has experimental support for reading external data directly from Amazon S3. You need to enable the
[experimental_external_s3](/docs/hyper-api/hyper_process#experimentalsettings)
setting to use this feature and be aware that it can **change or be removed at any time without prior notice**.
* Hyper now has experimental support for reading external data directly from Amazon S3. ~~You need to enable the experimental_external_s3
setting to use this feature and be aware that it can **change or be removed at any time without prior notice**.~~ The feature is considered stable as of v0.0.15735. The experimental flag is not necessary anymore.

Hyper's S3 capabilities are highly optimized
(using techniques such as concurrent requests, request hedging and prefetching). For maximum performance,
Expand Down
7 changes: 0 additions & 7 deletions website/docs/sql/external/formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,13 +371,6 @@ production workloads yet.

## Apache Arrow Format {#external-format-arrow}

:::warning Experimental Feature
Hyper's support for the Arrow format is still in an early state and
should be considered experimental. We recommend to not use it in
production workloads yet. Set the `experimental_external_format_arrow` [setting](/docs/hyper-api/hyper_process#experimentalsettings)
to `true` to activate it.
:::

The `FORMAT => 'arrowfile'` or `FORMAT => 'arrowstream'` option enables reading the columnar binary format [Apache Arrow](https://arrow.apache.org/).
As data is stored in a columnar format, Hyper does not need to
read the whole file if only a subset of columns is selected. Hyper can read only
Expand Down

0 comments on commit c578a03

Please sign in to comment.