examples

examples/

This folder holds a variety of examples and proof-of-concepts using Flow.

Using these examples

These examples are written as Flow YAML specifications, and demonstrate how to manage Flow data pipelines using files checked into a git repository. You would typically use flowctl catalog test and flowctl catalog publish in order to test and publish YAML specifications. But you would need to change the names to use a prefix that you have admin access to in order to actually run these yourself.

Note that these examples are tested using the flowctl-go CLI, not the typical flowctl that users typically interact with. But all the specs here are the same as what you'd use with regular flowctl. The main difference is that flowctl will run tests on the server side and perform authorization checks (which will fail because users don't have access to these prefixes in the Estuary control-plane).

Running All Tests

Flow makes it easy to write tests that verify the end-to-end behaviors of catalog collections. We recommend that every Flow catalog include tests, and these examples are no exception:

$ flowctl-go test --source examples/flow.yaml

You can also directly test catalog sources which are hosted remotely:

$ flowctl-go test --source https://raw.githubusercontent.com/estuary/flow/master/examples/all.flow.yaml

Examples

• bank/ is the example from the Derivations Concepts documentation
• citi-bike/ is a comprehensive example using Citi Bike system data.
• stock-stats/ models per-day market security statistics that update with ticks.
• temp-sensors/ shows how to do some basic aggregations, like min, max, and average.

Reference

• derive-patterns/ demonstrates common patterns and approaches in building derivations.
• reduction-types/ discusses reduction annotations available in Flow.