Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add whats new and upgrade guide #14996

Merged
merged 20 commits into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
ef46bba
Add whats new and upgrade guide
jlowin Aug 19, 2024
4b02011
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
94a2451
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
9ce7b8b
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
9de8dff
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
8b9cadb
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
9d7fd39
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
455bed0
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
946665c
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
8593ba2
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
c670cd2
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
5b12378
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
220ce82
Update docs/3.0rc/resources/upgrading-to-prefect-3.mdx
aaazzam Aug 21, 2024
a1e7a15
Update docs/3.0rc/resources/whats-new-prefect-3.mdx
aaazzam Aug 21, 2024
afbdbed
Rename upgrading-to-prefect-3.mdx to upgrade-to-prefect-3.mdx
discdiver Aug 21, 2024
3154ff7
Merge branch 'main' into upgrade
discdiver Aug 21, 2024
0ed1138
Merge branch 'main' into upgrade
discdiver Aug 22, 2024
8a3a6bd
Apply suggestions from code review
jlowin Aug 23, 2024
29d3d1a
Update doc
jlowin Aug 23, 2024
d11f86c
Merge branch 'main' into upgrade
jlowin Aug 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/3.0rc/get-started/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ if __name__ == "__main__":
<Card title="Prefect Cloud" icon="cloud" href="/3.0rc/manage/cloud/index">
Supercharge Prefect with enhanced governance, security, and performance capabilities.
</Card>
<Card title="Upgrade to Prefect 3" icon="sparkles" href="/3.0rc/resources/upgrade-prefect-3">
<Card title="Upgrade to Prefect 3" icon="sparkles" href="/3.0rc/resources/upgrade-to-prefect-3">
jlowin marked this conversation as resolved.
Show resolved Hide resolved
Upgrade from Prefect 2 to Prefect 3 to get the latest features and performance enhancements.
</Card>
</CardGroup>
Expand Down
75 changes: 75 additions & 0 deletions docs/3.0rc/resources/upgrade-to-prefect-3.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: Upgrading to Prefect 3
description: Learn how to upgrade from Prefect 2 to Prefect 3.
---

Prefect 3 introduces exciting new features and improvements while maintaining compatibility with most Prefect 2 workflows. For the majority of users, upgrading to Prefect 3 will be a seamless process that requires few or no code changes. This guide highlights key changes that you may need to consider when upgrading.

## Quickstart

To upgrade to Prefect 3, run:

```bash
pip install prefect>=3.0.0rc1 --pre
```

Note that the `--pre` flag is required to install the release candidate until the final release is available.


If you self-host a Prefect server, run this command to update your database:

```bash
prefect server database upgrade
```

If you use a Prefect integration or extra, remember to upgrade it as well. For example:

```bash
pip install prefect[aws]>=3.0.0rc1 --pre
```

## Upgrade notes

### Pydantic V2

<Info>
This change affects you if: You use custom Pydantic models with Prefect features.
</Info>

Prefect 3 is built with Pydantic 2 for improved performance. All Prefect objects will automatically upgrade, but if you use custom Pydantic models for flow parameters or custom blocks, you'll need to ensure they are compatible with Pydantic 2. You can continue to use Pydantic 1 models in your own code if they do not interact directly with Prefect.

Refer to [Pydantic's migration guide](https://docs.pydantic.dev/latest/migration/) for detailed information on necessary changes.

### Module location and name changes

Some less-commonly used modules have been renamed, reorganized, or removed for clarity. The old import paths will continue to be supported for 6 months, but emit deprecation warnings. You can look at the [deprecation code](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/_internal/compatibility/migration.py) to see a full list of affected paths.

### Async behavior

<info>
This change affects you if: you use advanced asynchronous behaviors in your flows.
</info>

Prefect 3 makes a few changes to handling async There are three key changes to be aware of:

1. **Async Tasks**: In Prefect 2, it was possible to call native `async` tasks from synchronous flows, a pattern that is not normally supported in Python. Prefect 3 removes this ability to reduce complexity and potential issues. If you relied on asynchronous tasks in synchronous flows, you must either make your flow asynchronous or use a task runner that supports asynchronous execution.

2. **PrefectFutures**: PrefectFutures now have a synchronous interface instead of an asynchronous one.

3. **Future Resolution in Flows**: In Prefect 2, futures were fire-and-forget and the flow would automatically block until they completed. In Prefect 3, you must explicitly wait for futures to complete before exiting a flow. This change ensures that all tasks are complete before the flow finishes.

### Caching behavior

<info>
This change affects you if: You rely on side effects in your tasks
</info>

Prefect 3 introduces a powerful idempotency engine. By default, tasks in a flow run are automatically cached if they are called more than once with the same inputs. If you rely on tasks with side effects, this may result in surprising behavior. To disable caching, pass `cache_policy=NONE` to your task.

### Workers

<info>
This change affects you if: You're using agents from an early version of Prefect 2.
</info>

In Prefect 2, agents were deprecated in favor of next-generation workers. Workers are now standard in Prefect 3. For detailed information on upgrading from agents to workers, please refer to our [upgrade guide](https://docs-3.prefect.io/3.0rc/resources/upgrade-agents-to-workers).
46 changes: 46 additions & 0 deletions docs/3.0rc/resources/whats-new-prefect-3.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
title: What's New in Prefect 3
jlowin marked this conversation as resolved.
Show resolved Hide resolved
sidebarTitle: What's New
jlowin marked this conversation as resolved.
Show resolved Hide resolved
---

Prefect 3 represents a significant leap forward in workflow orchestration, bringing a host of new features, performance improvements, and expanded capabilities to enhance your data engineering experience. Let's explore the exciting new additions and enhancements in this release.

Most Prefect 2 users can upgrade without changes to their existing workflows. Please review the [upgrade guide](/3.0rc/resources/upgrading-to-prefect-3) for more information.
jlowin marked this conversation as resolved.
Show resolved Hide resolved

## Open-Source Events and Automation System
jlowin marked this conversation as resolved.
Show resolved Hide resolved

One of the most anticipated features in Prefect 3 is the introduction of the events and automation system to the open-source package. Previously exclusive to Prefect Cloud, this powerful system now allows all users to create sophisticated, event-driven workflows.

With this new capability, you can trigger actions based on specific event payloads, cancel runs if certain conditions aren't met, or automate workflow runs based on external events. For instance, you could initiate a data processing pipeline automatically when a new file lands in an S3 bucket. The system also enables you to receive notifications for various system health events, giving you greater visibility and control over your workflows.

## New transactional interface

Another major addition in Prefect 3 is the new transactional interface. This powerful feature makes it easier than ever to build resilient and idempotent pipelines. With the transactional interface, you can group tasks into transactions, automatically roll back side effects on failure, and significantly improve your pipeline's idempotency and resilience.

For example, you can define rollback behaviors for your tasks, ensuring that any side effects are cleanly reversed if a transaction fails. This is particularly useful for maintaining data consistency in complex workflows involving multiple steps or external systems.

## Flexible task execution

Prefect 3 has no restrictions on where tasks can run. Tasks can be nested within other tasks, allowing for more flexible and modular workflows; they can also be called outside of flows, essentially enabling Prefect to function as a background task service. You can now run tasks autonomously, apply them asynchronously, or delay their execution as needed. This flexibility opens up new possibilities for task management and execution strategies in your data pipelines.


## Enhanced client-side engine

Prefect 3 comes with a thoroughly reworked client-side engine that brings several improvements to the table. You can now nest tasks within other tasks, adding a new level of modularity to your workflows. The engine also supports generator tasks, allowing for more flexible and efficient handling of iterative processes.

One of the most significant changes is that all code now runs on the main thread by default. This change improves performance and leads to more intuitive behavior, especially when dealing with shared resources or non-thread-safe operations.

## Improved artifacts and variables

Prefect 3 enhances the artifacts system with new types, including progress bars and image artifacts. These additions allow for richer, more informative task outputs, improving the observability of your workflows.

The variables system has also been upgraded to support arbitrary JSON, not just strings. This expansion allows for more complex and structured data to be stored and retrieved as variables, increasing the flexibility of your workflow configurations.

## Workers

Workers were first introduced in Prefect 2 as next-generation agents, and are now standard in Prefect 3. Workers offer a stronger governance model for infrastructure, improved monitoring of jobs and work pool/queue health, and more flexibility in choosing compute layers, resulting in a more robust and scalable solution for managing the execution of your workflows across various environments.

## Performance enhancements

Prefect 3 doesn't just bring new features; it also delivers significant performance improvements. Users running massively parallel workflows on distributed systems such as Dask and Ray will notice substantial speedups. In some benchmark cases, we've observed up to a 98% reduction in runtime overhead. These performance gains translate directly into faster execution times and more efficient resource utilization for your data pipelines.

13 changes: 7 additions & 6 deletions docs/mint.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@
"url": "https://communityinviter.com/apps/prefect-community/prefect-community"
},
{
"icon": "link",
"name": "Prefect.io",
"url": "https://www.prefect.io/"
"icon": "cloud",
"name": "Prefect Cloud",
"url": "https://app.prefect.cloud"
}
],
"api": {
Expand Down Expand Up @@ -52,6 +52,7 @@
"group": "Get started",
"pages": [
"3.0rc/get-started/index",
"3.0rc/resources/whats-new-prefect-3",
"3.0rc/get-started/install",
"3.0rc/get-started/quickstart"
],
Expand Down Expand Up @@ -190,7 +191,7 @@
{
"group": "Resources",
"pages": [
"3.0rc/resources/upgrade-prefect-3",
"3.0rc/resources/upgrading-to-prefect-3",
jlowin marked this conversation as resolved.
Show resolved Hide resolved
"3.0rc/resources/upgrade-agents-to-workers",
"3.0rc/resources/cancel",
"3.0rc/resources/visualize-flow-structure",
Expand Down Expand Up @@ -1267,8 +1268,8 @@
}
],
"topbarCtaButton": {
"name": "TRY PREFECT CLOUD",
"url": "https://app.prefect.cloud/"
"type": "github",
"url": "https://github.com/PrefectHQ/Prefect"
},
"versions": [
"3.0rc"
Expand Down