Releases: PrefectHQ/prefect
Release 2.16.8
Release 2.16.7
💻 Introducing prefect shell
for observing CLI commands
You can now observe CLI commands as a Prefect flow. For example:
» prefect shell watch "curl http://wttr.in/Chicago?format=3"
17:32:39.562 | INFO | prefect.engine - Created flow run 'powerful-mushroom' for flow 'Shell Command'
17:32:40.171 | INFO | Flow run 'powerful-mushroom' - Chicago: ⛅️ +50°F
17:32:40.315 | INFO | Flow run 'powerful-mushroom' - Finished in state Completed()
See these docs to learn how to:
- run a shell command as a Prefect flow on-demand with
watch
- schedule a shell command as a recurring Prefect flow using
serve
See the PR for implementation details: #11998
Other Enhancements 🌟
- Integrate composite triggers with the
DeploymentTrigger
YAML representation — #12413 - Add JSON Artifacts — #12295
- Add auto-provisioning option for Cloud Run V2 push work pools — #12422
... and numerous 🐛 fixes!
Full Changelog: 2.16.6...2.16.7
See the release notes for more!
Release 2.16.6
Fix new behavior in typer 0.10.0
that broke the prefect
CLI
See the following pull request for implementation details:
- Handle new
typer.Option
behavior — #12398
All changes: 2.16.5...2.16.6
Release 2.16.5
Multi-select deletion of flow runs
It is now easier to bulk select and delete flow runs through the UI. Listings of filterable and selectable flow runs (e.g. on the flow runs, flow, and deployment pages) now include a top-level checkbox for (de)selecting all currently filtered flow runs for bulk deletion.
See the following pull request for implementation details:
Visualize state changes and artifacts in the UI
Additionally, the flow run graph UI enhancements for visualizing state changes and artifacts added in 2.16.4 are now enabled by default. See the release notes in 2.16.4 for more details!
Full Changelog: 2.16.4...2.16.5
See the release notes for more!
Release 2.16.4
Release 2.16.4
Flow Run Graph updates
The Flow Run Graph has been updated to display additional layers of information! Interactive and real-time state changes and artifacts are now visible in context on the graph.
These new layers are available for opt-in usage via the PREFECT_EXPERIMENTAL_ENABLE_ARTIFACTS_ON_FLOW_RUN_GRAPH
and PREFECT_EXPERIMENTAL_ENABLE_STATES_ON_FLOW_RUN_GRAPH
settings.
Agents
A year ago, we released workers as a replacement for agents. Workers significantly enhance the experience of deploying flows and simplify the specification of each flow's infrastructure and runtime environment.
With this release we are adding a six month (September 14) deprecation warning to agents and related concepts. Please note that:
- Deprecation will not impact or break any work running with agents and agent-related concepts - although we encourage users to upgrade to workers because they provide a better deployment experience, you can continue to use existing agents and related concepts after deprecation
- After September 14, Prefect Cloud users will not be able to create new agent work pools or infrastructure blocks
- After September 14, new minor versions of the Prefect Python package will not include agents
Like agents, workers support creating deployments through the Prefect CLI and through Python, but require different syntax. For more information, please refer to the Upgrade from Agents to Workers guide.
Enhancements
- Give better client-side feedback on websocket authT/authZ issues - #12221
- Allow table artifact cells to render markdown content -
#2190
- Add an 'AzureBlobStorageContainer' block -
#139
- API for task run counts by state - #12244
- Improved UI handling of custom flow run states. Badges for a state with a custom name will now more closely resemble their underlying state - PrefectHQ/prefect-ui-library#2210 and PrefectHQ/prefect-ui-library#2208
Fixes
- Fix support for legacy schedule in
build_from_flow
- #12257 - Remove extra field from create endpoints - #12240
- Prevent invalid link problem for
prefect deployment run
- #12267 - Hydrate run input when resuming a flow run - #12259
- Fix state select in notifications
#2216
- Adds porting of network configuration to generated base job templates
#392
Experimental
- Expose
serve
method on tasks - #12239 - Avoid relative module path problem with subscription keys - #12227
Deprecations
- Deprecate
PrefectAgent
andprefect agent
command group - #12273 - Deprecate Infrastructrue blocks - #12282
- Deprecate
Deployment
class and deploymentbuild
andapply
commands - #12283 - Deprecate the
packaging
module - #12291 - Deprecate storage blocks with implementations in Prefect integration packages - #12299
- Do not create
default-agent-pool
in empty state server - #12214
Typing
Documentation
- Fix formatting in concept -> work-pools.md - #12230
- Add
get_run_logger
to imports in interactive workflow examples - #12284 - Add
on_running
state change hook to State -> Concepts docs - #12293 - Fix some argument names in deploy guide - #12301
New Contributors
- @hainenber made their first contribution in #12232
Full Changelog: 2.16.3...2.16.4
Release 2.16.3
This release includes enhanced deployment parameters in the Prefect UI. We’ve refined the deployment parameters UI to significantly enhance default value handling and list management.
🛠️ This release introduces support for tuple-type parameters and a revamped list UI, freeing users from the constraints of JSON for list inputs.
🔄 You now have the flexibility to utilize JSON or Prefect variables for any parameter value, enhancing the versatility of deployment configurations.
💪 Moreover, we’ve synchronized validation across the UI and deployment schemas, ensuring that user inputs are consistently checked against the defined parameter requirements, which simplifies the deployment process and minimizes configuration errors.
✅ These improvements are aimed at providing a more efficient and user-friendly interface for managing deployment parameters. 🌟
This release also includes many other enhancements and fixes, including a new on_running
state change hook 🎣, two memory leak fixes 🔧, and much more.
See the release notes for details! 📜🚀
Release 2.16.2
⛑️ Fixing a logging bug introduced by 2.16.1
In 2.16.1
, a regression related to logging non-str
objects (when a PREFECT_API_KEY
was set) was introduced. The fix is included in 2.16.2
.
See #12151 for implementation details.
🎯 Use a module path entrypoint when using .serve
or .deploy
from prefect_dbt.cloud.jobs import trigger_dbt_cloud_job_run_and_wait_for_completion
from prefect.deployments import EntrypointType
if __name__ == "__main__":
trigger_dbt_cloud_job_run_and_wait_for_completion.deploy(
name="demo-deploy-from-module",
work_pool_name="olympic",
entrypoint_type=EntrypointType.MODULE_PATH,
job_variables={"env": { "EXTRA_PIP_PACKAGES": "prefect-dbt" } }
)
See #12134 for implementation details.
Full Changelog: 2.16.1...2.16.2
See the release notes for more!
Release 2.16.1
Enhanced multiple schedule support 📅
prefect.yaml
now supports specifying multiple schedules via the schedules
key.
This allows you to define multiple schedules for a single deployment, and each schedule can have its own cron
, interval
, or rrule
configuration:
...
schedules:
- cron: "0 0 * * *"
active: false
- interval: 3600
active: true
- rrule: "FREQ=YEARLY"
active: true
In addition you can specify multiple schedules via arguments to prefect deploy
:
prefect deploy ... --cron '4 * * * *' --cron '1 * * * *' --rrule 'FREQ=DAILY'
More detail
We've also added support for multiple schedules to flow.serve
, flow.deploy
and prefect.runner.serve
. You can provide multiple schedules by passing a list to the cron
, interval
, or rrule
arguments:
import datetime
import random
from prefect import flow
@flow
def trees():
tree = random.choice(["🌳", "🌴", "🌲", "🌵"])
print(f"Here's a happy little tree: {tree}")
if __name__ == "__main__":
trees.serve(
name="trees",
interval=[3600, 7200, 14400],
)
This will create a deployment with three schedules, one that runs every hour, one that runs every two hours, and one that runs every four hours. For more advanced cases, use the schedules
argument.
trees.serve(
name="trees",
schedules=[
IntervalSchedule(interval=datetime.timedelta(minutes=30)),
{"schedule": RRuleSchedule(rrule="FREQ=YEARLY"), "active": True},
MinimalDeploymentSchedule(schedule=CronSchedule(cron="0 0 * * *"), active=False),
]
)
Dive into these new scheduling capabilities today and streamline your workflows like never before.
For implementation details, see the following pull request:
- #12107
New Contributors 🌟
- Shoutout to @jrbourbeau for their first contribution!
See the release notes for more details!
Release 2.16.0
🕢 Deployments now support multiple schedules 🕐
With today’s release, we’re excited to roll out initial support for using multiple schedules with Deployments! You can now use multiple schedules in the following ways:
- Specifying schedules in a Deployment YAML file
- Creating Python-based Deployments with the
Deployment
class - New CLI commands:
prefect deployment schedule <create, delete, pause, resume, ls, clear>
- New UI components aware of multiple schedules
Coming soon, we’ll round out support for multiple schedules in other areas, such as:
- When running a flow with
flow.serve()
andflow.deploy()
- When using
prefect deploy
The easiest way to get started with multiple schedules is to try out the new CLI commands:
$ prefect deployment schedule ls happy-flow/my-deployment
Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID ┃ Schedule ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 │ cron: 0 0 1 * 1 │ True │
└──────────────────────────────────────┴─────────────────┴────────┘
$ prefect deployment schedule create happy-flow/my-deployment --interval 60
Created deployment schedule!
$ prefect deployment schedule ls happy-flow/my-deployment
Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID ┃ Schedule ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 3638ed58-cab2-4462-a680-2f92fcf6c797 │ interval: 0:01:00s │ True │
│ c7d3ddc4-9a5a-4dec-bd59-eed282ae55d5 │ cron: 0 0 1 * 1 │ True │
└──────────────────────────────────────┴────────────────────┴────────┘
Enhancements
- Add support for deploying to a process work pool using
flow.deploy
anddeploy
— #12017 - Add support for multiple schedules to Prefect server and CLI — #11971
- Add CLI command to read runs in a work queue — #11989
Fixes
- Update the text for the CLI command
deployment run --help
so it renders for more args — #11960 - Fix
Flow.with_options
logic for retries, retry_delay_seconds, flow_run_name — #12020 - Fix memory leaks related to cancellation scopes and async contextvar usage — #12019
- Revert the recent change that runs on the main thread while we investigate a concurrency issue — #12054
- Add a more readable error if Docker is not running — #12045
Documentation
- Improve language and formatting in Profiles and Configuration guide — #11996
- Improves docs formatting consistency and adds some minor content updates — #12004
- Updates formatting for guide: creating-interactive-workflows.md — #11991
- Add import statement for
wait_for_flow_run
— #11999 - Add deep dive on overriding job variables — #12033
- Remove extraneous trailing quotation marks in concepts/artifacts docs — #12040
- Add links to overriding job variables guide — #12043
- Update scheduling docs to include information about multiple schedules — #12064
Experimental
- Only allow using
Task.submit()
for autonomous task submission — #12025
Contributors
- @hamzamogni made their first contribution in #12000
- @eladm26 made their first contribution in #12045
Full release notes: https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES.md
Release 2.15.0
🔧 Task runs now execute on the main thread
We are excited to announce that task runs are now executed on the main thread!
When feasible, task runs are now executed on the main thread instead of a worker thread. Previously, all task runs were run in a new worker thread. This allows objects to be passed to and from tasks without worrying about thread safety unless you have opted into concurrency. For example, an HTTP client or database connection can be shared between a flow and its tasks now (unless synchronous concurrency is used). Some asynchronous and sequential use cases may see performance improvements.
Consider the following example:
import sqlite3
from prefect import flow, task
db = sqlite3.connect("threads.db")
try:
db.execute("CREATE TABLE fellowship(name)")
except sqlite3.OperationalError:
pass
else:
db.commit()
db.execute("DELETE FROM fellowship")
db.commit()
cur = db.cursor()
@task
def my_task(name: str):
global db, cur
cur.execute('INSERT INTO fellowship VALUES (?)', (name,))
db.commit()
@flow
def my_flow():
global db, cur
for name in ["Frodo", "Gandalf", "Gimli", "Aragorn", "Legolas", "Boromir", "Samwise", "Pippin", "Merry"]:
my_task(name)
print(cur.execute("SELECT * FROM fellowship").fetchall())
db.close()
if __name__ == "__main__":
my_flow()
In previous versions of Prefect, running this example would result in an error like this:
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 7977619456 and this is thread id 6243151872.
But now, with task runs executing on the main thread, this example will run without error! We're excited this change makes Prefect even more intuitive and flexible!
See the following pull request for implementation details:
- #11930
🔭 Monitor deployment runs triggered via the CLI
You can monitor the status of a flow run created from a deployment via the CLI. This is useful for observing a flow run's progress without navigating to the UI.
To monitor a flow run started from a deployment, use the --watch
option with prefect deployment run
:
prefect deployment run --watch <slugified-flow-name>/<slugified-deployment-name>
See the following pull request for implementation details:
- #11702
Enhancements
- Enable work queue status in the UI by default — #11976 & PrefectHQ/prefect-ui-library#2080
Fixes
- Update vendored
starlette
version to resolve vulnerability inpython-mulipart
— #11956 - Fix display of interval schedules created with a different timezone than the current device - PrefectHQ/prefect-ui-library#2090
Experimental
- Prevent
RUNNING
->RUNNING
state transitions for autonomous task runs — #11975 - Provide current thread to the engine when submitting autonomous tasks — #11978
- Add intermediate
PENDING
state for autonomous task execution — #11985 - Raise exception when stopping task server — #11928
Documentation
- Update work pools concepts page to include Modal push work pool — #11954
- Add details to
run_deployment
tags parameter documentation — #11955 - Add Helm chart link in Prefect server instance docs — #11970
- Clarify that async nested flows can be run concurrently — #11982
- Update work queue and flow concurrency information to include push work pools — #11974
Contributors
All changes: 2.14.21...2.15.0