Reporting rendering regression on PRs

[mockersf]

Good news! Since #13248 (and #15894, #15911 and #15914) CI is now checking screenshots taken during example runs on macOS and Linux for changes.

The list of examples that are executed can be found there: https://github.com/bevyengine/bevy/tree/main/.github/example-run. At the moment, only 2 examples take a screenshot: breakout and load_gltf

The macOS run is done on every push to a PR, in the merge queue, and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339649537/job/31534799604

The Linux run is done in the merge queue and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339392720/job/31534437159

As you may have noticed, they both fail: this is because the example load_gltf is a tiny bit time dependent for the light movement, so not fully deterministic. Failures are currently not reported or blocking for PRs.

It is possible to make examples deterministic even when relying on time by applying the patches there, or using the example-showcase tool.

You can see a list of run here: https://pixel-eagle.com/project/B04F67C0-C054-4A6F-92EC-F599FEC2FD1D

I have some improvements planned for the reporting, but now my question to you, dear contributors: which examples should be ran and checked for changes in CI?

This issue will be fixed by changing the list of screenshots compared to a fully deterministic one (either by selecting deterministic examples, or by enabling stable time with the patches), and changes being reported by a comment on PRs (not blocking merging yet)

[ickshonpe]

Really like having this.

I think the bevy_ui examples that would be the most valuable to watch are:

• pbr
• borders
• ui
• text_wrap_debug
• grid
• overflow
• ui_texture_slice_flp_and_tile
• z_index
• box_shadow

We could add some extra elements to the ui example if that list is a bit long.

[rparrett]

Do we have a target for the amount of examples we want to run?

It would be nice to have more "roll-up examples" like ui that show off many features of an engine area. Beyond ui, the following might be nice IMO:

• lighting
• split_screen
• breakout
• animated_fox (unfortunately this became non-deterministic at some point)

Maybe our game examples could be fleshed out to use more features, or some additional purpose-built examples could be made.

edit: Totally missed the "testbed" stuff that was just linked here. Seems like a great idea.

[mockersf]

I'm trying "testbed" examples, see #15954

I would like to have one for 2d, one for 3d and one for UI, at least as a starting point. Once we have ones for 2d and UI we can remove the "breakout" example running in CI, once we have 3d we can remove "load_gltf" and "alien_cake_addict" to keep the execution time under control