request: check whether text and icons are consistent in CI tests

[Giuspepe]

Is there an existing feature request for this?

• I have searched the existing issues.

Command

I would love if CI tests would check texts and icons.

Description

It would be great if changing icons or texts would be caught by golden tests. I understand that currently text is not rendered as text to make CI tests stable between platforms, but maybe there is another solution to get the best of both worlds.

Take the following test file as an example. Note that both tests use the same file as golden image reference. The second golden test should fail because it contains a different icon than the first test, but the CI test passes because the icon (which is text as well) is rendered as a black box on both tests.

void main() {
  AlchemistConfig.runWithConfig(
    config: AlchemistConfig.current().merge(
      const AlchemistConfig(
        ciGoldensConfig: CiGoldensConfig(enabled: true),
        platformGoldensConfig: PlatformGoldensConfig(enabled: false),
      ),
    ),
    run: () {
      goldenTest(
        'should pass and passes',
        fileName: 'icon_test',
        builder: () => GoldenTestScenario(
          name: 'icon A',
          child: const Icon(Icons.thumb_up),
        ),
      );

      goldenTest(
        'should fail, but passes too!',
        // use same golden image as the previous test
        fileName: 'icon_test',
        builder: () => GoldenTestScenario(
          name: 'icon B',
          // but use a different icon
          child: const Icon(Icons.bug_report),
        ),
      );
    },
  );
}

Reasoning

This feature would make golden tests even more powerful by preventing more types of regressions.

Additional context and comments

My suggestion is the following:

During the replacement of text blocks with colored rectangles, write all blocked text blocks into a Map<Offset, List<String>> textBlocks. At the end of a test, compare not only the golden image file but also the map containing the text blocks.

Add something like textBlocks.putIfAbsent(offset, () => []).add(child.text.toPlainText()); to the if(child is RenderParagraph) block. Or maybe child.text.toStringDeep() would be better because that also contains information on font family, color, etc.

alchemist/lib/src/blocked_text_image.dart

Lines 38 to 43 in f0de484

	void paintChild(RenderObject child, Offset offset) {
	if (child is RenderParagraph) {
	final paint = Paint()
	..color = child.text.style?.color ?? const Color(0xFF000000);
	canvas.drawRect(offset & child.size, paint);
	} else {

If a test is being run with --update-goldens or forceUpdateGoldens: true, save the Map<Offset, List> under goldens/ci/<golden_test_file_name>.texts. Else compare the current textBlocks map with the persisted golden textBlocks map.

The map should be a Map<Offset, List<String>> and not a Map<Offset, String> because multiple texts can be drawn at the same offset. Another advantage of List is that it also keeps track of the correct painting order.

[jolexxa]

Very cool idea! I think we can consider a PR, assuming @Kirpal and @jeroen-meijer don't have any big reasons against.

I'm not sure how this approach will pan out without doing it myself, but I might suggest starting by injecting a new service into BlockedTextPaintingContext during construction here:

alchemist/lib/src/blocked_text_image.dart

Lines 14 to 19 in 109b799

	class BlockedTextPaintingContext extends PaintingContext {
	/// {@macro blocked_text_painting_context}
	BlockedTextPaintingContext({
	required ContainerLayer containerLayer,
	required Rect estimatedBounds,
	}) : super(containerLayer, estimatedBounds);

and here:

alchemist/lib/src/golden_test_adapter.dart

Lines 294 to 297 in 405788c

	paintingContextBuilder(
	layer,
	renderObject.paintBounds,
	).paintSingleChild(renderObject);

This service, which would be responsible for the text/icons, can use the goldenTestAdapter to interface directly with flutter_test. It can also interface with the file package for any filesystem operations you might need, since it is friendly to testing.