Add mutation tests to pipeline #334

dkreuer · 2021-12-07T21:03:01Z

Implementation for #332

Ocramius · 2021-12-08T06:31:05Z

Merging to 3.9.x, but will rename the branch to 4.0.x afterwards

Ocramius · 2021-12-08T06:31:33Z

baseline.xml

@@ -3,17 +3,4 @@
  <file src="src/ComposerRequireChecker/Cli/CheckCommand.php">
    <MixedArgumentTypeCoercion occurrences="2"/>


Oh, last one left? Can we completely get rid of the baseline, perhaps?

Currently I don’t know how to process all the ongoing generator stuff ^^

Ocramius · 2021-12-08T06:32:12Z

infection.json.dist

+            "src"
+        ]
+    },
+    "timeout": 30,


Let's lower the timeout to 5 seconds or such: we don't have long-running integration tests anyway

With default 10 seconds a lot mutations were skipped due to timeout. But I‘ll try to reduce it in the dist.

Done. Many skipped tests now which reduces the MSI. They are not considered T(imed out).

@maks-rafalko do you perhaps know if this is default behavior of @infection, or if it's a bug to be reported? 🤔

from the quick look, Skipped mutants should not decrease MSI:

but if you found a case when it does, please report

Doesn't that reduce the coverage itself?

Just added a bunch of obvious @covers to the test classes ... result:

496 mutations were generated: 384 mutants were killed 95 mutants were not covered by tests 17 covered mutants were not detected 0 errors were encountered 0 syntax errors were encountered 0 time outs were encountered 0 mutants required more time than configured

Not very promising. Probably comes from the fact that there are a lot of integration tests in contrary to unit tests. If the coverage would come from unit tests I guess @covers would be fine.

Yes, but tests should be isolated to a certain unit :D

Mutation testing integration tests is expensive, so we need to decide how coverage leaks in there.

Possible way to move forward:

accept the 30s timeout for now

in subsequent issue add @covers And @group integration|unit

infection and coverage only on @group unit so that coverage/MSI does not decrease

I volunteer to write additional tests ;-)

I volunteer to write additional tests ;-)

Cool, your problem now :D

Cool, your problem now :D

Not in this PR maybe please? ;-)

src/ComposerRequireChecker/FileLocator/LocateComposerPackageSourceFiles.php

Ocramius · 2021-12-08T06:41:17Z

src/ComposerRequireChecker/JsonLoader.php

@@ -37,6 +41,7 @@ public static function getData(string $path): array
            throw new InvalidJson('error parsing ' . $path . ': ' . $exception->getMessage(), 0, $exception);
        }

+        /** @psalm-var ComposerData $decodedData */


Is any meaningful assertion possible here? Trusting the JSON data to be valid seems a bit extreme.

I usually use @azjezz /psl for that

I guess you mean PSL\Json\typed()?

i can introduce the dependency if you like.

IMO worth adding indeed 👍

Not sure about whether phpstan/phpstan will be happy about it - we'll see :D

doc: https://php-standard-library.github.io/#/components/json/

phpstan probably won't be happy :D especially if you are using Type\shape([...]), as that only works properly when using the psl psalm plugin ( https://github.com/php-standard-library/psalm-plugin/blob/main/src/EventHandler/Type/Shape/FunctionReturnTypeProvider.php )

Yep, but dropping phpstan is no biggie either

If you would like, you can merge this PR as is, and i will send a PR similar to the ones i sent for Roave/* and Laminas/* 😉

@dkreuer your choice on that :D

Having help in that would be great.

cool, will send a PR for PSL integration this weekend 👍

Ocramius · 2021-12-08T06:50:55Z

test/ComposerRequireCheckerTest/Exception/FileParseFailedTest.php

+use Exception;
+use PHPUnit\Framework\TestCase;
+
+final class FileParseFailedTest extends TestCase


Consider generally adding /** @covers annotations - helps with keeping executed mutation tests small :)

The timeout must be raised currently to not have integration tests (which currently count to coverage) being skipped.

Ocramius

🚢

Thanks @dkreuer!

dkreuer added 2 commits December 7, 2021 17:42

Upgrade phpunit.xml.dist

e8f83af

Require roave/infection-static-analysis-plugin as dev dependency

1f4a21a

dkreuer force-pushed the feature/mutation-tests-pipeline branch from 35eea62 to 1985cf7 Compare December 7, 2021 21:04

dkreuer marked this pull request as ready for review December 7, 2021 21:24

Ocramius added the enhancement label Dec 8, 2021

Ocramius added this to the 4.0.0 milestone Dec 8, 2021

Ocramius self-assigned this Dec 8, 2021

Ocramius requested changes Dec 8, 2021

View reviewed changes

dkreuer added 4 commits December 8, 2021 08:38

Add configuration for mutation testing with infection-php

114f938

Add mutation testing pipeline

387d399

Squash mutations

636ff58

Squash psalm errors to have perfect baseline for mutation testing

796c406

dkreuer force-pushed the feature/mutation-tests-pipeline branch from 1985cf7 to 796c406 Compare December 8, 2021 07:39

dkreuer added 2 commits December 9, 2021 10:26

Make pipeline fail on mutation tests

48cb4f0

Raise timeout for mutation tests

91ba74e

The timeout must be raised currently to not have integration tests (which currently count to coverage) being skipped.

dkreuer force-pushed the feature/mutation-tests-pipeline branch 2 times, most recently from cd52e0e to 97b46ea Compare December 9, 2021 12:37

Squash mutants

2c6fe9c

dkreuer force-pushed the feature/mutation-tests-pipeline branch from 97b46ea to 2c6fe9c Compare December 9, 2021 12:43

Ocramius approved these changes Dec 13, 2021

View reviewed changes

Ocramius merged commit 7d73b3e into maglnet:3.9.x Dec 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mutation tests to pipeline #334

Add mutation tests to pipeline #334

dkreuer commented Dec 7, 2021

Ocramius commented Dec 8, 2021

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius Dec 8, 2021

maks-rafalko Dec 8, 2021

dkreuer Dec 8, 2021 •

edited

Loading

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius Dec 8, 2021

azjezz Dec 8, 2021

Ocramius Dec 8, 2021

azjezz Dec 8, 2021

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

azjezz Dec 8, 2021 •

edited

Loading

Ocramius Dec 8, 2021

dkreuer Dec 8, 2021

Ocramius left a comment

		@@ -3,17 +3,4 @@
		<file src="src/ComposerRequireChecker/Cli/CheckCommand.php">
		<MixedArgumentTypeCoercion occurrences="2"/>

Add mutation tests to pipeline #334

Add mutation tests to pipeline #334

Conversation

dkreuer commented Dec 7, 2021

Ocramius commented Dec 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dkreuer Dec 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

azjezz Dec 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ocramius left a comment

Choose a reason for hiding this comment

dkreuer Dec 8, 2021 •

edited

Loading

azjezz Dec 8, 2021 •

edited

Loading