Update xctest test result parsing for Xcode 16
#431
Conversation
priitlatt
changed the title
xctest test result parsing for Xcode 16
| @@ -30,13 +30,17 @@ def disabled(self) -> int: | |||
| return sum(suite.disabled for suite in self.test_suites if suite.disabled) | |||
|
|
|||
| @property | |||
| def errors(self) -> int: | |||
| def errors(self) -> Optional[int]: | |||
There was a problem hiding this comment.
Can you elaborate why we should introduce Optional[int] here, instead just return 0 ?
There was a problem hiding this comment.
Errors and failures in JUnit XML format are optional. By having optional integer value for those here we can more closely represent the reality. Xcode 16+ does not tell anything about failed test cases, there are only errored tests. So the corresponding JUnit XML test suite should not include failures at all for such cases. But if we'd always return 0 then it is not possible to differentiate the cases where test suite does not specify this value from those where it is defined but legitemately 0.
Ultimately it is to omit undefined failures or errors from the test result overview tables.
|
Changelog was updated in e0fd0ef to reflect modifications of JUnit |
This PR is a follow up to issue #423 and its fix in #424.
Xcode 16.0 introduced changes to
xcresulttoolAPI which deprecatedxcresultparsing options that were previously working:was replaced with
where subcommand is any of
summary,tests,test-details,activities,insights.Along with the CLI change, the output format was also significantly altered.
This PR adds Python mappings for
xcresulttool get test-resultsoutputs and uses those to generate JUnit test suites. In most cases the conversion results are identical to what it used to be with legacy options, but for some time already Xcode does not differentiate test cases that errored from those for which some assertions failed. While the legacy implementation kept track of failed results separately, this is not possible with newxcresulttooloutput and now all test cases that failed due to whatever reasons are counted aserrored.Examples of
xcresulttool get test-resultJSON outputs can be seen under mocks intests/models/xctests/mocksdirectory.Conversion from
xcresultto JUnit compatible test suite is still done inXcResultConverterclass whose API remained fully backwards compatible. Now when selected Xcode version is 16+ then new conversion implementation is used, and the legacy one otherwise.Models JSON schemas
xcrun xcresulttool get test-results testsoutput{ "schemas": { "Tests": { "type": "object", "properties": { "testPlanConfigurations": { "type": "array", "items": { "$ref": "#/schemas/Configuration" } }, "devices": { "type": "array", "items": { "$ref": "#/schemas/Device" } }, "testNodes": { "type": "array", "items": { "$ref": "#/schemas/TestNode" } } }, "required": [ "testPlanConfigurations", "devices", "testNodes" ] }, "Configuration": { "type": "object", "properties": { "configurationId": { "type": "string" }, "configurationName": { "type": "string" } }, "required": [ "configurationId", "configurationName" ] }, "Device": { "type": "object", "properties": { "deviceId": { "type": "string" }, "deviceName": { "type": "string" }, "architecture": { "type": "string" }, "modelName": { "type": "string" }, "platform": { "type": "string" }, "osVersion": { "type": "string" } }, "required": [ "deviceName", "architecture", "modelName", "osVersion" ] }, "TestNode": { "type": "object", "properties": { "nodeIdentifier": { "type": "string" }, "nodeType": { "$ref": "#/schemas/TestNodeType" }, "name": { "type": "string" }, "details": { "type": "string" }, "duration": { "type": "string" }, "result": { "$ref": "#/schemas/TestResult" }, "tags": { "type": "array", "items": { "type": "string" } }, "children": { "type": "array", "items": { "$ref": "#/schemas/TestNode" } } }, "required": [ "nodeType", "name" ] }, "TestResult": { "type": "string", "enum": [ "Passed", "Failed", "Skipped", "Expected Failure", "unknown" ] }, "TestNodeType": { "type": "string", "enum": [ "Test Plan", "Unit test bundle", "UI test bundle", "Test Suite", "Test Case", "Device", "Test Plan Configuration", "Arguments", "Repetition", "Test Case Run", "Failure Message", "Source Code Reference", "Attachment", "Expression", "Test Value" ] } } }xcrun xcresulttool get test-results summaryoutput{ "schemas": { "Summary": { "type": "object", "properties": { "title": { "type": "string" }, "startTime": { "type": "number", "format": "double", "description": "Date as a UNIX timestamp (seconds since midnight UTC on January 1, 1970)" }, "finishTime": { "type": "number", "format": "double", "description": "Date as a UNIX timestamp (seconds since midnight UTC on January 1, 1970)" }, "environmentDescription": { "type": "string", "description": "Description of the Test Plan, OS, and environment that was used during testing" }, "topInsights": { "type": "array", "items": { "$ref": "#/schemas/InsightSummary" } }, "result": { "$ref": "#/schemas/TestResult" }, "totalTestCount": { "type": "integer" }, "passedTests": { "type": "integer" }, "failedTests": { "type": "integer" }, "skippedTests": { "type": "integer" }, "expectedFailures": { "type": "integer" }, "statistics": { "type": "array", "items": { "$ref": "#/schemas/Statistic" } }, "devicesAndConfigurations": { "$ref": "#/schemas/DeviceAndConfigurationSummary" }, "testFailures": { "$ref": "#/schemas/TestFailure" } }, "required": [ "title", "environmentDescription", "topInsights", "result", "totalTestCount", "passedTests", "failedTests", "skippedTests", "expectedFailures", "statistics", "devicesAndConfigurations", "testFailures" ], }, "InsightSummary": { "type": "object", "properties": { "impact": { "type": "string" }, "category": { "type" "string" }, "text": { "type" "string" } }, "required": [ "impact", "category", "text" ] }, "TestResult": { "type": "string", "enum": [ "Passed", "Failed", "Skipped", "Expected Failure", "unknown" ] }, "Statistic": { "type": "object", "properties": { "title": { "type": "string" }, "subtitle": { "type": "string" } }, "required": [ "title", "subtitle" ] }, "DeviceAndConfigurationSummary": { "type": "object", "properties": { "device": { "$ref": "#/schemas/Device" }, "testPlanConfiguration": { "$ref": "#/schemas/Configuration" }, "passedTests": { "type": "integer" }, "failedTests": { "type": "integer" }, "skippedTests": { "type": "integer" }, "expectedFailures": { "type": "integer" } }, "required": [ "device", "testPlanConfiguration", "passedTests", "failedTests", "skippedTests", "expectedFailures" ] }, "Device": { "type": "object", "properties": { "deviceId": { "type": "string" }, "deviceName": { "type": "string" }, "architecture": { "type": "string" }, "modelName": { "type": "string" }, "platform": { "type": "string" }, "osVersion": { "type": "string" } }, "required": [ "deviceName", "architecture", "modelName", "osVersion" ] }, "Configuration": { "type": "object", "properties": { "configurationId": { "type": "string" }, "configurationName": { "type": "string" } }, "required": [ "configurationId", "configurationName" ] }, "TestFailure": { "type": "object", "properties": { "testName": { "type": "string" }, "targetName": { "type": "string" }, "failureText": { "type": "string" }, "testIdentifier": { "type": "integer", "format": "int64" } }, "required": [ "testName", "targetName", "failureText", "testIdentifier" ] } } }QA notes:
Tested with the following Xcode versions:
Updated actions:
xcode-project run-testsxcode-project test-summaryxcode-project junit-test-results