Feature: Add capabilities to run tests with xcode-project (#54)

* WIP: add xcode-project action to list devices

* Add option to filter runtimes

* Finalize list devices action

* Add option to specify test destinations

* Fix types

* Fix test_get_selected

* Do not print default device info when acquiring it for test run

* Extract xcode-project arguments into separate file and restructure code

* WIP: xcresults parsing

* Fix cli_app references

* Fix type hints

* Remove metaclass from RunningCliAppMixin

* Skip not-implemented test

* Mirror xcresult json model as Python class interface

* WIP: convert xcresults to junit xml

* Add parser that converts Xcresult to JUnit XML

* Split xcresult collection and conversion and include more meta info in test suites

* Show test results in nice format

* Remove unnecessary xml

* Validate runtimes for test-destinations

* Improve formatting

* Add tests

* Generate docs for new actions

* Update docstrings

* Update docs

* Update test arguments

* Add option to test only specified tests

* Add option to limit maximum concurrent test destinations

* Improve test results table layout

* Fix typing

* Extract table into separate module

* Update changelog and bump version

* Add option to toggle code coverage off

* Exit with error status by default in case of failed tests

* Update docs

* Add action 'test-summary' to show xcresult overview

* Gather xcresults and simulator logs to test outputs dir

* Include device info in testsuite name if present

* Fix tests

* Update docs

CHANGELOG.md

@@ -1,3 +1,18 @@
+Version 0.4.0
+-------------
+
+**Improvements**
+
+- Feature: Add action `clean` to `xcode-project` to clean Xcode project.
+- Feature: Add action `default-test-destination` to `xcode-project` to show default test destination for the chosen Xcode version.
+- Feature: Add action `test-destinations` to `xcode-project` to list available destinations for test runs.
+- Feature: Add action `junit-test-results` to `xcode-project` to convert Xcode Test Result Bundles (*.xcresult) to JUnit XML format.
+- Feature: Add action `run-tests` to `xcode-project` to run unit or UI tests for given Xcode project or workspace.
+- Feature: Add action `test-summary` to `xcode-project` to show test result summary from given Xcode Test Result Bundles (*.xcresult).
+- Refactoring: Create `RunningCliAppMixin` to avoid passing around currently invoked app instance.
+
+- Update [cryptography](https://github.com/pyca/cryptography) Python dependency to version ~=3.2.
+
 Version 0.3.2
 -------------
 

docs/README.md

@@ -9,4 +9,4 @@ CLI tools
 |[`git-changelog`](git-changelog/README.md)|Generate a changelog text from git history|
 |[`keychain`](keychain/README.md)|Utility to manage macOS keychains and certificates|
 |[`universal-apk`](universal-apk/README.md)|Generate universal APK files from Android App Bundles.     DEPRECATED! Use `android-app-bundle` instead.|
-|[`xcode-project`](xcode-project/README.md)|Utility to prepare iOS application code signing properties for build|
+|[`xcode-project`](xcode-project/README.md)|Utility to work with Xcode projects. Use it to manage iOS application     code signing properties for builds, create IPAs and run tests.|

docs/xcode-project/README.md

@@ -3,7 +3,7 @@ xcode-project
 =============
 
 
-**Utility to prepare iOS application code signing properties for build**
+**Utility to work with Xcode projects. Use it to manage iOS application     code signing properties for builds, create IPAs and run tests.**
 ### Usage
 ```bash
 xcode-project [-h] [--log-stream STREAM] [--no-color] [--version] [-s] [-v]
@@ -40,5 +40,11 @@ Enable verbose logging for commands
 |Action|Description|
 | :--- | :--- |
 |[`build-ipa`](build-ipa.md)|Build ipa by archiving the Xcode project and then exporting it|
+|[`clean`](clean.md)|Clean Xcode project|
+|[`junit-test-results`](junit-test-results.md)|Convert Xcode Test Result Bundles (*.xcresult) to JUnit XML format|
 |[`detect-bundle-id`](detect-bundle-id.md)|Try to deduce the Bundle ID from specified Xcode project|
+|[`default-test-destination`](default-test-destination.md)|Show default test destination for the chosen Xcode version|
+|[`test-destinations`](test-destinations.md)|List available destinations for test runs|
+|[`run-tests`](run-tests.md)|Run unit or UI tests for given Xcode project or workspace|
+|[`test-summary`](test-summary.md)|Show summary of Xcode Test Result|
 |[`use-profiles`](use-profiles.md)|Set up code signing settings on specified Xcode projects         to use given provisioning profiles|

docs/xcode-project/build-ipa.md

@@ -12,11 +12,12 @@ xcode-project build-ipa [-h] [--log-stream STREAM] [--no-color] [--version] [-s]
     [--target TARGET_NAME]
     [--config CONFIGURATION_NAME]
     [--scheme SCHEME_NAME]
+    [--clean]
     [--archive-directory ARCHIVE_DIRECTORY]
     [--archive-flags ARCHIVE_FLAGS]
     [--archive-xcargs ARCHIVE_XCARGS]
     [--ipa-directory IPA_DIRECTORY]
-    [--export-options-plist EXPORT_OPTIONS_PATH]
+    [--export-options-plist EXPORT_OPTIONS_PATH_EXISTING]
     [--export-flags EXPORT_FLAGS]
     [--export-xcargs EXPORT_XCARGS]
     [--remove-xcarchive]
@@ -45,6 +46,10 @@ Name of the Xcode build configuration
 
 
 Name of the Xcode Scheme
+##### `--clean`
+
+
+Whether to clean the project before building it
 ##### `--archive-directory=ARCHIVE_DIRECTORY`
 
 
@@ -61,7 +66,7 @@ Pass additional arguments to xcodebuild for the archive phase. For example `COMP
 
 
 Directory where the built ipa is stored. Default: `build/ios/ipa`
-##### `--export-options-plist=EXPORT_OPTIONS_PATH`
+##### `--export-options-plist=EXPORT_OPTIONS_PATH_EXISTING`
 
 
 Path to the generated export options plist. Default: `$HOME/export_options.plist`

docs/xcode-project/clean.md

@@ -0,0 +1,73 @@
+
+clean
+=====
+
+
+**Clean Xcode project**
+### Usage
+```bash
+xcode-project clean [-h] [--log-stream STREAM] [--no-color] [--version] [-s] [-v]
+    [--project XCODE_PROJECT_PATH]
+    [--workspace XCODE_WORKSPACE_PATH]
+    [--target TARGET_NAME]
+    [--config CONFIGURATION_NAME]
+    [--scheme SCHEME_NAME]
+    [--disable-xcpretty]
+    [--xcpretty-options OPTIONS]
+```
+### Optional arguments for action `clean`
+
+##### `--project=XCODE_PROJECT_PATH`
+
+
+Path to Xcode project (\*.xcodeproj)
+##### `--workspace=XCODE_WORKSPACE_PATH`
+
+
+Path to Xcode workspace (\*.xcworkspace)
+##### `--target=TARGET_NAME`
+
+
+Name of the Xcode Target
+##### `--config=CONFIGURATION_NAME`
+
+
+Name of the Xcode build configuration
+##### `--scheme=SCHEME_NAME`
+
+
+Name of the Xcode Scheme
+##### `--disable-xcpretty`
+
+
+Do not use XCPretty formatter to process log output
+##### `--xcpretty-options=OPTIONS`
+
+
+Command line options for xcpretty formatter. For example "--no-color" or "--simple  --no-utf". Default: `--color`
+### Common options
+
+##### `-h, --help`
+
+
+show this help message and exit
+##### `--log-stream=stderr | stdout`
+
+
+Log output stream. Default `stderr`
+##### `--no-color`
+
+
+Do not use ANSI colors to format terminal output
+##### `--version`
+
+
+Show tool version and exit
+##### `-s, --silent`
+
+
+Disable log output for commands
+##### `-v, --verbose`
+
+
+Enable verbose logging for commands

docs/xcode-project/default-test-destination.md

@@ -0,0 +1,43 @@
+
+default-test-destination
+========================
+
+
+**Show default test destination for the chosen Xcode version**
+### Usage
+```bash
+xcode-project default-test-destination [-h] [--log-stream STREAM] [--no-color] [--version] [-s] [-v]
+    [--json]
+```
+### Optional arguments for action `default-test-destination`
+
+##### `--json`
+
+
+Whether to show the resource in JSON format
+### Common options
+
+##### `-h, --help`
+
+
+show this help message and exit
+##### `--log-stream=stderr | stdout`
+
+
+Log output stream. Default `stderr`
+##### `--no-color`
+
+
+Do not use ANSI colors to format terminal output
+##### `--version`
+
+
+Show tool version and exit
+##### `-s, --silent`
+
+
+Disable log output for commands
+##### `-v, --verbose`
+
+
+Enable verbose logging for commands

docs/xcode-project/junit-test-results.md

@@ -0,0 +1,58 @@
+
+junit-test-results
+==================
+
+
+**Convert Xcode Test Result Bundles (*.xcresult) to JUnit XML format**
+### Usage
+```bash
+xcode-project junit-test-results [-h] [--log-stream STREAM] [--no-color] [--version] [-s] [-v]
+    [-p XCRESULT_PATTERNS]
+    [-d XCRESULT_DIRS]
+    [-o OUTPUT_DIRECTORY]
+    [-e OUTPUT_EXTENSION]
+```
+### Optional arguments for action `junit-test-results`
+
+##### `-p, --xcresult=XCRESULT_PATTERNS`
+
+
+Path to Xcode Test result (\*.xcresult) to be be converted. Can be either a path literal, or a glob pattern to match xcresults in working directory. If no search paths are provided, look for \*.xcresults from current directory. Multiple arguments
+##### `-d, --dir=XCRESULT_DIRS`
+
+
+Directory where Xcode Test results (\*.xcresult) should be converted. If no search paths are provided, look for \*.xcresults from current directory. Multiple arguments
+##### `-o, --output-dir=OUTPUT_DIRECTORY`
+
+
+Directory where the Junit XML results will be saved. Default: `build/ios/test`
+##### `-e, --output-extension=OUTPUT_EXTENSION`
+
+
+Extension for the created Junit XML file. For example `xml` or `junit`. Default: `xml`
+### Common options
+
+##### `-h, --help`
+
+
+show this help message and exit
+##### `--log-stream=stderr | stdout`
+
+
+Log output stream. Default `stderr`
+##### `--no-color`
+
+
+Do not use ANSI colors to format terminal output
+##### `--version`
+
+
+Show tool version and exit
+##### `-s, --silent`
+
+
+Disable log output for commands
+##### `-v, --verbose`
+
+
+Enable verbose logging for commands

docs/xcode-project/run-tests.md

@@ -0,0 +1,133 @@
+
+run-tests
+=========
+
+
+**Run unit or UI tests for given Xcode project or workspace**
+### Usage
+```bash
+xcode-project run-tests [-h] [--log-stream STREAM] [--no-color] [--version] [-s] [-v]
+    [--project XCODE_PROJECT_PATH]
+    [--workspace XCODE_WORKSPACE_PATH]
+    [--target TARGET_NAME]
+    [--config CONFIGURATION_NAME]
+    [--scheme SCHEME_NAME]
+    [--clean]
+    [--disable-coverage]
+    [--graceful-exit]
+    [--max-concurrent-devices MAX_CONCURRENT_DEVICES]
+    [--max-concurrent-simulators MAX_CONCURRENT_SIMULATORS]
+    [-d TEST_DEVICES]
+    [--test-only TEST_ONLY]
+    [--sdk TEST_SDK]
+    [-o OUTPUT_DIRECTORY]
+    [-e OUTPUT_EXTENSION]
+    [--test-flags TEST_FLAGS]
+    [--test-xcargs TEST_XCARGS]
+    [--disable-xcpretty]
+    [--xcpretty-options OPTIONS]
+```
+### Optional arguments for action `run-tests`
+
+##### `--project=XCODE_PROJECT_PATH`
+
+
+Path to Xcode project (\*.xcodeproj)
+##### `--workspace=XCODE_WORKSPACE_PATH`
+
+
+Path to Xcode workspace (\*.xcworkspace)
+##### `--target=TARGET_NAME`
+
+
+Name of the Xcode Target
+##### `--config=CONFIGURATION_NAME`
+
+
+Name of the Xcode build configuration
+##### `--scheme=SCHEME_NAME`
+
+
+Name of the Xcode Scheme
+##### `--clean`
+
+
+Whether to clean the project before building it
+##### `--disable-coverage`
+
+
+Turn code coverage off when testing
+##### `--graceful-exit`
+
+
+In case of failed tests or unsuccessful test run exit the program with status code 0
+##### `--max-concurrent-devices=MAX_CONCURRENT_DEVICES`
+
+
+The maximum number of device destinations to test on concurrently.
+##### `--max-concurrent-simulators=MAX_CONCURRENT_SIMULATORS`
+
+
+The maximum number of simulator destinations to test on concurrently.
+##### `-d, --device=TEST_DEVICES`
+
+
+Test destination description. Either a UDID value of the device, or device name and runtime combination. If runtime is not specified, the latest available runtime for given device name will be chosen. For example "iOS 14.0 iPhone SE (2nd generation)", "iPad Pro (9.7-inch)", "tvOS 14.1 Apple TV 4K (at 1080p)", "Apple TV 4K". If no devices are specified, then the default destination will be chosen (see `xcode-project default-test-destination` for more information about default destination). Multiple arguments
+##### `--test-only=TEST_ONLY`
+
+
+Limit test run to execute only specified tests, and exclude all other tests
+##### `--sdk=TEST_SDK`
+
+
+Name of the SDK that should be used for building the application for testing. Default: `iphonesimulator`
+##### `-o, --output-dir=OUTPUT_DIRECTORY`
+
+
+Directory where the Junit XML results will be saved. Default: `build/ios/test`
+##### `-e, --output-extension=OUTPUT_EXTENSION`
+
+
+Extension for the created Junit XML file. For example `xml` or `junit`. Default: `xml`
+##### `--test-flags=TEST_FLAGS`
+
+
+Pass additional command line options to xcodebuild for the test phase. For example `-derivedDataPath=$HOME/myDerivedData -quiet`.
+##### `--test-xcargs=TEST_XCARGS`
+
+
+Pass additional arguments to xcodebuild for the test phase. For example `COMPILER_INDEX_STORE_ENABLE=NO OTHER_LDFLAGS="-ObjC -lstdc++`.
+##### `--disable-xcpretty`
+
+
+Do not use XCPretty formatter to process log output
+##### `--xcpretty-options=OPTIONS`
+
+
+Command line options for xcpretty formatter. For example "--no-color" or "--simple  --no-utf". Default: `--color`
+### Common options
+
+##### `-h, --help`
+
+
+show this help message and exit
+##### `--log-stream=stderr | stdout`
+
+
+Log output stream. Default `stderr`
+##### `--no-color`
+
+
+Do not use ANSI colors to format terminal output
+##### `--version`
+
+
+Show tool version and exit
+##### `-s, --silent`
+
+
+Disable log output for commands
+##### `-v, --verbose`
+
+
+Enable verbose logging for commands