Skip to content

Commit

Permalink
Add the benchmarking config used in the announcement post on swift.org
Browse files Browse the repository at this point in the history
  • Loading branch information
lorentey committed Apr 5, 2021
1 parent 6860968 commit bd92400
Show file tree
Hide file tree
Showing 12 changed files with 226 additions and 0 deletions.
62 changes: 62 additions & 0 deletions Documentation/Announcement-benchmarks/Library.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"kind": "group",
"title": "All results",
"directory": "Results",
"contents": [
{
"kind": "chart",
"title": "Deque: random access lookups",
"tasks": [
"Deque<Int> subscript get, random offsets (discontiguous)",
"Array<Int> subscript get, random offsets",
"std::deque<intptr_t> at, random offsets",
]
},
{
"kind": "chart",
"title": "Deque: prepending individual integers",
"tasks": [
"Deque<Int> prepend, reserving capacity",
"Array<Int> prepend, reserving capacity",
"std::deque<intptr_t> push_front",
]
},
{
"kind": "chart",
"title": "OrderedSet lookups",
"tasks": [
"OrderedSet<Int> successful contains",
"Set<Int> successful contains",
"std::unordered_set<intptr_t> successful find",
"Array<Int> successful contains"
]
},
{
"kind": "chart",
"title": "OrderedSet insertions",
"tasks": [
"OrderedSet<Int> append, reserving capacity",
"Set<Int> insert, reserving capacity",
"std::unordered_set<intptr_t> insert, reserving capacity",
]
},
{
"kind": "chart",
"title": "OrderedDictionary lookups",
"tasks": [
"OrderedDictionary<Int, Int> subscript, successful lookups",
"Dictionary<Int, Int> subscript, successful lookups",
"std::unordered_map<intptr_t, intptr_t> successful find",
]
},
{
"kind": "chart",
"title": "OrderedDictionary insertions",
"tasks": [
"OrderedDictionary<Int, Int> subscript, append, reserving capacity",
"Dictionary<Int, Int> subscript, insert, reserving capacity",
"std::unordered_map<intptr_t, intptr_t> insert, reserving capacity",
]
},
]
}
17 changes: 17 additions & 0 deletions Documentation/Announcement-benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Benchmark data from the swift.org announcement

This directory contains the benchmark configuration that we used to collect benchmarking data and render the charts in the [Swift Collections announcement on swift.org][announcement].

* [`generate-results.sh`](./generate-results.sh): A script that you can use to run the benchmark on your own machine, and to reproduce the charts.
* [`Library.json`](./Library.json): The benchmark library definition collecting the chart definitions used in the blog post.
* [`Theme.json`](./Theme.json): The chart theme to set up colors, line widths, font sizes etc. (Slightly adapted.)
* [`Results.md`](./Results.md): The generated results summary.
* [`Results/`](./Results/): The generated subdirectory containing the PNG files for the charts.
* [`results.json`](./results.json): The result data we collected.

Note: If you'd like to try reproducing our results (it's easy!), note that the script is configured to collect 20 rounds of data on up to 16 million items, and this will likely take a very long time. Feel free to reduce the maximum size, or just plan to allow the measurement run overnight.

For more information on how to use the Swift Collections benchmarking tool, please see [Swift Collections Benchmark][swift-collections-benchmark].

[announcement]: https://swift.org/blog/swift-collections
[swift-collections-benchmark]: https://github.com/apple/swift-collections-benchmark
32 changes: 32 additions & 0 deletions Documentation/Announcement-benchmarks/Results.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Benchmark results

Click to expand individual items below.
<details open>
<summary><strong>All results</strong></summary>
<ul>
<details>
<summary>01: Deque: random access lookups</summary>
<img src="Results/01%20Deque:%20random%20access%20lookups.png">
</details>
<details>
<summary>02: Deque: prepending individual integers</summary>
<img src="Results/02%20Deque:%20prepending%20individual%20integers.png">
</details>
<details>
<summary>03: OrderedSet lookups</summary>
<img src="Results/03%20OrderedSet%20lookups.png">
</details>
<details>
<summary>04: OrderedSet insertions</summary>
<img src="Results/04%20OrderedSet%20insertions.png">
</details>
<details>
<summary>05: OrderedDictionary lookups</summary>
<img src="Results/05%20OrderedDictionary%20lookups.png">
</details>
<details>
<summary>06: OrderedDictionary insertions</summary>
<img src="Results/06%20OrderedDictionary%20insertions.png">
</details>
</ul>
</details>
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
98 changes: 98 additions & 0 deletions Documentation/Announcement-benchmarks/Theme.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
{
"axisLabels" : {
"color" : "#000000FF",
"font" : {
"family" : "Helvetica",
"isBold" : false,
"isItalic" : false,
"size" : 13
}
},
"axisLeading" : 3,
"background" : "#FFFFFFFF",
"bandDimmingFactor" : 0.29999999999999999,
"border" : {
"capStyle" : "round",
"color" : "#000000FF",
"joinStyle" : "round",
"width" : 0.5
},
"curveFallback" : {
"color" : "#000000FF",
"lineWidth" : 4
},
"curves" : [
{
"color" : "#F05138FF",
"lineWidth" : 8
},
{
"color" : "#FFC07FFF",
"lineWidth" : 6
},
{
"color" : "#FFE57FFF",
"lineWidth" : 6
},
{
"color" : "#93E6C0FF",
"lineWidth" : 6
},
],
"hairlines" : {
"capStyle" : "round",
"color" : "#0000004D",
"joinStyle" : "round",
"width" : 0.5
},
"legendCornerOffset" : [
24,
24
],
"legendLabels" : {
"color" : "#000000FF",
"font" : {
"family" : "Menlo",
"isBold" : false,
"isItalic" : false,
"size" : 13
}
},
"legendLineLeading" : 3,
"legendLineSampleWidth" : 24,
"legendPadding" : {
"bottom" : 12,
"left" : 12,
"right" : 12,
"top" : 12
},
"legendPosition" : "topLeft",
"legendSeparation" : 9,
"majorGridline" : {
"capStyle" : "butt",
"color" : "#0000007F",
"joinStyle" : "round",
"width" : 0.75
},
"margins" : {
"bottom" : 9,
"left" : 9,
"right" : 9,
"top" : 9
},
"minorGridline" : {
"capStyle" : "butt",
"color" : "#00000042",
"dash" : {
"phase" : 0,
"style" : [
6,
3
]
},
"joinStyle" : "round",
"width" : 0.5
},
"xPadding" : 6,
"yPadding" : 3
}
16 changes: 16 additions & 0 deletions Documentation/Announcement-benchmarks/generate-results.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/bin/sh
#===----------------------------------------------------------------------===//
#
# This source file is part of the Swift Collections open source project
#
# Copyright (c) 2021 Apple Inc. and the Swift project authors
# Licensed under Apache License v2.0 with Runtime Library Exception
#
# See https://swift.org/LICENSE.txt for license information
#
#===----------------------------------------------------------------------===//

set -eux

../../Utils/run-benchmarks.sh library run results.json --library Library.json --max-size 16M --cycles 20 --mode replace-all
../../Utils/run-benchmarks.sh library render results.json --library Library.json --max-time 10us --min-time 1ns --theme-file Theme.json --percentile 90 --output .
1 change: 1 addition & 0 deletions Documentation/Announcement-benchmarks/results.json

Large diffs are not rendered by default.

0 comments on commit bd92400

Please sign in to comment.