Author: Chris Timperley (ctimperley at cmu dot edu)
This project provides a reusable and reproducible infrastructure for evaluating C/C++ security vulnerability repair techniques. The project consists of Dockerized repair tools and bug benchmarks, and can accessed either via one monolithic Docker image, containing all tools and bugs, or via separate Docker images for individual bug scenarios.
Out of the box, we provide Dockerized versions of ExtractFix and HiFix, as well as a useful helper script for quickly running experiments using those tools. We also provide the VulnLoc benchmarks.
To build a single Docker image, secbugs
, containing all of the bug scenarios and experimental
infrastructure, execute the following:
$ git submodule update --init --recursive
$ make
Beware that this will take a considerable amount of time (1-3 hours depending on hardware), memory (16 GB is a minimum), and disk space (tens of GBs) to build for the first time. Subsequent builds will be faster since image caching can be used.
Note that you may observe failures when building the complete benchmark Docker image
(e.g., failed to copy non-existent layer for COPY --from=...
instructions).
This is an issue that non-deterministically crops up in Docker when attempting to copy
from large images. The (rather hacky) solution is to simply attempt to rebuild the image.
You may need to attempt building the image several times until it finishes successfully,
but in most cases, you should see progress being made between build attempts.
We provide a convenient executable, secbugs
, within the container.
This command is used to run a given tool against a specified benchmark inside
the container.
To attempt to repair a given bug scenario using a certain tool, run the following:
$ scripts/run.sh
# secbugs repair [name-of-tool] [benchmark-directory]
where [name-of-tool]
is replaced by either hifix
or extractfix
,
and [benchmark-directory]
gives the absolute path to the scenario directory
inside the container.
For example, to fix the jasper/cve_2016_9557
scenario within the VulnLoc dataset using
ExtractFix, execute the following:
$ scripts/run.sh
# cd /benchmarks/jasper/cve_2016_9557
# secbugs repair extractfix .
Running the original, baseline version of Darjeeling requires a different set of steps, described here: https://github.com/squaresLab/security-repair-benchmarks/blob/main/docs/darjeeling.rst
Each bug scenario within this benchmark is given its own directory (e.g., /benchmarks/vulnloc/binutils/cve_2017_14745
),
which provides the files necessary to perform repair on that bug using a variety of tools.
Below is a brief description of the files that MUST be included in each bug scenario directory:
bug.json
provides several details about the bug as well as tool-specific configuration options that should be used when attempting to repair it.test
provides a script for executing the single failing test (i.e., payload) for the bug scenario. The script takes a single, optional argument, providing the absolute path of the binary that should be used to run the test. (This is useful in cases where a repaired binary is created in a different location to the original binary.) If no argument is provided, the script will use the original binary to perform the test.clean
(equivalent tomake clean
) removes any compiled binaries, objects, etc. This is useful for recompiling the program with coverage instrumentation enabled (e.g., when performing repair with Darjeeling).prebuild
(similar to./configure
) appropriately configures the project. Note that this script requires that aREPAIR_TOOL
environment variable is set, stating the repair tool, if any, that is being used in conjunction with the code. This feature is used to compile the program in slightly different ways according to the assumptions of various tools. To build the program without targeting any specific tool, you can just pass inREPAIR_TOOL=none
. (E.g.,REPAIR_TOOL=none ./prebuild
.)build
(similar tomake
) builds the project. As withprebuild
, this step requires that theREPAIR_TOOL
environment variable be appropriately set.
Below is an example of a bug.json
file for coreutils/gnubug_19784
.
{
"subject": "libtiff",
"name": "cve_2016_10092",
"binary": "tools/.libs/tiffcrop",
"options": {
"extractfix": {
"bug-type": "buffer_overflow",
"binary": {
"name": "tiffcrop",
"path": "tools/tiffcrop"
},
"lowfat": {
"CFLAGS": "-fsanitize=lowfat -mllvm -lowfat-symbolize -lstlimpl"
}
},
"hifix": {
"linker-options": "/benchmarks/libtiff/cve_2016_10092/source/libtiff/.libs/libtiff.a -ljpeg -llzma -lm -ljbig -lz"
},
"darjeeling": {
"coverage-files": [
"tools/tiffcrop.c"
]
}
}
}
Here is a brief description of the fields in the above file:
subject
gives the name of the project/program that the bug occurs in (e.g., libtiff, coreutils)name
gives the name of the bug, usually based on its CVE or issue number (e.g., cve_2016_10092)binary
specifies the path of the affected binary, relative to the root of thesource
directory for the bug.options
provides tool-specific options for repairing the bug.
- Jyoti Prakash (jyoti at u dot nus dot edu)