fixcheck

FixCheck is a tool for improving patch correctness analysis. Given a target Java patch, it uses static analysis and random testing to generate new inputs to test the patch, and LLMs to generate meaningful assertions for the new inputs. The new tests are executed and those that fail are selected and prioritised according to their likelihood of revealing a defect in the patch.

---

Setting up FixCheck

Requirements

• Java >= 17 (tested with 17)
• Python3 (tested with 3.11.5)
• Ollama (tool providing access to open source LLMs)

Local Installation

To install FixCheck, clone the repository, build the project with gradle and install the python requirements:

git clone https://github.com/facumolina/fixcheck
cd fixcheck
./gradlew shadowJar 
pip3 install -r experiments/requirements.txt
pip3 install -r llms/requirements.txt

Set up the FixCheck environment variable:

export FIXCHECK=<path_to_thisrepo>

Although FixCheck support multiple LLMs (and can be extended), it is necessary to download the LLMs that will be used, or interact with them through an API. Here we provide instructions on the currently supported LLMs:

replit-code

To use this model, other python requirements are needed, which, for the sake of simplicity, are not included in the requirements.txt files. To install them, run the following command:

pip3 install sentencepiece==0.1.99
pip3 install torch==2.0.1
python3 llms/replit-code.py

codellama

Codellama needs to be downloaded using Ollama:

ollama run codellama

Note

This command will download the 7B version of the codellama model. Other version can be downloaded by specifying the version in the command, but in order to use other versions you will need to extend FixCheck as described in the Extending FixCheck section.

After setting any of the LLMs, FixCheck is ready to be used.

Docker

We provide a Dockerfile that can be used to build a docker image with FixCheck and all its dependencies. To build and run the docker image, execute the following commands:

docker build -t fixcheck .
docker run -it fixcheck

Note

Building the image may take a while, as it will download and install all the dependencies for running FixCheck, and also to analyze the example patches discussed below.

Note

Also, no LLM will be configured in the Docker image. Thus, the user will need to set up the LLMs as described in the previous section.

---

Using FixCheck

In general, to analyse a patch with FixCheck the following steps are needed:

1. Build the buggy version of the project corresponding to the patch under analysis
2. Run an initial bug revealing test to collect its failure trace
3. Apply the patch to the project and built the patched version
4. Run FixCheck with the right properties

Example

This section contains a simple example analysing a patch from DefectRepairing, a repository containing benchmark of correct and incorrect patches for defects4j bugs and generated by APR tools, used in the evaluation of PATCH-SIM (a patch correctness assessment tool).

Note

If you are using the Docker image, defects4j and DefectRepairing are already installed, so you can move to the setup step.

To run FixCheck on any of these patches, perform the following steps:

1. Download and install defects4j (make sure the command defects4j is available)
2. Download the DefectRepairing benchmark, and set the environgment variable DEFECT_REPAIRING_DATASET to the path where the benchmark is located.

Before executing FixCheck, a setup step is needed, that essentially will clone the project corresponding to the patch under analysis, apply the patch and build the project. For instance, to analyse the Patch1, the setup can be performed by running:

python3 experiments/setup-defect-repairing.py Patch169

Important

When running the setup script, Java 8 should be configured, as it is required to build defects4j projects.

From this point, we can now execute FixCheck as follows:

python3 experiments/run-fixcheck-defect-repairing.py Patch169 codellama

The script will automatically extract the arguments from the file experiments/defect-repairing-subjects.csv, and call FixCheck with the right properties. For instance, for the Patch1 subject the executed command was the following:

java -cp build/libs/fixcheck-all-1.0.0.jar:$DEFECT_REPAIRING_DATASET/tmp/Patch169/Math69b/target/classes:$DEFECT_REPAIRING_DATASET/tmp/Patch169/Math69b/target/test-classes org.imdea.fixcheck.FixCheck 

, where the file fixcheck.properties contained the following properties:

test-classes-path=$DEFECT_REPAIRING_DATASET/tmp/Patch169/Math69b/target/test-classes
test-class=org.apache.commons.math.stat.correlation.PearsonsCorrelationTest
test-methods=testPValueNearZero 
test-classes-src=$DEFECT_REPAIRING_DATASET/tmp/Patch169/Math69b/src/test/java 
inputs-class=int
test-failure-trace-log=$DEFECT_REPAIRING_DATASET/tmp/Patch169/Math69b/failing_tests 
number-of-prefixes=100 
assertion-generator=codellama

Once it finishes, the results will be stored in the folder fixcheck-output/defects-repairing. Other patches from the DefectRepairing benchmark can also be analysed following the same procedure, as they are all configured in the csv file experiments/defect-repairing-subjects.csv.

FixCheck Properties

Property	Description
test-classes-path	Path to the test classes directory
test-classes-src	Path to the test classes sources directory
test-class	Fully qualified name of the target test class
test-methods	List of names of the initial fault revealing test methods, seperated by ':'
test-failure-trace-log	File containing the failure trace of the target test method
inputs-class	Name of the inputs class (int,float,double,java.lang.String, etc)
number-of-prefixes	Number of prefixes variations to generate
assertion-generator	Assertion generator class fully qualified name (e.g., org.imdea.fixcheck.assertion.ReplitCodeLLM)

Extending FixCheck

FixCheck can be extended by adding new assertion generators, e.g., based on other LLMs or other techniques. To extend FixCheck with a new assertion generator, the following steps are needed:

1. Create a new class that extends org.imdea.fixcheck.assertion.AssertionGenerator
2. Implement the generateAssertions method, which, given a test prefix, returns a list of assertions for it.
3. Invoking FixCheck with the assertion-generator property defined with the fully qualified name of the new assertion generator.

Examples of different assertion generators can be found in the package org.imdea.fixcheck.assertion.

---

Contact

If you experience any issues, please submit an issue or contact us at [email protected]!