Setup JUnit 5 Tests in RapidWright

RapidWright uses JUnit 5 for Unit Testing. This article aims to give an overview about how to run tests, as well as how to write your own.

Running the Tests

All testcases are located in the test/ directory. JUnit does not need a central list of testcases. Instead, it searches the directory for all classes that contain tests. Tests are marked by annotations (see later). After builing a list of all tests, it executes them one by one.

Some tests depend on DCPs which are stored in a Git submodule — a feature that allows a specific commit of another Git repository to exist as a subdirectory of the current repository. To check out the specific commit of a submodule, run:

git submodule update --init

from the parent RapidWright repository where --init is only strictly necessary (but harmless otherwise) on the first invocation.

To run the tests via Gradle, use the task test or build (which depends on test). After running the tests, Gradle will output the results both as an HTML document in build/reports/tests/test* and as JUnit-internal XML in build/test-results/test*. Note that Gradle knows whether the input to the tests changed and will not rerun them if they are up to date.

There is integration for JUnit in all major IDEs. When loading RapidWright into your IDE, you should set test as source directory for tests. Your IDE should allow you to run all the tests or choose a single class to run. Alternatively, one can execute Gradle from the command line with the testJava or testPython task to run all tests, and restricted to specific tests with one or more --tests <filter> arguments. For example:

./gradlew testJava --tests com.xilinx.rapidwright.design.* --tests *PartNameTools.testGetPartCase

would run all Java test methods under all classes within the com.xilinx.rapidwright.design package, as well as the single test method testGetPartCase from the com.xilinx.rapidwright.device.TestPartNameTools class. Note that the test task depends on testJava and testPython but does not support filtering.

Writing Testcases

JUnit uses Annotations to tag methods as testcases. While there are more specialized annotations, most testcases will be tagged with the annotation @Test (from the org.junit.jupiter.api package).

A test class with a single (empty) test method might look like this:

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
public class MyTestClass {
    @Test
    public void test() {

    }
}

Test methods should be public and cannot be static and not have parameters. JUnit will create an instance of the class, so the class cannot have any constructor parameters.

Testcases communicate failures by throwing an exception. JUnit will then mark it accordingly. Instead of using an if to check for something and then manually creating an exception, you can use the Assertions class (from the package org.junit.jupiter.api). It offers convenience methods for often used checks:

  • assertEquals

  • assertArrayEquals

  • assertNotEquals

  • assertSame

All these methods have a parameter for an expected value and an actual value. Optionally, a message parameter can be passed to explain what part of the test encountered an error.

A very simple test to check that addition works as expected might look like this:

import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
public class MyTestClass {
    @Test
    public void test() {
        Assertions.assertEquals(2, 1 + 1);
    }
}

Parameterized Tests

Normal test methods do not have parameters. If you want to run the same test on a range of data, you can use a loop. However, once the test fails for one set of data, the whole testcase execution is over. Data after the first failure will not be run.

JUnit allows parameters on testcases. They are marked with @ParameterizedTest instead of @Test. The annotation has an optional parameter (name) that allows you to override the generated test’s name to make it more descriptive.

You need to specify a source for values for these parameters. One option is use a separate method that return a Collection<Arguments> or Stream<Arguments>. One instance of Arguments describes one invocation of the testcase method. The value source is specified as another annotation (here: @MethodSource).

A simple example that calls testNonzero(int i) on all numbers from 1 to 10:

import java.util.stream.IntStream;
import java.util.stream.Stream;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;

public class MyTestClass {
    @ParameterizedTest(name = "Check that {0} is nonzero")
    @MethodSource()
    public void testNonzero(int i) {
        Assertions.assertNotEquals(0, i);
    }

    public static Stream<Arguments> testNonzero() {
        return IntStream.rangeClosed(1, 10).mapToObj(i -> Arguments.of(i));
    }
}

RapidWright-specific Considerations

RapidWright’s tests are automatically run on Github Actions. There are rather strict restrictions in terms of maximum memory (7GB) and some parts of RapidWright can exceed that limit. You should keep this limitation in mind while writing testcases:

  • Testcases should be limited to a single Device. If you have to use multiple Devices, take care that only one Device is referenced at the same time.

  • When instantiating a Design, use a small Device for it.

To identify issues with files being left open, there is a JUnit extension that compares the list of open files before and after a testcase. It will fail the testcase if there are changes. This extension (com.xilinx.rapidwright.support.CheckOpenFilesExtension) is automatically registered when JUnit tests are run with Gradle.

Testcase DCPs

Tests requiring new DCP(s) will need to fork the RapidWrightDCP repository to gain write permissions.

The DCP(s) to be added should have no encrypted components inside and the EDIF inside the DCP should be readable (not encrypted). A readable EDIF file can be generated using Vivado either automatically upon load in RapidWright or via write_edif (see RapidWright and Design Checkpoint Files). Use the ReplaceEDIFInDCP tool to replace the EDIF inside a DCP, for example:

rapidwright ReplaceEDIFInDCP design.dcp readable_design.edf

will replace the EDIF file inside design.dcp with readable_edif.edf.

Next, execute the following:

# Add, commit, push new DCP(s) into new branch on fork
cd test/RapidWrightDCP
git remote add fork https://github.com/<user>/RapidWrightDCP  # Only necessary first invocation
git checkout -b <branch>
git add <dcp_name>
git commit
git push -u fork <branch>
cd ../..

# Commit new submodule reference
git commit test/RapidWrightDCP -s -m "(Description)"

The submodule can now be used as a regular Git repository during development; remember to commit new submodule references from the RapidWright repository using:

git commit test/RapidWrightDCP -s -m "(Description)"

Once ready, please create new pull requests in both the upstream RapidWright and RapidWrightDCP repositories. When both pull requests have been approved, the following situation will be present:

RapidWrightDCP (upstream) ... o--o---------------x
                                  \             / (PR#123)
RapidWrightDCP (fork)          ... o--o ... o--o
                                               ^ (commit `abc`)


   RapidWright (upstream) ... o--o---------------x
                                  \             / (PR#456)
   RapidWright (fork)          ... o--o ... o--o
                                               ^ (submodule refers to commit `abc`
                                                  on RapidWrightDCP fork)

Here, RapidWright’s PR#456 refers to commit abc which is present only on the fork. The expectation would be that the RapidWrightDCP’s PR#123 would be merged first after which PR#456 can then update its RapidWrightDCP submodule reference to include upstream’s newly merged result:

RapidWrightDCP (upstream) ... o--o---------------o (commit `def` including
                                  \             /   PR#123)
RapidWrightDCP (fork)          ... o--o ... o--o


   RapidWright (upstream) ... o--o------------------o
                                  \             /  /  (PR#456)
   RapidWright (fork)          ... o--o ... o--o--o
                                                  ^ (submodule updated to commit `def`
                                                     on RapidWrightDCP upstream)

This submodule reference can be updated back to upstream as follows:

# Return submodule to upstream master
cd test/RapidWrightDCP
git checkout master
git pull
cd ../..

# Commit new submodule reference
git commit test/RapidWrightDCP