To catch regressions in our code base, we run thousands of tests on all the packages in the ROS distributions. The tests range from small unit tests, to large simulation based integration tests. Our Jenkins server manages and schedules many of the tests. Below you find more details on each class of test that we run.
Contents
Development Tests
What gets tested?
The development tests will build and test all packages from the development branch of each repository. The development branch is specified in the source sections of the distribution.yaml rosdistro file (One from ROS Indigo | ROS Kinetic). The devel branch is chosen by the maintainer of a repository, and is often trunk/default/master or <DISTRO>-devel. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Currently the development tests are only run for Ubuntu LTS releases for amd64 due to limited resources on the build farm.
When do tests run?
Every time you commit to a development branch the corresponding development tests will get triggered within an hour. It might take a while until the job is processed if the build farm is busy though. For example, you can find the development jobs for Indigo on Jenkins.
Who gets a notification?
The person that did the commit will receive an email if the build succeeds or fails. If the development job is marked as unstable (due to warnings or failing tests) all committers since the last stable build get notified.
How long does it take to run them?
One test typically takes less than 10 minutes. If the build farm is busy rebuilding Debian packages it might take longer until the job actually gets started.
Pull Request Tests
Pull request tests are identical to development tests but they are being triggered for the branches of pull requests. Currently only GitHub repositories are supported. Commonly pull requests need to be specifically enabled (see REP 143). You can find the pull request jobs for Indigo on Jenkins.
Prerelease Tests
Before pushing out a new release of your package(s), you should run the prerelease tests on your repository. This will help you to avoid breaking the ROS Debian builds for the entire ROS community, and having to push out a bugfix release five minutes later. Starting with ROS Jade, prereleases are run locally on your machine using docker. There is a webpage to help you configure the command line tool that runs the prerelease tests on your machine: http://prerelease.ros.org/.
What gets tested?
For the selected repositories (you can select one or multiple) all packages will be built and the tests will be run. All necessary dependencies will be installed from Debian packages in the shadow-fixed repo before. Additionally, it can download, build, and test packages which depend on your packages. The prerelease website can help you to figure out which settings will test the right amount of packages for you. You can control the number of packages that will be tested by adjusting level of recursive dependent packages. You can additionally specify arbitrary additional packages to test. If you want to avoid building a package, for example a package which is known to be broken for another reason, you can also explicitly exclude individual packages, though this might prevent the testing of other packages. The prerelease website can help you see how different settings affect the packages that will be tested.
Which repositories can be tested?
For repositories of catkin packages, you can choose between multiple versions:
version number: a specific released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file
latest: the latest released version of the repo/stack which is fetched from the release repository as listed in the distribution.yaml rosdistro file. While the distribution.yaml file might not contain a version the latest released version from the GBP repository will be used.
devel: the development branch which is fetched from the source repository as listed in the same file. If the source rosdistro file does not contain a version for the repository the default branch is used. Note that this file is manually generated and not part of the automatic bloom release process - you will need to seed it for your repositories with a pull request.
- All dependent repos/stacks will always be fetched from their released version as listed in the rosdistro file
Here is an example "repository" entry in the distribution.yaml rosdistro file:
1 rviz: # <-- this is the repository's name 2 doc: # <-- this describes what to checkout for doc generation 3 type: git 4 url: https://github.com/ros-visualization/rviz.git 5 version: hydro-devel 6 release: # <-- this is the "release" entry 7 tags: 8 release: release/hydro/{package}/{version} 9 url: https://github.com/ros-gbp/rviz-release.git 10 # ^-- url to the release repository 11 version: 1.10.18-0 # <-- maps to "latest" 12 source: # <-- this is the "source" entry 13 # ^-- maps to the "devel" prerelease option 14 type: git 15 url: https://github.com/ros-visualization/rviz.git 16 version: hydro-devel 17 status: maintained
When do tests run?
The prerelease tests only run when you run them on your local machine.
How long does it take to run them?
Low level repositories can take more than an hour to test, while high level repositories can be finished in 5 minutes. The test duration mainly depends on how many repositories depend on the repositories being tested. You can use the prerelease website to help you adjust the number of packages that are tested and built to avoid extremely long builds due to a large number of dependent packages being tested.
Prerelease Instructions
Setting up Your Computer for Prerelease
First, you need to install Docker. We recommend following this guide at docker.io because you'll need a newer version of docker than the default version that comes with Ubuntu 14.04. Make sure to follow the directions to add your user to the docker group so you can invoke docker without sudo.
Install the ros_buildfarm package to get the scripts and tools you need to run the prerelease:
$ sudo apt-get install python3-ros-buildfarm
If you haven't installed ROS on your machine before, you'll also need to setup apt-get sources to include packages.ros.org. Run the following commands.
$ sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list' $ sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116 $ sudo apt-get update
You can also install the Python 2 equivalent of the buildfarm, which is called python-ros-buildfarm. For the Python 2 package the template evaluation of the prerelease scripts still use Python 3 internally. Therefore you'll need to have EmPy for Python 3 installed. The Debian package is named python3-empy and is available from Ubuntu Trusty onward. For older versions of Ubuntu, or on other platforms, it can be installed with the following commands:
$ sudo apt-get install python3 python3-pip $ sudo python3 -m pip install -U EmPy
Creating a Prerelease Command
Once you've set up your computer, use the prerelease website to help you generate a command to run the prerelease: http://prerelease.ros.org/
First, the website will prompt you to select a ROS distribution. Select whichever distribution you want to target with your new package. Next, the site will guide you through a 3-step process to generate a terminal command.
Select Repositories. In this step, you add packages to the prerelease. You're essentially making a mini build of ROS, and you don't want to build and test every package! So in this step, you'll add your new package(s) to the list. If your package is already on the build farm, you should be able to find it in the drop-down list. If your package is brand new or otherwise doesn't appear in the list, you should use the "Add a custom repository" button and fill in the requested information. When you're done, click Next.
Additional Options. In this step, you can choose a custom build farm config file (for advanced users). You should be able to use the default one without issues. You can also choose an operating system to target. This can be useful to test your release on a different OS than the one you're using for local development. Once you've chosen an OS, click Next.
Select Dependents to Test. This step may take a bit of time to load. If your package is brand new, you can ignore the rest of this step and go to Step 4. If your package is already on the build farm and has dependencies, this step will automatically add those dependencies to your prerelease for testing, to ensure that your package's changes didn't break those packages. You can add or remove dependencies for testing as desired (but you should probably test all of them).
- The wizard is complete. Click the Generate Command button and copy the commands into your terminal and run them.
Running the Prerelease
Wait, you still haven't done the prerelease yet! Look at the terminal output carefully. The custom command you ran just created prerelease.sh. To actually run the prerelease, run that file:
$ ./prerelease.sh
Be patient, the command takes a while. The prerelease takes place in a Docker container, and should automatically build your packages, test them, and print the results using catkin_test_results. Make sure that there are no failed tests, and your prerelease is done!
After prerelease, you can run catkin_test_results to get the result of the prerelease tests again. This is an optional step. In order to run that command you need catkin installed and sourced. You'll have to have ROS and catkin installed. Install catkin with the command
$ sudo apt-get install ros-kinetic-catkin
At some point, you also need to source the setup.bash file before you try to use catkin_test_results, by using the command
$ source /opt/ros/kinetic/setup.bash
You can do this at any point, rest assured that your system and environment will not affect the prerelease since they are executed inside the docker container.
Examples
Below a few common prerelease examples are described. The ros_buildfarm documentation contains additional information and examples on prereleases: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/jobs/prerelease_jobs.rst
A set of commands the website provides
The command set provided by the prerelease test website above would look like the following. Variables that you can configure on the website are surrounded with %%.
mkdir -p /tmp/prerelease_job cd /tmp/prerelease_job generate_prerelease_script.py \ https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \ %ROS_DISTRO% default ubuntu %UBUNTU_DISTRO% amd64 \ %PACKAGE_TARGETED% \ --level %DOWNSTREAM_DEPTH% \ --output-dir ./
- A concrete example:
mkdir -p /tmp/prerelease_job cd /tmp/prerelease_job generate_prerelease_script.py \ https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml \ kinetic default ubuntu xenial amd64 \ moveit \ --level 0 \ --output-dir ./
In this example, we're testing: - %ROS_DISTRO%: ROS Kinetic
%UBUNTU_DISTRO%: Ubuntu xenial
%PACKAGE_TARGETED%: moveit
- %DOWNSTREAM_DEPTH%: 0 (i.e. we're not testing if the tests in the downstream packages pass. Depending on the number of packages, the downstream testing can take long)
Customizing prerelease test commands
As you see, the configuration of prerelease test is done by a script generate_prerelease_script.py (that was installed on your computer with python3-ros-buildfarm package). See its option with the help to get some idea for customization.
generate_prerelease_script.py --help
usage: generate_prerelease_script.py [-h] --output-dir OUTPUT_DIR
[--custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]]
[--custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]]
[--pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
[--exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]]
[--level DEPENDENCY_LEVEL]
config_url rosdistro_name
source_build_name os_name os_code_name
arch [REPO_NAME [REPO_NAME ...]]
Generate a 'prerelease' script
positional arguments:
config_url The url of the ROS buildfarm configuration index
rosdistro_name The name of the ROS distro from the index
source_build_name The name / key of the 'source-build' file from the
index
os_name An OS name from the build file
os_code_name An OS code name from the build file
arch An architecture from the build file
optional arguments:
-h, --help show this help message and exit
--output-dir OUTPUT_DIR
The output directory
Repositories in underlay workspace:
The repositories in the underlay workspace will be built and installed as
well as built and tested. Dependencies will be provided by binary
packages.
REPO_NAME A name of a 'repository' from the distribution file
--custom-branch [REPO_NAME:BRANCH_OR_TAG_NAME [REPO_NAME:BRANCH_OR_TAG_NAME ...]]
A name of a 'repository' from the distribution file
followed by a colon and a branch / tag name
--custom-repo [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME [REPO_NAME:REPO_TYPE:REPO_URL:BRANCH_OR_TAG_NAME ...]]
The name, type, url and branch / tag name of a repository,
'e.g. "common_tutorials:git:https://github.com/ros/common_tutorials:pullrequest-1"')
Packages in overlay workspace:
The packages in the overlay workspace will be built and tested. The
workspace will be extended with packages from GBP repositories for
dependencies between the overlay and underlay workspaces. All other
dependencies will be provided by binary packages.
--pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
A name of a released 'package' from the distribution
file to be included in the overlay workspace
--exclude-pkg [PACKAGE_NAME [PACKAGE_NAME ...]]
A name of a released 'package' from the distribution
file which should be excluded from the overlay
workspace
--level DEPENDENCY_LEVEL
The depth of the depends-on tree to be included in the
overlay workspace (-1 implies unlimited, default: 2)For example,
- to run prerelease test using a pull request that's not yet merged, you can do something like this:
generate_prerelease_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml indigo default ubuntu trusty amd64 moveit --level 0 --custom-repo moveit:git:https://github.com/130s/moveit:fix/5 --output-dir ./
What can I do when a prerelease fails?
Once the generate prerelease command is finished, it will instruct you to run the ./prerelease.sh shell script. This in turn runs a shell script for each step of the prerelease. If a step fails, then you can edit the files which are local (not stored inside the docker container) and then each step of the prerelease process can be rerun by executing the corresponding shell script. These are the steps that you might want to rerun:
prerelease_clone_underlay.sh: checkouts source code for the repositories you specified
prerelease_clone_overlay.sh: checkouts source code for the dependent packages you are going to test on top of the repositories you specified
prerelease_build_underlay.sh: builds and tests the packages in the repositories you specified
prerelease_build_overlay.sh: builds and tests the dependent packages on top of the repositories built and tested in the "build_underlay" step
If you look in the folder where you ran the prerelease then you'll also notice some folders of interest when debugging:
catkin_workspace: "underlay" catkin workspace in which the repositories you selected are built and tested
catkin_workspace_overlay: catkin workspace in which dependent packages are built; it is chained on top of the "underlay" catkin workspace
You can modify code in these two repositories and rerun one or more of the above shell scripts to iterate on the prerelease test and try to resolve issues.
