Instructions for building sbncode

Table of Contents

1. Overview
2. Sbncode packages
   1. Sbnd and Icarus packages
   2. Sbn suite
3. Step by step instructions for building sbncode releases
   1. Set up a new mrb test release
   2. Check out all sbncode and related packages
   3. Prepare a working branch for each package
   4. Update dependent package versions
   5. Merge approved pull requests
   6. Do initial test build
   7. Update sbncode package versions
   8. Do final test build
   9. Commit changes and make tags
   10. Do Jenkins build
   11. Upload built packages to SciSoft
   12. Upload built packages to cvmfs
   13. Create a suite tag
   14. Update develop branch
   15. Final checks
   16. Generate release notes
   17. Update the sbncode release list
   18. Send announcement
4. Shortcuts and helper scripts
   1. update_larsoft_version.sh
   2. update_sbncode_version.sh
   3. makeSBNRelNotes

Overview

This page contains detailed instructions for building ups style / SL7 sbncode releases. It does not cover everything in Miquel’s release manager instrucions.

Sbncode packages

Sbncode releases generally consist of the following four packages, which are listed below in dependency order, low to high.

• sbnobj
• sbnanaobj
• sbnalg
• sbncode

Normally sbncode releases are only concerned with updating and tagging these four packages. For a list of related packages that may need to be built separately, refer to Miquel’s release manager instrucions.

Sbnd and Icarus packages

It is often desirable to check out and test sbnd or icarus packages together with sbncode packages. Here are the related packages for sbnd.

• sbndutil
• sbndcode

Here are the related packages for icarus.

• icarusutil
• icarus_signal_processing
• icarusalg
• icaruscode

SBN suite

When checking out packages using mrb, you can check out all sbn-related packages using sbn_suite.

$ mrb g sbn_suite

Checking out sbn_suite will check out all of the sbncode, sbnd, and icarus packages listed above, plus package sbnci. However, it is not generally possible to build sbnd and icarus packages in the same mrb test release. Therefore, if you check out sbn_suite, you should prune the checked out packages down to just the ones you are interested in.

Step by step instructions for building sbncode releases

This section contains step-by-step instructions for building sbncode releases.

Set up a new mrb test release

Log in to your own account on one of the sbnd or icarus gpvm’s (or any other compatible computer) and set up the standard software development environment for that experiment.

$ sh /exp/sbnd/data/users/vito/podman/start_SL7dev_jsl.sh
$ source /cvmfs/sbnd.opensciencegrid.org/products/sbnd/setup_sbnd.sh

or

$ sh /exp/icarus/data/users/vito/podman/start_SL7dev_jsl.sh
$ source /cvmfs/icarus.opensciencegrid.org/products/icarus/setup_icarus.sh

Make a new mrb test release against an existing larsoft base release.

$ mkdir <some empty directory>
$ cd <some empty directory>
$ mrb n -v <larsoft-version> -q <larsoft-qualifiers> [-f]    # Use -f to use existing srcs.
$ source localProducts*/setup

Check out all sbncode and related packages

Check out all sbncode packages. Optionally check out either sbnd or icarus packages for testing.

$ cd $MRB_SOURCE
$ mrb g sbn_suite
$ rm -rf <unneeded packages>
$ mrb uc

Prepare a working branch for each package

Visit each checked out package and make sure that the correct working branch is checked out. For integration releases, check out branch “main” for sbncode packages. Sbnd and icarus packages that are checked out for testing should generally have branch “develop” checked out. For production releases, check out the appropriate production branch. If you are working with an existing mrb test release, make sure to “git pull” the working branch.

$ cd $MRB_SOURCE/<package> 
$ git checkout <working branch>
$ git pull

For integration releases, for each sbncode package, merge already commited updates from develop branch to main.

$ git merge origin/develop
$ git merge develop

Update dependent package versions

Update the larsoft base version, if necessary. Use command “mrb uv.”

$ mrb uv larsoft <larsoft version>

Update any other changed dependent package versions and fix any remaining version conflicts. In general, as long as the larsoft version has been properly updated, any remaining version conflicts can be discovered by initializing the build environment using mrbsetenv.

$ mrb uv <package> <version>

Merge approved pull requests

You can see a list of open pull requests for any package using the following command, or using the github web interface.

$ cd $MRB_SOURCE/<package>
$ gh pr list

To include a github pull request in your test release, first check out the pull request into a local branch, then merge the pull request branch into your working branch.

$ gh pr checkout <number>
$ git checkout <working-branch>
$ git merge <pr-branch>

Do initial test build

If there are any additional required updates that are not covered in the previous sections, do them now. After all such updates, do a full build-and-test.

$ cd $MRB_BUILDDIR
$ mrbsetenv
$ mrb i -jN
$ mrb t -jN

If there are errors at this point, fix them before proceeding.

Update sbncode package versions

Make a “notag” directory adjacent to $MRB_SOURCE.

$ cd $MRB_SOURCE
$ mkdir ../notag

Visit each sbncode package in dependency order (low to high), ending with package sbncode. Compare each package to the previous suite tag.

$ git diff SBN_SUITE_vxx_yy_zz

If the package does not have any updates. move it out of $MRB_SOURCE.

$ cd $MRB_SOURCE
$ mv <package> ../notag

If the package does have updates, update its version using command “mrb uv.”

$ mrb uv <package> <new version>

Do final test build

After making all version updates, and after removing unneeded packages from $MRB_SOURCE, update the master CMakeLists.txt and do another clean build and test.

$ cd $MRB_BUILDDIR
$ mrb uc
$ mrb z
$ mrb zi
$ mrbsetenv
$ mrb i -jN
$ mrb t -jN

Again, if there are errors, they need to be fixed before proceeding.

Commit changes and make tags

For each package that is still checked out in $MRB_SOURCE, commit changes and make a tag on the working branch. Push the working branch and tag to the main repository.

$ cd $MRB_SOURCE/<package>
$ git status
$ git add <modified files>
$ git commit -m<version>
$ git tag -a -m<version> <version>
$ git push origin <working branch>
$ git push origin <version>

Do Jenkins build

Use Jenkins build project sbn-release-build for both integration releases and production releases. The configuration of this build project includes the following parameters which may need updating.

• Ups version.
• sbncode package tags (four parameters).
• sbndaq_artdaq_core version.
• s-qualifier

The base qualifier and build type (debug, prof) are included in the Jenkins configuration matrix.

Upload built packages to SciSoft

After a successful Jenkins build, fetch build artifacts (tarballs and manifests) using copyFromJenkins. Add “-q” options (repeatable) for any qualifiers that are part of the configuration matrix.

$ copyFromJenkins -q e26 sbn-release-build

Upload tarballs and manifests to the scisoft products server using copyToSciSoft.

$ copyToSciSoft *

Note that scripts copyFromJenkins and copyToSciSoft can be downloaded from https://scisoft.fnal.gov/scisoft/bundles/tools/ .

Upload built packages to cvmfs

Log in to cvmfssbn@oasiscfs.fnal.gov.

$ ssh cvmfssbn@oasiscfs.fnal.gov

Start a cvmfs transaction and run the install script.

$ cvmfs_server transaction sbn.opensciencegrid.org
$ ~/sbnbuild/CVMFS/install_on_cvmfs.sh sbn-xx.yy.zz 
$ cvmfs_server publish sbn.opensciencegrid.org

Create a suite tag

Create a suite tag SBN_SUITE_ for each sbncode package, whether updated or not.

$ cd $MRB_SOURCE/<package>
$ git tag -a -m"Sbn suite vxx_yy_zz" SBN_SUITE_vxx_yy_zz
$ git push origin SBN_SUITE_vxx_yy_zz

Update develop branch

For integration releases, merge updates from branch main to develop.

$ cd $MRB_SOURCE/<package>
$ git checkout develop
$ git merge main
$ git push origin develop

Final checks

• Make sure that newly released sbncode version is able to be set up.
• Verify that pull requests you intended to merge are closed.

Generate release notes

• Navigate to the main sbncode github web page.
• Click on Releases
• Click on button Draft a new release.
• Choose the correct tag from the pull down menu.
• Fill in the release title box with the tag.
• Fill in the release description box.
• Use the preview tab to make sure everything looks OK.
• Click on the “Publish release” button.

Update the sbncode release list

• Navigate to the sbncode release list wiki page.
• Click on button Improve this page
• Add a line in the table of releases with link to newly generated release notes.
• Commit update.

Send announcement

Announce the new release on slack channel #sbn_release_management or in other appropriate ways.

Shortcuts and helper scripts

The procedures described in the previous sections are intended to be self-contained. This section describes some helper scripts that can partially automate some of the steps described above. The scripts described in this section are stored in sbn github package sbnbuild. These scripts are not currently available as ups products, but direct download links are included below.

Here are the helper scripts described in this section, with direct download links.

• update_larsoft_version.sh
• update_sbncode_version.sh
• makeSBNRelNotes

update_larsoft_version.sh

Script update_larsoft_version.sh updates the version of larsoft and all of its dependent packages. It eliminates the need to manually resolve most version conflicts. This script should be invoked before the initial test build. It is invoked as follows.

$ update_larsoft_version.sh -q <qualifiers> <larsoft-version>

Here the qualifier option should include base qualifer (e.g. “e26”), built type (debug or prof), and the s-qualifier (e.g. “s131”).

Update_larsoft_version.sh should work with any set of checked out packages, not just sbncode.

update_sbncode_version.sh

Script update_sbncode_version.sh updates the version of an sbncode package only if necessary. That is, it updates the version of an sbncode package only if the checked out working version of a package differs from the matching tag. This script should e invoked after a successful initial test build.
One invocation of this script updates all sbncode package versions that require updates. Script update_sbncode_version.sh is invoked with a single argument, which is the new version tag.

$ update_sbncode_version.sh <new version>

makeSBNRelNotes

Script makeSBNRelNotes makes a skeleton version of sbncode release notes. It is adapted from the standard larsoft release notes script makePatchRelNotes from package larutils. Script makeSBNRelNotes is invoked with three arguments, as follows.

$ makeSBNRelNotes <working-dir> <new-tag> <old-tag>

Here the first argument <working-dir> is the name of a directoty that doesn’t yet exist (the script will make it). The second argument <new-tag> is the name of the tag for which release notes should be generated. The third argument <old-tag> is the name of the previous tag, which is used for a comparison base line.