How to Build a Spack Environment for Distribution in CVMFS (Manual Method)
Table of Contents
Overview
This page contains instructions for building a binary spack environment for distribution in CVMFS. This article assumes that all of the relevant packages have already been tagged. This article covers the manual method, meaning that the build does not use Jenkins.
Step-by-step Instructions
Step by step instructions follow.
Create a Spack Instance
In your own account on the sbn gpvm’s, create a spack instance that is a subspack of a suitable parent spack instance. Use padding.
$ source /cvmfs/sbn.opensciencegrid.org/spack-sbn-v1.1.1/setup-env.sh $ spack subspack --with-padding /path/to/spack/instance $ source /path/to/spack/instance/setup-env.sh
If your experiment already has a spack instance in cvmfs, use that as the parent. If you are creating a new spack instance, use the next upstream spack instance as the parent.
Check file $SPACK_ROOT/etc/spack/upstreams.yaml to make sure that upstream instances are in the correct order (downstream to upstream). Usually, the larsoft spack instance should be listed last.
$ cat $SPACK_ROOT/etc/spack/upstreams.yaml
upstreams:
spack_1791905173.805394:
install_tree: /cvmfs/sbn.opensciencegrid.org/spack-sbn-v1.1.1/opt/spack
modules:
tcl: $spack/share/spack/modules
spack_1790350987.766301:
install_tree: /cvmfs/larsoft.opensciencegrid.org/spack-fnal-v1.1.1/opt/spack
modules:
tcl: $spack/share/spack/modules
If you are reusing an existing local spack instance, update the spack recipes.
$ spack repo update
Update Environment Configuration
Check out sbn-release-configs from github.
$ git clone git@github.com:SBNSoftware/sbn-release-configs $ cd sbn-release-configs
This repository contains release configurations for sbn, sbnd, and icarus. You should be on branch main. Make any required edits to the top level release config (sbn-release.yaml, sbnd-release-yaml, or icarus-release.yaml), as well as files in subdirectory included_yaml. Make sure that the top level release config includes a mirrors: section pointing to any experiment-specific binary caches.
mirrors:
uboone: https://spack-cache-1.fnal.gov/binaries/sbn
Commit updates and tag the newly created head version. The tag should match the tag mentioned in the release config.
Create a new environment
Use the newly updated release config to make a new environment.
$ spack env create <environment> /path/to/release/config $ spack env activate <environment>
If you need to edit the environment configuration during the build process, the top level configuration file can be edited directly at the following location.
$SPACK_ROOT/var/spack/environments/<environment>/spack.yaml
If you need to edit a configuration file in the included_yaml subdirectory, those files need to be commited back to github and the tag moved. If you do edit the environment configuration directly in the spack instance, remember to copy and commit it back to github later.
Update Recipes
In most cases, you will want to visit various recipe repositories in order to check out the latest recipes. Some collections of recipes for which you may want to do this are as follows.
fnal_art larsoft-spack-recipes sbn-spack
For each collection, do the following.
$ cd ${SPACK_ROOT}/var/spack/repos/
$ git fetch origin
$ git branch --show-current # Note which branch is checked out
$ git branch -u origin/ # Reset upstream branch
$ git pull
</pre>
### Concretize the new environment
Concretize the new environment.
$ spack concretize
Verify the correctness and reasonableness of the concretized environment using the following command.
$ spack find -lc
### Build and install the new environment
Update package recipes with any new checksums. This step can be interleaved with `spack install` to identify packages requiring updated checksums.
$ spack checksum <package> <version>
Install packages.
$ spack install -j<Njobs>
Verify that all packages have been installed using `spack find -lc`.
When you are finished, push updated recipes to github. If necessary, update the git configuration to use a pushable url.
$ git remote set-url origin git@github.com:SBNSoftware/sbn-spack.git
### Install a Siging Key
Make sure that your working spack instance has a signing key.
$ spack gpg trust <public-key-file>
$ spack gpg trust <secret-key-file>
$ spack gpg list
For instructions on how to generate key files, refer to [Fermilab instructions](https://fnalssi.github.io/spack-at-fnal/pages/build_manager_process.html).
### Create Local Binary Cache
Make sure that this spack instance has a local binary build cache, and populate it with newly installed packages.
$ spack cd -e
$ spack localbuildcache --key <key> --local --not-bc --with-build-dependencies
The key name `` is the name assigned when the key was created. The name of an installed key can be discovered using command `spack gpg list`.
### Upload Local Binary Cache to Cache Server
Upload local binary build cache content to the cache server.
$ scp -r bc/* products@fifeutilgpvm03.fnal.gov:/spack_cache/sbn
$ ssh products@fifeutilgpvm03.fnal.gov
$ source /cvmfs/larsoft.opensciencegrid.org/spack-fnal-v1.1.1/setup-env.sh
$ spack buildcache update-index -k /spack_cache/sbn
$ logout
### Install the Environment in CVMFS
Copy the concretized environment configuration `spack.lock` from the newly built environment to the cvmfs server.
$ spack cd -e
$ scp spack.lock cvmfssbn@oasiscfs.fnal.gov:
Log in to the cvmfs server and open a transaction.
$ ssh cvmfssbn@oasiscfs.fnal.gov
$ cvmfs_server transaction sbn.opensciencegrid.org
If necessary create a new stand alone or subspack instance in cvmfs. Do this once per instance. Do not use padding.
$ source /cvmfs/larsoft.opensciencegrid.org/spack-fnal-v1.1.1/setup-env.sh
$ spack subspack /cvmfs/sbn.opensciencegrid.org/spack-sbn-v1.1.1
Create a new environment using the uploaded `spack.lock`. Use the same name as your working spack instance.
$ source /cvmfs/sbn.opensciencegrid.org/spack-sbn-v1.1.1/setup-env.sh
$ spack env create <environment> spack.lock
$ spack env activate <environment>
The environment should already be concretized. Verify that the concretized environment is as expected.
$ spack find -lc
Update package recipes.
$ spack repo update
Install packages from cache.
$ spack install --include-build-deps --no-add --only-concrete --cache-only [--no-check-signature]
This step should be able to install all packages from cache (option `--cache-only`).
Verify that all packages have been installed.
$ spack find -lc
Publish cvmfs transaction
$ cvmfs_server publish sbn.opensciencegrid.org