SBN DAQ Software Setup Instructions

These instructions should be largely the same for the different locations in which we typically run/install software: the DAB teststand at Fermilab, and the SBN-FD and SBN-ND DAQ clusters. Please be sure to substitute for the environment where appropriate.

Prerequisites
Logging in and setting up
Build/install local build
- How to update your development area when a new release is available
- Suggested summary install script
Installation on a remote server

Prerequisites

You need an account on the appropriate cluster. Fill out a Test Stand Service Request Form on Fermilab ServiceNow and ask that you be added to the SBN DAQ clusters at DAB and FD/ND as needed.
You need to have a github account and be added to the SBN Software Github Organization. Contact Wes (wketchum AT fnal DOT gov) or fill out this form.

Logging in and setting up

ssh username@sbnd-daq00.fnal.gov

ssh username@icarus-gateway01.fnal.gov

sbnd-gateway01.fnal.gov

Setup the common artdaq products area, and the dev area if appropriate (you need it):

source /daq/software/products/setup   #common area
source /daq/software/products_dev/setup   #dev area

Setup the version of sbndaq we are based. To see a list of the different available versions, use
```
ups list -aK+ sbndaq -qe26:s131:prof
```
It’s recommended you use the latest available version (this is currently v1_10_03 as of 11/2024) with e26:prof:s131 as the option. Then, to set up, use:
```
setup sbndaq v#_##_## -qe26:prof:s131
```
Quick detour:
- e qualifiers specify the gcc compiler version
- s qualifiers specify the art suite version
- prof specifies optimization flags in the compiler, while debug specifies debug flags.
If you want to just run from there, that’s all you need to do!

Build/install local build

If you’re doing development work, then you’ll want to be able to create a development area and download source code, modify it, then build and install your local changes. We use mrb for that.

Setup the mrb environment:

setup mrb v6_08_01
export MRB_PROJECT=sbndaq
export my_sbndaq_version=$(echo  $SETUP_SBNDAQ |cut -d " " -f 2)
export my_sbndaq_quals=$(echo  $SETUP_SBNDAQ |cut -d " " -f 8)
echo sbndaq version: $my_sbndaq_version
echo sbndaq qualifiers: $my_sbndaq_quals

Create and setup a new mrb development directory:

mkdir <your_working_area>
cd <your_working_area>
mrb newDev  -q $my_sbndaq_quals -v $my_sbndaq_version
source localProducts_*/setup

Important: If using a shared user account, create a working area in the ${HOME}/DAQ_DevAreas directory. When naming the area, please use the pattern $(date +"DAQ_%Y-%m-%d")_${your_initials}_${purpose}, e.g. DAQ_2023-08-29_AA_testing_CRT.

export your_initials=FNL
purpose=testing_name
export your_working_area=${HOME}/DAQ_DevAreas/$(date +"DAQ_%Y-%m-%d")_${your_initials}_${purpose}
mkdir -p ${your_working_area}
cd ${your_working_area}
mrb newDev  -q $my_sbndaq_quals -v $my_sbndaq_version
source localProducts_*/setup

Pull down the necessary repositories using mrb. Note, the dashes in the repository name need to be replaced with underscores, e.g. sbndaq-artdaq-core should be typed as sbndaq_artdaq_core:
```
cd $MRB_SOURCE
mrb g <repo_name_with_underscore>
```
Note: you may need to do some setup of your github account with proper SSH keys. See here.) Note: This will pull from the HEAD of the develop branch and include any changes that may have been pushed since the last tag (v#_##_##) was created. Alternatively, if you want to build the tagged, working version that was set up in the previous steps, directly:
```
mrb g <repo_name_with_underscore>@<desired tag version *e.g.* v1_10_03, or branch *e.g.$ feature/me_myAwesomeFeature>
```
If you’re not sure what tags are needed for the various repos you can do a quick check:
```
ups active | grep <repo_name_with_underscore>
```
Repository names are likely one of (or multiples of) these:
```
wibtools
sbndaq-artdaq-core
sbndaq-artdaq
sbndaq
```
Build by doing:

Important: If prompted to run the mrb uc command (see the output example below) then run it followed by mrbsetenv and rerun the build mrb i -j8.

------------------------------------
INFO: stage cmake for MRB project sbndaq v1_00_00
------------------------------------
CMake Error at /daq/software/products/mrb/v5_18_01/Modules/Mrb.cmake:37 (message):
		Current CMake subdirectory inclusion order is not consistent with current
		packages and their interdependencies.

		Please run "mrb uc" to regenerate ${MRB_SOURCE}/CMakeLists.txt with
		subdirectories listed for inclusion in the correct order.
Call Stack (most recent call first):
		/daq/software/products/mrb/v5_18_01/Modules/Mrb.cmake:79 (mrb_check_subdir_order)
		CMakeLists.txt:5 (include)
-- Configuring incomplete, errors occurred!
FATAL ERROR: stage cmake FAILED for MRB project sbndaq v1_00_00 with code 1

cd $MRB_BUILDDIR
mrbsetenv
mrb i -j8
#run only if prompted by mrb
#mrb uc && mrbsetenv && mrb i -j8
mrbslp

The -j8 option tells it to parallelize the build on 8 threads. You may use more or less depending on the number of cores available on the machine you are building on.

If prompted to run the mrb uc command then run it followed by mrbsetenv and rerun the build mrb i -j8.

If you need to rebuild repositories you’ve already pulled down (like, after changing a few lines of code), you will typically only need to do:
```
cd $MRB_BUILDDIR
make i -j8
```
However, if you pull down new repositories or make changes to product dependencies, it’s often true you may need to do a cleanup (“zap”) and complete rebuild:
```
cd $MRB_BUILDDIR
mrb z
mrb zi
mrbsetenv
mrb i -j8
#run only if prompted by mrb
#mrb uc && mrbsetenv && mrb i -j8
mrbslp
```
Note: If using a shared user account, create a symbolic link to ${MRB_TOP}/srcs/sbndaq/<cluster_name>/DAQInterface in the ${MRB_TOP} directory.
```
export cluster_name=sbn-fd #use sbn-fd for Icarus, sbn-nd for SBND, or dab for DO/DAB
ln -s ${MRB_TOP}/srcs/sbndaq/${cluster_name}/DAQInterface ${MRB_TOP}/
```

How to update your development area when a new release is available

Sometimes, you will be working on a new feature, and need to update your code to incorporate a new release of the sbndaq software suite to keep your feature up-to-date. Do the following from a fresh login.

Set up the common products:

source /daq/software/products/setup
source /daq/software/products_dev/setup

Set up the sbndaq version corresponding to the new release:
```
setup sbndaq {version} -q {qualifiers}
```
Create a new localProducts directory, while keeping the same srcs directory:
```
setup mrb v6_08_01
export MRB_PROJECT=sbndaq
cd <your_existing_working_area>
mrb newDev -p
source localProducts_<new one>/setup  #the new localProducts dir!
```
Make sure you source the new localProducts directory! (It is usually good to delete the outdated localProducts directory once you have things working.)
Update your git repositories with the latest release(s).

Then, make a clean build:

cd $MRB_BUILDDIR
mrb z
mrbsetenv
mrb i -j8
#run only if prompted by mrb
#mrb uc && mrbsetenv && mrb i -j8

Suggested summary install script

Put the commands that never change into your bash login script

source /daq/software/products/setup
source /daq/software/products_dev/setup
setup mrb v6_08_01
export MRB_PROJECT=sbndaq

And then to checkout and build sbndaq and sbndaq-artdaq (most common situation) source the script below from your work directory:

setup sbndaq v1_10_03 -q e26:prof:s131
export my_sbndaq_version=$(echo  $SETUP_SBNDAQ |cut -d " " -f 2)
export my_sbndaq_quals=$(echo  $SETUP_SBNDAQ |cut -d " " -f 8)
echo sbndaq version: $my_sbndaq_version
echo sbndaq qualifiers: $my_sbndaq_quals
mrb newDev  -q $my_sbndaq_quals -v $my_sbndaq_version
source localProducts_*/setup

mrb g sbndaq
mrb g sbndaq_artdaq

cd build*
mrbsetenv
mrb i -j8 |& tee make.log
#run only if prompted by mrb
#mrb uc && mrbsetenv && mrb i -j8 |& tee make.log

mrbslp

Installation on a remote server

First set up local directories that you will need for building and running. We suggest you put the /daq mount point on one physical disk or disk array, with each directory below a separate partition. This is not stricly needed, but will keep your local areas identical to the ones at Fermilab, but modify as local conditions necessitate.

Directory/Partition	Suggested Size	Purpose
/daq/software	2 TB	Prepackaged software copied from FNAL
/daq/log	3 TB	Locally generated log files
/daq/scratch	3 TB	Temporary scratch data files
/daq/run_records	1 TB	Record of locally produced runs and their configurations

In order to install sbndaq on a server which doesn’t have software directory (e.g. outside FNAL), one first needs to copy the software with following commands:

rsync -arvtzl --del -e 'ssh -o "ProxyCommand ssh -A USERNAME@icarusgpvm01.fnal.gov -W %h:%p"' USERNAME@sbnd-daq32.fnal.gov:/software/products/ /software/products/.  --delete-excluded --exclude 'Xilinx*' --exclude 'Quartus'  --exclude="*.tar.bz2" --exclude "products/" 
rsync -arvtzl --del -e 'ssh -o "ProxyCommand ssh -A USERNAME@icarusgpvm01.fnal.gov -W %h:%p"' USERNAME@sbnd-daq32.fnal.gov:/software/products_dev/ /software/products_dev/.

Replace USERNAME with your login. SBND users may need to replace icarusgpvm01 with another VM. Replace sbnd-daq32.fnal.gov with the location you are installing.

As of Jul 2020 /software occupies 202 GB. The –exclude options help to omit tens of GB of data which doesn’t need to be synchronised (in most cases).