What to do on your (remote) shift - WORK IN PROGRESS

What to do on your (remote) shift - WORK IN PROGRESS

  • Table of contents
  • [What to do on your (remote) shift - WORK IN PROGRESS]
    • [General requirements for a remote shift]
      • [How do I know who is on shift?]
      • [If you're shifting remotely from your personal laptop (Standard conditions since March 2020)]
      • [If you're shifting from a remote control room]
    • [More than 6 weeks before your shift: Request accounts and passwords!]
      • [Fermilab accounts]
      • [Fermilab VPNs]
      • [iFix account and remote desktop apps]
      • [Slack Account]
    • [More than 1 week before your shift: Shadow shift and test connections]
      • [Shadow shift]
      • [Test connections]
    • [AT THE BEGINNING OF YOUR SHIFT]
      • [Shift Check-in]
        • [Bring up the remote connections]
        • [Setup GoogleVoice]
      • [Start Shift]
    • [DURING YOUR SHIFT]
      • [ECL forms to be filled out during your shift]
      • [Start and Stop a run]
      • [Request help]
    • [AT THE END OF YOUR SHIFT]

::: Edit this section{.icon-only .icon-edit} :::

General requirements for a remote shift

This page is your main guide through your remote shift. It is definitely a nice bookmark to this page during your time in ICARUS. The page is organized in chronological steps to better serve the needs and memory of the shifter in a quick and simple way. It won't be exhaustive of all the potential issues a remote shifter may encounter. Please refer to the provided links for additional resources or get in touch with a @icarus_shift_help on Slack or the Commissioner/Run coordinator.

Remember that during a shift reaction time is precious! Take some time in advance before your shift block to revise this material and don't forget to skim through this page at any time you start a new shift and verify if something new has been added!

::: Edit this section{.icon-only .icon-edit} :::

How do I know who is on shift?

  • Who is currently on shift? Check the ECL Who is on shift now section . You can also visit the #icarus-shift-operations channel in the SBN Slack to find out who the current shifter is by scrolling through the messages.
  • Who will or has been on shift? Check the ECL Shifts Calendar .

::: Edit this section{.icon-only .icon-edit} :::

If you're shifting remotely from your personal laptop (Standard conditions since March 2020)

Carry out a remote shift is not a simple task and you're required to be flexible and adapt your workflow and local environment to the requirements of the shift. Here below there are some suggestions and resources to make your shift experience smooth and nice:

  1. Fermilab Windows account (mandatory)
  2. Fermilab service account (mandatory)
  3. Fermilab Unix account (mandatory)
  4. Fermilab VPN (mandatory)
  5. Browser Firefox or Chrome (recommended)
  6. Extra screen (recommended)
  7. OS: Linux or MacOSX (recommended)
  8. Slack account (mandatory) and desktop app (recommended)
  9. Remote desktop app (mandatory), see here some suggested choices

Other shift resources:

  • ROC WEST ZOOM: Can be used for useful verbal communication bethween shifters and experts. The connection details are at the beginning of DocDB 15944 (or in this alternate link)

::: Edit this section{.icon-only .icon-edit} :::

If you're shifting from a remote control room

At this time the ICARUS experiment is not supporting remote control rooms. Please make sure to have the contact of the person who setup the control room in case of need or refer to the previous paragraph.

::: Edit this section{.icon-only .icon-edit} :::

More than [6 weeks]{.ul} before your shift: [Request accounts and passwords!]{.ul}

If you're not a Fermilab newbie and you took a shift within ICARUS before, it is very likely you don't need this section. Be sure however that none of your accounts are about to expire in proximity with your shift time and take the necessary steps to renew them with sufficient advance.

::: Edit this section{.icon-only .icon-edit} :::

Fermilab accounts

::: Edit this section{.icon-only .icon-edit} :::

Fermilab VPNs

::: Edit this section{.icon-only .icon-edit} :::

iFix account and remote desktop apps

::: Edit this section{.icon-only .icon-edit} :::

Slack Account

::: Edit this section{.icon-only .icon-edit} :::

More than [1 week]{.ul} before your shift: [Shadow shift and test connections]{.ul}

If you've not taken any shift in the past 4 months you're then required to run a shadow shift. Instead, if you've been on a shift in the past 4 months, you are required to test your remote connections, to make sure your accounts are properly set and take the necessary steps to renew them or find a replacement shifter in case you're not able to serve your shift.

::: Edit this section{.icon-only .icon-edit} :::

Shadow shift

ICARUS DOES NOT YET REQUIRE TO TAKE A SHADOW SHIFT

::: Edit this section{.icon-only .icon-edit} :::

Test connections

In order to test the connections, please try to complete the steps of paragraphs Bring up the remote connection and Setup GoogleVoice. Inform the sitting shifter on the #icarus-shift-operation, before you start testing your connections and at the end.

Take action as soon as possible if you find connections that are not working. @icarus_shift_help may provide support. In case you need to swap your shift: please inform the Commissioning/Run Coordinator and follow this information

::: Edit this section{.icon-only .icon-edit} :::

AT THE BEGINNING OF YOUR SHIFT

::: Edit this section{.icon-only .icon-edit} :::

Shift Check-in

Please show up for your shift 15 minutes in advance. The very first step you have to do are:

  1. Communicate with the previous shifter on the Slack channel #icarus-shift-operation and inform him of your arrival. Ask him to brief you about the detector and run condition and if issues have been encountered during the shift or experts have been called. If you have the chance to speak with the shifter in person or over the phone, please post a message on Slack #icarus-shift-operation channel, that you're ready to take over the shift.
  2. Read the Bulletin board. Along with the shift communication, make sure also to read the bulletin board and refresh your memory about the shift procedure.
  3. Communicate with your shadow shifter If you have a shadow shifter, make sure he's ready to shadow you.
  4. Fill the Shift Check-in checklist

::: Edit this section{.icon-only .icon-edit} :::

Bring up the remote connections

::: Edit this section{.icon-only .icon-edit} :::

Setup GoogleVoice

::: Edit this section{.icon-only .icon-edit} :::

Start Shift

Now that you're well set for your shift you can

  1. Call the Main Control Room (if required) at +1 630 840 3721
  2. Make sure your shadow shift is well set as well
  3. Fill the Start shift form

Congratulations!! You're now well set to start your shift. You're the pilot of ICARUS at this point!

::: Edit this section{.icon-only .icon-edit} :::

DURING YOUR SHIFT

The duty of the shifter is to routinely check the status of the detector and fill periodic checklists. Below is a list of the ECL forms that you should fill out throughout your shift. It is your responsibility to fill out all these forms during every shift. There are also other forms that you should fill out in certain situations and others that will be filled by experts. To fill up a form:

  1. Log into the ICARUS ECL.
  2. Click on New entry, in the left vertical menu.
  3. A new window will pop up.
  4. Select a form from the Form menu.
  5. Fill up the fields. Some comments to help you complete the form are shown on the right of each field. You may need to make the form pop-up window larger to see these instructions/comments.

In addition, please make sure to make an ECL post regarding any activity, anomaly or issue you encounter while on shift.

::: Edit this section{.icon-only .icon-edit} :::

ECL forms to be filled out during your shift

When What Form Where to find more info Whom to contact? Slack tag Before the previous shifter leaves Shift check-in On the checklist Expert shift @icarus_shift-help Immediately after shift check-in Start of shift On the checklist Expert shift @icarus_shift-help Once during your shift EventBuilder PDU check See paragraph below: "Run EventBuilder PDU check script from the terminal in Slow Controls machine" Expert shift @icarus_shift_help Right after the start of day shift Daily ICARUS Access On the checklist Expert shift @icarus_shift_help Once every hour PMT HV Stability PMT HV Monitoring Wiki PMT Expert @icarus_pmt_experts Once every hour Cryogenic Stability Checklist iFix tutorial for shifters Cryogenic experts n/a Once every hour Cathode HV Checklist Cathode HV Wiki Drift HV expert n/a Every other hour (with a run ongoing) DAQ Commissioning Checklist On the checklist and/or shifter bulletin for run condition DAQ Expert @sbn_daq_expert Every other hour TPC Commissioning monitoring checklist On the checklist TPC PS Expert @icaus-tpcps Every other hour BNB simplified checklist Beam overview Wiki Beam experts @icarus_beam_experts Every four hours Impedance monitor form Impedance monitor wiki Linda Bagby n/a After the new shifter has arrived and you have handed things over Shift check-out On the checklist Expert shift @icarus_shift-help ——————————————————————- —————————————————————————————————————————————————– ————————————————————————————————————– ———————- ————————

Run EventBuilder PDU check script from the terminal in Slow Controls machine

Instructions
From a terminal in the Slow Controls VNC, run the command

/home/nfs/howard/pdu_check/runCheckEvbPdu.sh

and report its result in the ECL. A suggested title would be "Event Builder PDU Check"

Specifically, you want to check the results of the lines EVB rack, PDU Bank 1: and EVB rack, PDU Bank 2:

If the value reported in the line for Bank 1 is > 20 A or the value reported in the line for Bank 2 > 2 A, then call and alert the Commissioning Coordinator (Angela Fava) or Deputy (Andrea Scarpelli), whose numbers can be found on the Expert Call List.

::: Edit this section{.icon-only .icon-edit} :::

Start and Stop a run

::: Edit this section{.icon-only .icon-edit} :::

Request help

::: Edit this section{.icon-only .icon-edit} :::

AT THE END OF YOUR SHIFT

It is your duty to remain on shift until the new shifter is well set. Inform the commissoning/run coordinators if shifter doesn't show up after 15 minutes at the end of your shift to coordinate the transition and make sure the shift after yours is well covered

  • When the new shift arrives you should interact with him/her and communicate anything relevant that happened during your shift. At this point the incoming shifter will prepare the remote connections and log-in to the Google Voice account.
<!-- -->
  • Pass down Google Voice code: the incoming shifter requests the access code to Google Voice. This will happen at any new login since a two-factor authentication is in place. As the shifter before you did for you, you should send the code to the incoming shifter. The code can be found in the Google Voice main page under the received messages ( Click the message icon from the menu bar on the left side of the screen ). The code has the format G-****. Remember to log-out from the ICARUS Google account once the shifter confirms that he/she is well set!

!(/redmine/attachments/download/63879/gvoice_messages.png)

  • Fill the Shift Check-out form once the incoming shifter is ready to take over
<!-- -->
  • Log off from the VNC sessions: * please press the *DISCONNECT button in the top right corner of the VNC sessions before closing the Web browser (e.g. Firefox) tabs. It is in the NoVNC menu on the left side of the screen (it may be just an arrow that you have to click to expand). The disconnect button is on the bottom, see the NoVNC Wiki for slightly more info as needed!

!(/redmine/attachments/download/63878/Disconnect_NewNoVNC.png)

Congratulations!!! You have completed you shift and you can now enjoy the rest of your day ;) :::

Files (2)

::: {style=”display: none;”} ::: {.attachments} ::: {.contextual} Edit attached files{.icon-only .icon-edit} :::

Disconnect_NewNoVNC.png{.icon .icon-attachment} [(51 KB)]{.size} Disconnect_NewNoVNC.png{.icon-only .icon-download} [Andrea Scarpelli, 05/01/2021 09:12 PM]{.author} Delete{.delete .icon-only .icon-del} gvoice_messages.png{.icon .icon-attachment} [(62.9 KB)]{.size} gvoice_messages.png{.icon-only .icon-download} [Andrea Scarpelli, 05/01/2021 09:16 PM]{.author} Delete{.delete .icon-only .icon-del} —————————————————————————————————————————————————————————————————————————————————– – ————————————————– —————————————————————————– :::

::: ::: {.box} [ [ ]{.attachments_fields} [ (Maximum size: 400 MB) ]{.add_attachment style=””} ]{.attachments_form} ::: ::: :::

Also available in: PDF{.pdf} HTML{.html} TXT{.txt}

::: {style=”clear:both;”} ::: ::: ::: :::

::: Loading... :::

::: :::

::: ::: {.bgl} ::: {.bgr} Powered by Redmine © 2006-2019 Jean-Philippe Lang ::: ::: ::: ::: :::