by Pietro Vischia

So, here I am: it is Friday night, technically already Saturday, since time is 00:42 in Geneva (CERN) timezone, and I am awake.

Pretty normal, for a Friday night, uh?

Thing is, I am not partying, nor am I relaxing at home: I am in the control room of the CMS detector, to monitor that everything is OK, by looking every few seconds at 8 27″ computer screens. I can do some very light work, but nothing that might prevent me from looking at monitor screens every few seconds.

The LHC collider accelerates proton beams in two opposite directions across a circle, and makes them cross in a few points. The conventional name of those points is simply P(number): since in those points the beams collide, the detectors that enable us to detect the products of those collisions are situated precisely in those points. CMS, the experiment I work in, has built the detector in P5.

The photo shows the physical location of CMS (the one circled in black in the top right corner) and of the main CERN complewherep5isx (the one you usually mean when you say “I visited CERN”, which also includes the hostel for visitors – the one circled in black in the lower left corner). Near to this last one, you might notice the Geneva International Airport, and the lake. The circles have labels in Italian, because that is the screenshot I sent to my parents to explain them  the location of the detector compared to the location of the CERN hostel.

Having a large particle physics detector running is not an elementary task: the detector is constituted by hundreds of subsystems, and in order to have all those subsystems to work well together there is a huge shitload of monitoring and tuning to do.

For a system of this complexity, the tuning is not something you do once and for all, but it is more of a continuous task, made of tests and slight adjustments. Consequently, to monitor the outcome of all the tests and to ensure everything is working more or less smoothly (and to eventually fix what is not working properly), we need to have 24/7 some people watching at monitoring screens and performing various actions: these people are called shifters.

The funny thing is that you can stalk watch us all the time via the webcams that are installed and always active.

The photo is a screenshot I took a few minutes ago to freeze the action. Starting from the guy to the extreme right, the DAQ (Data AcQuisition) shifter, whose job is to start and stop data taking (the run) and activate/deactivate different parts of the detector, depending on the requests by the responsibles of the subsystems.shifters

In the center, in a very very relaxed posture and circled in red, you can see yours truly, on Trigger shift. We have an enormous amount of collisions per second, but most of them are only partial collisions that do not produce the kind of processes we are interested in: we are interested in very hard collisions.

Imagine two trucks passing very close to each other, such that their lateral mirrors bump and crash into each other, leaving a few broken glass and plastic pieces. Then imagine two trucks bumping head on into each other, leaving the full crashed wreck on the road. We are interested in the full impact scenario: we need a way of deciding if we are interested in any given event, and we need to decide it fast.

Furthermore, each collision generates an event in our detector: the event is made of all the activation signals of the various subsystems of the detector, so it is something that needs quite some space to be stored (10MB if I remember correctly – it’s now 1:14 AM and the systems are all green). You know when you buy a hard disk looking at the speed of the disk (if you are a gamer, you pay a lot of attention to that)? It’s the same: the rate (i.e. the number per second) of events that we can store is limited by the speed of our processing units and storage devices.

To make the story short, we can store something like 1000 events per second, and for each event we check if there are specific objects (muons, electrons, jets, etc.) to decide if the event is worth keeping: each of the systems checking a single object is called a trigger, and the list of all triggers at any given time (around 200-300) is called trigger menu.

The trigger shifter mainly looks at the list of detector components that are activated, compares it with the list of triggers that are firing (i.e. deciding to keep a particular event), and checks that the rates (the number of events per second that fire each trigger) are consistent with the status of the detector.

For example, now there are a proton beam and a lead beam in the accelerator, but they are not made to collide at P5, this night: since there are no collisions, but the beams are passing inside the detector, I expect to have some noise in a few triggers, to have ~600 random fires per second, and to have exactly zero rate in all triggers that search for very energetic objects (that would leave strong signals in the detector).

If now I saw that suddenly a jet trigger starts firing as if we had energetic jets of particles coming from hard collisions, I would immediately notify the guy to my left with the white sweater (Sasha Nikitenko, an awesome physicist and a very good colleague), which is the shift leader, i.e. the coordinator of the whole shift crew, and then most likely call the HCAL DOC (Hadronic CALorimeter Detector On-Call): DOCs are experts specialized in single subsystems (in this case, the hadronic calorimeter, that is intimately connected with hadronic jets activity). They are not usually all the time at P5, but they must be reachable by phone at any given time, and be able to arrive at P5 in maximum 20 minutes, should the need arise.

Fortunately for them, usually issues can be solved with a phone call plus some online work. During periods of heavy detector testing activity, or of physics collisions, many DOCs are very often at P5 in the room with us, to follow things closer.

Between me and Sasha, there is another guy that in the photo is partially…

01:31 – TWINMUX yellow for a couple seconds, back to green again with no action needed

Sorry, I had to put a note in the electronic log that summarizes my shift.

…so, partially covered by Sasha there is the DQM (Data Quality Monitoring) shifter: he looks mainly at plots that show which part of each subsystem is giving a signal, and to summary histograms of various quantities. He must spot inconsistencies, and report them to make sure any issue is then corrected.

Last but not least, at the far left there is the DCS (Detector Control System) shifter, also known as the Lonely Guy, because he sits at the opposite side of the room, rarely interacts with the Shift Leader, and practically never interacts with the other shifters.

DAQ and Trigger shifters interact quite often, because they need to signal to each other the start and end of each run, and sometimes the trigger shifter spots issues that must be solved by the DAQ shifter by resetting some subsystem. The shift leader interacts with everyone, because it is his job 😀

Why did I choose to be the trigger shifter, when a couple of years ago I started taking detector shifts (which is mandatory in the CMS collaboration)? Half of the reason was that, at the time, I did not know anything about triggers apart from basic things, so I wanted to gain knowledge in that.

The other half of the reason is…. ehem… honestly… that the trigger shifter is the one that has under his direct control the largest number of computer screens (eight of them), and I am kind of a freak of large computer screens 😀

The last very very important element is… the coffee machine room, where you can purchase some comfort food and some drinks, especially coffee (the fuel of high energy physics…): why is this needed? Well, you must know how long will I stay here, this night.

owllogShifts are organized in three tranches. 7AM to 3PM, 3PM to 11PM, and finally the one I am taking these weekends: the 11PM-7AM one, which is called (officially, as you can see in the screenshot)…


Photo: Mark Broadhurst, released under CC0 license