Almost two months ago, Tommaso and I designed a challenge about guessing the b-flavour content of jets in simulated QCD processes. The aim of the competition was to predict the fraction of events with 0,1,2,3 and 4 selected b-jets (i.e. jets which contain b-hadrons) after an event selection which resembles the one used for the HH → bbbb analysis we are working on.

To make the game more interesting, we increased the number of variables to predict by dividing the data into four subsets corresponding to four different jet invariant mass ranges: 200-400 GeV, 400-600 GeV, 600-800 Gev and 800-1000 GeV. Therefore, a total of 20 variables were to be guessed, but four of them were not independent because the sum of fractions per bin had to be 1.

Both made an educated guess for these variables and agreed that the loser had to buy a refreshing beverage for the winner. We also invited readers of this blog to join us in the game, but we have not heard of any external participants, which is kind of understandable given the strangeness and  specifics of this bet.

Some time has passed and it is time to see who won. Furthermore, given that we are currently generating QCD processes enriched in b-jets, we are genuinely interested in how common they are in the final state for the inclusive simulated sample.

Simulated datasets contain sets of events corresponding to a physical process. Simulated (aka Monte Carlo) samples can be treated as data, but they also include what we call truth information, which includes type and momenta of all the particles that were generated as products of the simulated collision. For more information on the Monte Carlo simulations have a look at Giles’ last post.

The MC truth can be linked with data-like observables (i.e. reconstructed objects) with matching procedures. There are severals ways of doing this, especially for compound objects as jets, usually leading to one-to-many and many-to-one pairings.

For this problem, I have used what is called the hadron flavour of the jets, which basically tells me if a jet is matched with a generated jet that clustered the products of heavy flavour hadrons (bottom or charm).

Therefore,  to check who was the winner of the bet, I had to count the number of b-flavoured jets selected as a function of the four jet invariant masses for all the events that pass the event selection mentioned in the precursory in simulated QCD samples.

So let’s see graphically how our guesses and results compare:

All figures follow the same format. The fraction of events with 0,1,2,3 and 4 b-jets is stacked for each invariant mass bin using different colours. One the left you see Tommaso’s guess, the one on the right is mine, and the center figure shows the results from the simulated QCD dataset.

Can you guess who won without further reading and forgetting the title of this post? That would depend on the figure of merit agreed upon when we defined the challenge.

In this case, we opted for a sum squared of the deviations of our prediction from the QCD simulation values, what is referred to as χ2  in the previous post. I prefer to use root mean squared deviation (RMSD) instead, which is a monotonic function of χ2  (i.e. same winner), but has the advantage that it is more convenient for interpretation and makes my score closer to my supervisor’s.

In the figure preceding this paragraph, we see that our guesses are clearly different. Tommaso expected that a small but not negligible fraction of events would have 0 and 1 selected b-jets, while approximately the same number of events with 2, 3, and 4 b-flavoured jets would be present. Maybe he can explain his rationale for the invariant mass dependence chosen.

However, I thought that b-tagging criteria would be tight enough to reduce 0 and 1 b-jet contribution under the 5% level, leaving a small 2 b-jet fraction and making the 3 b-jet category dominant.

Looking at the QCD simulation results, we see that neither of us was very close to truth. The largest fraction is constituted by events with 4 b-jets, which neither of us had expected and event with a low number of b-jets have a non-negligible contribution.

This was a difficult challenge indeed, because the variables to be predicted depend on the number and spectrum of jets from different flavours in an event, the efficiency and fake rate of online and offline b-tagging algorithms and how the invariant mass of four bodies depends on the jet variables.

In the next figures I compare predictions and QCD simulation results for each category independently. You can see that Tommaso did much better in the 0,1,2 and 3 b-jet categories, while I almost nailed the 4 b-jet category (loser consolation prize). In the last plot, you can compare our scores. I had a RMSD of 0.15 while Tommaso won the bet with a mean squared error of 0.10!

category_flav_compostion (1)
Comparison of predictions and QCD simulation for each number of b-jet category and final scores of the challenge for both participants.  The error bars in the histograms are due to the limited statistics of the simulated sample. Predictions are in 5% units as agreed.

The take-home message is that we need simulated data for modelling complex phenomena in colliders, it is really hard to predict compound magnitudes without crunching numbers.

How could we test the hypothesis that our educated guesses were better than random number pickings? In other words, has previous acquired knowledge helped us to achieve a higher prediction accuracy? Not sure, I am eager to discuss it in the comments if you please.

The only thing I am certain about is that I will have to buy Tommaso ${another euphemism for beer} as agreed. Keep tuned to this blog for more challenges and games, which we will try to make more welcoming for other takers apart from us.