ROOT and OpenMp

The parallel analysis of data in the ROOT environment can be accomplished using OpenMP technologies. In particular, the Minuit2 library was written using openMP. When working with this library, you should set the USE_PARALLEL_MINUIT2 and USE_OPENMP environment variables.

In addition, the OpenMP technology is applied in the TMVA package, i.e. a comprehensive toolkit for multidimensional data analysis in ROOT. Large loops are parallelized in classes that implement the genetic algorithm, such as GeneticAlgorithm, GeneticPopulation, using OpenMP. The FitUtil class of the mathematical library implements the FitUtilParallel () method.

Finally, ROOT ensures the execution of user macros written using OpenMP library procedures, but currently only in a batch mode.

Let us look at an example of how to start filling histograms using OpenMP.

The first step is to edit the .bashrc file. In this file, you need to add the address where ROOT is located on your machine. For example, for the HybriLIT cluster:

# .bashrc
....................
....................
....................
# User specific aliases and functions
export ROOTSYS=/cvmfs/hybrilit.jinr.ru/sw/root
export PATH=$PATH:$ROOTSYS/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ROOTSYS/lib

Next, in order to access OpenMP in the user macro, you need to add

#include &ltomp.h&gt

Below is the macro code in which a random number is drawn, calculations with this number and filling of histograms.

#include "TROOT.h"
#include "TH1F.h"
#include "TRandom.h"
#include "TCanvas.h"
#include "TFrame.h"
#include "TDatime.h"
#include "TF1.h"
#include "TFile.h"
#include "TStopwatch.h"
#include &ltiostream&gt
#include &ltomp.h&gt

using namespace std;

int main(){
gROOT->Time();
TStopwatch timer;
Int_t nHist=48;
Long64_t n=10000000;
TH1F *fH1F[48];
Int_t i,j;
Double_t y;
TRandom *fRandom=new TRandom();
TFile *f=new TFile("ompYes.root","recreate");
cout&lt&lt"max threads="&lt&ltomp_get_max_threads()&lt&ltendl;
timer.Start();
for(i=0;i&ltnHist;i++){
fH1F[i] = new TH1F(Form("hpx%d",i),"The px distribution",200,0,200);
}
#pragma omp parallel for shared(fH1F) private(j,i)
for(i=0;i&ltnHist;i++){
for(j=0;j&ltn;j++){
Double_t x=fRandom->Gaus(0.,i+1.);
y=sqrt(x*x*x*x+x*x+1)-cos(x+5)*sin(x-5);
fH1F[i]->Fill(y);
}
}
f->Write();
timer.Stop();
cout&lt&lt"time: ";
timer.Print("m");
f->Close();
return 0;
}

To compile the macro, you should run the command

g++ -fopenmp NameMacros.C 'root-config --cflags --libs'

An executable binary file is generated upon successful compilation. By default, the name of the binary file is a.out. Details on how to run OpenMP applications on the HybriLIT cluster can be viewed here. Here we give the recommended type of a script file, for example, with 12 streams:

#!/bin/sh
#SBATCH -p cpu
#SBATCH -c 12
#SBATCH t 60
export OMP_NUM_THREADS=12
export OMP_PLACES=cores
./a.out

To start the application, use the following command:

$ sbatch omp_script

It should be noted that using OpenMP you can easily parallelize arithmetic operations occurring in macros. Specific methods of the ROOT package often do not allow parallelization using this technology. For example, the classes TTree and TFile are not thread-safe because they manipulate global data and not all of these data are fully protected to ensure thread safety. Therefore, the objects of these classes should not be shared (without blocking) between threads. However, you can create multiple TFile objects (and, therefore, TTree objects) by reading the same physical file.

Keep in mind the fact that threading also takes time. It may happen that the time gain from running the program on several threads will be less than the time spent on threading. The use of OpenMP is justified only if a significant number of arithmetic operations are performed in each thread. Otherwise, it is preferable to use PROOF for program parallelization. Here you can find an example (download a file) that compares the efficiency of program parallelization using OpenMP and PROOF.