<chapter name="Parallelism"> 
 
<h2>Parallelism</h2> 
 
A large fraction of studies using <code>Pythia</code> follow the same 
general steps: first the <code>Pythia</code> object(s) and histograms 
are initialized. Then there is a loop where events are generated using 
<code>Pythia::next</code>. These events are subsequently analyzed one 
by one, and the resulting statistics are stored in 
histograms. Finally, the histograms are normalized and the results are 
plotted. 
 
<p/> 
In this procedure, usually the events are independently generated, and in 
principle it should be possible to generate them in parallel. The 
<code>PythiaParallel</code> class provides a simple interface for 
doing this. Objects of this class behave similarly to <code>Pythia</code> 
objects in that they can be configured using <code>readString</code> and 
<code>readFile</code>, and are initialized using 
<code>init</code>. The difference is that instead of having a 
<code>next</code> method that generates a single event, they have the 
method <code>run</code> which generates a number of events in parallel 
and analyzes them using a used-defined callback function.  The program 
flow using the parallelism framework is as follows: 
 
<ol> 
<li> 
In addition to the standard header file, the parallel framework header 
must be included. 
<pre> 
    #include "Pythia8/Pythia.h" 
    #include "Pythia8/PythiaParallel.h" 
</pre> 
</li> 
 
<p/> 
<li> 
The next step is to create, configure, and initialize a parallel 
generator object. This process is identical to that for a standard 
<code>Pythia</code> object. 
<pre> 
    PythiaParallel pythia; 
    pythia.readString("HardQCD:all = on"); 
    pythia.init(); 
</pre> 
</li> 
<p/> 
<li> 
Finally, the <code>PythiaParallel</code> object is run with a user 
function which performs the necessary analysis on the events. In this 
example, an event multiplicity histogram is filled. 
<pre> 
    Hist mult("mult", 100, -0.5, 799.5); 
    pythia.run(10000, [&amp;](Pythia* pythiaPtr) { 
      int nFinal = 0; 
      for (int i = 0; i &lt; pythiaPtr-&gt;event.size(); ++i) 
        if (pythiaPtr-&gt;event[i].isFinal()) ++nFinal; 
      mult.fill( nFinal ); 
    }); 
</pre> 
Here the syntax <code>[&amp;](Pythia* pythiaPtr)</code> may be 
unfamiliar. It defines a <ie>lambda</ie> function that takes a pointer to a 
<code>Pythia</code> object as an argument. The <code>[&amp;]</code> indicates 
that all local variables used by the lambda function (in this case only the 
<code>mult</code> histogram object) are passed by reference. 
</li> 
</ol> 
 
<h3>Code Overview</h3> 
 
The scope of this class is to provide a lightweight way of 
parallelising simple <code>Pythia</code> studies. As such, it may not 
offer the necessary features to support more complicated use cases. 
The specific features are documented below, and examples are given in 
<code>main221</code>, <code>main222</code>, <code>main223</code>, and 
<code>main404</code>. 
 
<method name="vector&lt;long&gt; PythiaParallel::run(long nEvents, 
function&lt;void(Pythia*)&gt; callback)"> 
</method> 
<methodmore name="vector&lt;long&gt; PythiaParallel::run( 
function&lt;void(Pythia*)&gt; callback)"> 
this is the main method of <code>PythiaParallel</code>, analogous to 
<code>Pythia::next</code>. The method generates <code>nEvents</code> 
events in parallel, distributing the tasks automatically to different 
<code>Pythia</code> instances. If <code>nEvents</code> is not 
specified, <code>Main:numberOfEvents</code> is used instead. 
 
The <code>callback</code> is a function that will be called when each 
event is generated. It takes a pointer to the <code>Pythia</code> 
instance that generated the event as an argument, 
and the event can then be accessed 
through <code>Pythia::event</code>. If an event fails to generate 
successfully, it will not be passed to the callback. 
 
By default, callbacks are synchronized. That is, only one callback can 
be active at the same time, which means it is safe to e.g. write to 
histograms from within the callback. Asynchronous processing of 
callbacks can be enabled by setting <code>Parallelism:processAsync = 
on</code> (see below). 
 
Returns a <code>vector&lt;long&gt;</code> containing the number of 
events successfully generated by each thread. If any events fail to 
generate, the entries will sum to a number that is smaller than the 
requested number of events. 
</methodmore> 
 
<method name="bool PythiaParallel::init()"> 
</method> 
<methodmore name="bool PythiaParallel::init( 
function&lt;void(Pythia*)&gt; customInit)"> 
initialize each <code>Pythia</code> instance and returns whether successful. 
<argument name="customInit"> 
If specified, this function will be called for each 
<code>Pythia</code> instance after it has been constructed and its 
settings have been set, but before calling <code>Pythia::init</code>. 
This can be useful for example for setting a <code>UserHooks</code> object 
on each instance. 
</argument> 
</methodmore> 
 
<method 
name="void PythiaParallel::foreach(function&lt;void(Pythia*)&gt; action)"> 
perform the specified action for each <code>Pythia</code> instance. 
This can be useful for doing custom finalization on each instance, e.g. 
combining histograms. 
</method> 
 
<method name="void PythiaParallel::foreachAsync( 
function&lt;void(Pythia*)&gt; action)"> 
as <code>PythiaParallel::foreach</code>, but the actions are performed 
for all <code>Pythia</code> instances in parallel. 
</method> 
 
<method name="double PythiaParallel::weightSum() const"> 
returns the sum of weights from all <code>Pythia</code> instances, as given 
by <code>Info::weightSum()</code>. 
</method> 
 
<method name="double PythiaParallel::sigmaGen() const"> 
returns the weighted average of the generated cross section for each 
<code>Pythia</code> instances, as given by <code>Info::sigmaGen()</code>. 
</method> 
 
<p/> 
The following settings are available for the parallelism framework. 
 
<mode name="Parallelism:numThreads" default="0" min="0"> 
Number of threads to run in parallel. If set to 0, the number of threads will 
be estimated using <code>std::thread::hardware_concurrency</code>; if the 
program is unable to determine the number of threads this way, initialization 
will fail. 
</mode> 
 
<mvec name="Parallelism:seeds" default="{}"> 
The seeds to use for each Pythia object. If empty, <code>Random:seed</code> 
will be used, incrementing the seed by 1 for each object. If non-empty, it 
must have a number of entries equal to <code>Parallelism:numThreads</code>. 
</mvec> 
 
<flag name="Parallelism:processAsync" default="off"> 
By default, <code>PythiaParallel::run</code> will generate events in 
parallel, which are then processed serially. The advantage of this serial 
kind of processing is that it prevents race conditions, which can occur 
for example if two threads are trying to simultaneously write to the 
same histogram. Normally, event generation is far more time-consuming 
than analysis, and the time loss from processing in serial is 
completely negligible. 
 
However, there are situations where the analysis takes a non-negligible 
amount of time compared to event generation. In such scenarios, the user 
may enable parallel processing of the analysis by setting 
<code>Parallelism:processAsync = on</code>. In this case, the user 
is responsible for preventing race conditions, possibly by using 
<code>mutex</code> and <code>lock_guard</code> objects. An example of 
how this can be done is shown in <code>main163.cc</code>. 
</flag> 
 
<flag name="Parallelism:doNext" default="on"> 
By default, <code>PythiaParallel</code> generates events, then passes the 
<code>Pythia</code> object to the callback function. If this flag is turned 
off, the event is not automatically generated before calling the callback, 
so that the user is responsible for calling <code>Pythia::next</code>. 
This can be useful for example if you want to change the energy on an 
event-by-event basis, or otherwise want to do checks at the beginning of 
the event loop before performing the actual generation. 
 
If this flag is off, you must also set 
<code>Parallelism:processAsync = on</code>. If it is off, 
</flag> 
 
<mode name="Parallelism:index" default="-1"> 
When a <code>Pythia</code> instance is passed to the callback function in 
<code>PythiaParallel::run</code>, this setting will contain an index that is 
unique to each instance, starting at 0 for the first instance. This index is 
particularly useful if <code>Parallelism:processAsync = on</code>. For example 
if each instance writes to different histograms, this index can be used to 
specify which histogram to write to. 
 
Note that the user should not write to this setting directly. 
</mode> 
 
<flag name="Parallelism:balanceLoad" default="on"> 
By default, the event generation is divided evenly between the 
<code>Pythia</code> instances. This way, each instance will generate 
the same number of events each run with their respective seeds, so that 
the final statistics will be exactly the same between runs (except events 
might be processed in a different order). 
 
By turning this flag off, instead each <code>Pythia</code> instance will 
generate events until the desired number has been generated. Thus, different 
runs might result in slightly different statistics, even with the exact 
same input settings. The advantage of this is that it can be significantly 
more efficient if the event generation time can vary significantly (e.g. as 
it does in central vs. peripheral heavy ion collisions). 
</flag> 
 
<h3>Parallelism and Plugins</h3> 
 
There are a number of Pythia base classes that can be passed as 
external pointers to a <code>Pythia</code> instance, for example 
<code>UserHooks</code>, which are then used by that instance of 
Pythia. When working with the parallelism framework, the 
<code>PythiaParallel::init</code> method can be used to set external 
pointers per <code>Pythia</code> object. It is important to note that 
this method is threaded, and so whatever code is provided must be 
thread safe. This may not always be possible, in which case an 
external <code>mutex</code> can be passed to 
<code>PythiaParallel::init</code> by reference, locked when 
necessary, and then unlocked. 
 
<pre> 
    mutex initMutex; 
    pythia.init([&amp;initMutex](Pythia* pythiaPtr) { 
      initMutex.lock(); 
      // Thread unsafe code here. 
      shared_ptr&lt;PowhegHooks&gt; powhegHooks = 
        make_shared&lt;PowhegHooks&gt;(); 
      initMutex.unlock() 
      // Thread safe code here. 
      pythiaPtr-&gt;setUserHooksPtr((UserHooksPtr)powhegHooks); 
    }); 
</pre> 
In this example, the locking prevents the creation of the 
<code>PowhegHooks</code> instances from being threaded (which actually 
is not necessary). 
 
A similar issue can arise when loading plugins via the 
<code>Init:plugins</code> setting. Here, Pythia will automatically 
create a new plugin instance per thread, but again, if the plugin 
itself is not thread safe this can cause issues. To overcome this issue, 
every <code>PhysicsBase</code> object has a <code>mutexPtr</code>, 
corresponding to the top-level <code>PythiaParallel</code> object. 
Whenever this mutex is locked, then the subsequent code is not threaded. 
<pre> 
    mutexPtr->lock(); 
    // Thread unsafe code can be written here. 
    mutexPtr->unlock(); 
    // Thread safe code can be written here. 
</pre> 
This can be used within any method of a <code>PhysicsBase</code> derived 
class. All classes that can be passed to a <code>Pythia</code> instance by 
external pointer are derived from <code>PhysicsBase</code>. By default, 
plugins are assumed to not be thread safe unless the macro 
<code>PYTHIA8_PLUGIN_PARALLEL(true)</code> is explicitly set. See 
<aloc href="Plugins">Plugins</aloc> for more details. 
 
Finally, for plugins, it may be useful to combine the output of the 
threads together again. This can be accomplished via the 
<code>PythiaParallel::stat(bool combine)</code> method. If 
<code>combine = true</code>, then the following method is called for 
the first instance of the plugin. 
<pre> 
    void onStat(vector&lt;PhysicsBase*&gt; pluginPtrs, Pythia* pythiaPtrIn) { 
      // Loop over the plugin from each thread. 
      for (PhysicsBase* pluginPtr : pluginPtrs) { 
        // Skip the plugin that this method was called for. 
        if (pluginPtr == this) continue; 
        // Access all other plugins. 
        PluginClass* pluginNow = dynamic_cast&lt;PluginClass*&gt;(pluginPtr); 
        // Perform whatever merging is necessary here. 
      } 
      // Call onStat for this plugin instance. 
      onStat(); 
    } 
</pre> 
Here, <code>pluginPtrs</code> is a vector of the plugin pointer from 
each thread. Each of these plugin pointers then needs to be recast 
back to the plugin class. Finally, the <code>onStat()</code> for this 
instance of the plugin should be called. This ensures that the 
<code>onStat</code> behaviour is the same between a single 
<code>Pythia</code> instance, and <code>PythiaParallel</code>. 
 
</chapter> 
