<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">\chapter{Recurring Performance Issues}
\label{chap:issues}

{\it Oh no!  It's actually slower in parallel!}---almost everyone's
exclamation the first time they try to parallelize code

\bigskip

The available parallel hardware systems sound wonderful at first.  But
everyone who uses such systems has had the experience of
enthusiastically writing his/her first parallel program, anticipating
great speedups, only to find that the parallel code actually runs more
slowly than the original nonparallel program.  

In this chapter, we highlight some major issues that will pop up
throughout the book.

\section{Communication Bottlenecks}

Whether you are on a shared-memory, message-passing or other platform,
communication is always a potential bottleneck:  

\begin{itemize}

\item On a shared-memory system, the threads must contend with each
other in communicating with memory.  And the problem is exacerbated by
cache coherency transactions (Section \ref{cachecoherency}.  

\item On a cluster, even a very fast network is very slow compared to CPU
speeds.

\item GPUs are really fast, but their communication with their CPU hosts
is slow.  There are also memory contention issues as in ordinary
shared-memory systems.

\end{itemize}

Among other things, communication considerations largely drive the load
balancing issue, discussed next.

\section{Load Balancing}
\label{loadbalance}

Arguably the most central performance issue is \textbf{load
balancing,} i.e.  keeping all the processors busy as much as possible.
This issue arises constantly in any discussion of parallel processing.

% In the graphics ray-tracing
% application, for instance, suppose a ray is coming from the
% ``northeast'' section of the image, and is reflected by a solid object.
% Then the ray won't reach some of the ``southwest'' portions of the
% image, which then means that the processors assigned to those portions
% will not have any work to do which is associated with this ray. What we
% need to do is then try to give these processors some other work to do;
% the more they are idle, the slower our system will be.

A nice, easily understandable example is shown in Chapter 7 of the book,
{\it Multicore Application Programming:  for Windows, Linux and Oracle
Solaris}, Darryl Gove, 2011, Addison-Wesley.  There the author shows
code to compute the {\bf Mandelbrot set}, defined as follows.

Start with any number c in the complex plane, and initialize z to 0.
Then keep applying the transformation

\begin{equation}
z \leftarrow z^2 + c
\end{equation}

If the resulting sequence remains bounded (say after a certain number of
iterations), we say that c belongs to the Mandelbrot set.

Gove has a rectangular grid of points in the plane, and wants to
determine whether each point is in the set or not; a simple but
time-consuming computation is used for this determination.\footnote{You
can download Gove's code from
\url{http://blogs.sun.com/d/resource/map_src.tar.bz2}.  Most relevant is
{\bf listing7.64.c}.}

Gove sets up two threads, one handling all the points in the left half
of the grid and the other handling the right half.  He finds that the
latter thread is very often idle, while the former thread is usually
busy---extremely poor {\bf load balance}.  We'll return to this issue in
Section \ref{methodabest}.

\section{``Embarrassingly Parallel'' Applications}
\label{embpar}

The term {\bf embarrassingly parallel} is heard often in talk about
parallel programming.  

\subsection{What People Mean by ``Embarrassingly Parallel''}

Consider a matrix multiplication application, for instance, in which we
compute AX for a matrix A and a vector X.  One way to parallelize this
problem would be to have each processor handle a group of rows of A,
multiplying each by X in parallel with the other processors, which are
handling other groups of rows.  We call the problem embarrassingly
parallel, with the word ``embarrassing'' meaning that the problem is too
easy, i.e. there is no intellectual challenge involved.  It is pretty obvious
that the computation Y = AX can be parallelized very easily by splitting
the rows of A into groups.

By contrast, most parallel sorting algorithms require a great deal of
interaction.  For instance, consider Mergesort.  It breaks the vector to
be sorted into two (or more) independent parts, say the left half and
right half, which are then sorted in parallel by two processes.  So far,
this is embarrassingly parallel, at least after the vector is broken in
half.  But then the two sorted halves must be merged to produce the
sorted version of the original vector, and that process is {\it not}
embarrassingly parallel; it can be parallelized, but in a more complex,
less obvious manner.

Of course, it's no shame to have an embarrassingly parallel problem!  On
the contrary, except for showoff academics, having an embarrassingly
parallel application is a cause for celebration, as it is easy to
program.  

In recent years, the term {\bf embarrassingly parallel} has drifted to a
somewhat different meaning.  Algorithms that are embarrassingly parallel
in the above sense of simplicity tend to have very low communication
between processes, key to good performance.  That latter trait is the
center of attention nowadays, so the term {\bf embarrassingly parallel}
generally refers to an algorithm with low communication needs.

For that reason, many people would NOT considered even our prime finder
example in Section \ref{pthreadsprime} to be embarrassingly parallel.
Yes, it was embarrassingly easy to write, but it has high communication
costs, as both its locks and its global array are accessed quite often.

On the other hand, the Mandelbrot computation described in Section
\ref{loadbalance} is truly embarrassingly parallel, in both the old and
new sense of the term.  There the author Gove just assigned the points
on the left to one thread and the rest to the other thread---very
simple---and there was no communication between them.

\subsection{Iterative Algorithms}

Many parallel algorithms involve iteration, with a rendezvous of the
tasks after each iteration.  Within each iteration, the nodes act
entirely independently of each other, which makes the problem seem
embarrassingly parallel.  

But unless the granularity of the problem is coarse, i.e. there is
a large amount of work to do in each iteration, the communication
overhead will be significant, and the algorithm may not be considered
embarrassingly parallel.

\section{Static (But Possibly Random) Task Assignment Typically Better
Than Dynamic} 
\label{methodabest}

Say an algorithm generates t independent\footnote{Note the qualifying
term.} tasks and we have p processors to handle them.  In our
matrix-times-vector example of Section \ref{matvec}, say, each row of
the matrix might be considered one task.  A processor's work would then
be to multiply the vector by this processor's assigned rows of the
matrix.

How do we decide which tasks should be done by which processors?  In
{\bf static} assignment, our code would decide at the outset which
processors will handle which tasks.  The alternative, {\bf dynamic}
assignment, would have processors determine their tasks as the
computation proceeds.

In the matrix-times-vector example, say we have 10000 rows and 10
processors.  In static task assignment, we could pre-assign processor 0
rows 0-999, processor 1 rows 1000-1999 and so on.  On the other hand, we
could set up a {\bf task farm}, a queue consisting here of the numbers
0-9999.  Each time a processor finished handling one row, it would remove
the number at the head of the queue, and then process the row with that
index.

It would at first seem that dynamic assignment is more efficient, as it
is more flexible.  However, accessing the task farm, for instance,
entails communication costs, which might be very heavy.  In this section, we
will show that it's typically better to use the static approach, though
possibly randomized.\footnote{This is still static, as the randomization
is done at the outset, before starting computation.}

\subsection{Example:  Matrix-Vector Multiply}

Consider again the problem of multiplying a vector X by a large matrix
A, yielding a vector Y.  Say A has 10000 rows and we have 10 threads.  
Let's look at little closer at the static/dynamic tradeoff outlined
above.  For concreteness, assume the shared-memory setting.

There are several possibilities here:

\begin{itemize}

\item {\bf Method A:}  We could simply divide the 10000 rows into chunks
of 10000/10 = 1000, and parcel them out to the threads.  We would
pre-assign thread 0 to work on rows 0-999 of A, thread 1 to work on rows
1000-1999 and so on.  

This is essentially OpenMP's {\bf static} scheduling policy, with
default chunk size.\footnote{See Section \ref{schedulework}.} 

There would be no communication between the threads this way, but there
could be a problem of load imbalance.  Say for instance that by chance
thread 3 finishes well before the others.  Then it will be idle, as all
the work had been pre-allocated.

\item {\bf Method B:} 

OpenMP's {\bf dynamic} policy does what the name implies, which can be
described as follows (in non-OpenMP terms):

We would have a shared variable named,say, {\bf nextchunk}
similar to {\bf nextbase} in our prime-finding program in Section
\ref{pthreadsprime}.  Each time a thread would finish a chunk, it would
obtain a new chunk to work on, by recording the value of {\bf nextchunk}
and incrementing that variable by 1 (all atomically, of course).  

This approach would have better load balance, because the first thread
to find there is no work left to do would be idle for at most 100 rows'
amount of computation time, rather than 1000 as above.  Meanwhile,
though, communication would increase, as access to the locks around {\bf
nextchunk} would often make one thread wait for another.\footnote{Why
are we calling it ``communication'' here?  Recall that in shared-memory
programming, the threads communicate through shared variables.  When one
thread increments {\bf nextchunk}, it ``communicates'' that new value to
the other threads by placing it in shared memory where they will see it,
and as noted earlier contention among threads to shared memory is a
major source of potential slowdown.}

\item {\bf Method C:}  So, Method A above minimizes communication at the
possible expense of load balance, while the Method B does the opposite.
OpenMP also offers the {\bf guided} policy, which is like {\bf dynamic}
except the chunk size decreases over time.  In the earlier part of a
run, the large chunk size reduces communication, while later the small
chunk size avoid load imbalance.

\end{itemize}


I will now show that in typical settings, the Method A above (or a
slight modification) works well.  To this end, consider a chunk
consisting of m tasks, such as m rows in our matrix example above, with
times $T_1, T_2,..., T_m$.  The total time needed to process the chunk
is then $T_1+...,T_m$.  

The $T_i$ can be considered random variables; some tasks take a long
time to perform, some take a short time, and so on.  As an idealized
model, let's treat them as independent and identically distributed
random variables.  Under that assumption (if you don't have the
probability background, follow as best you can), we have that the mean
({\bf expected value}) and variance of total task time are

$$
E(T_1+...,T_m) = m E(T_1)
$$

and

$$
Var(T_1+...,T_m) = m Var(T_1)
$$

Thus

$$
\frac
{\textrm{standard deviation of chunk time}}
{\textrm{mean of chunk time}} 
~ \sim ~ O \left (\frac{1}{\sqrt{m}} \right )
$$

In other words:

\begin{itemize}

\item run time for a chunk is essentially constant if m is large, and

\item there is essentially no load imbalance in Method A

\end{itemize}

Since load imbalance was the only drawback to Method A and we now see
it's not a problem after all, then Method A is best.

For more details and timing examples, see N. Matloff, ``Efficient
Parallel R Loops on Long-Latency Platforms,'' {\it Proceedings of the
42nd Interface between Statistics and Computer Science}, Rice
University, June 2012.\footnote{As noted in the Preface to this book, I
occasionally refer here to my research, to illustrate for students the
beneficial interaction between teaching and research.}  

\subsection{Load Balance, Revisited}

But what about the assumptions behind that reasoning?  Consider for
example the Mandelbrot problem in Section \ref{loadbalance}.  There were
two threads, thus two chunks, with the tasks for a given chunk being
computations for all the points in the chunk's assigned region of the
picture.

Gove noted there was fairly strong load imbalance here, and that the
{\it reason} was that most of the Mandelbrot points turned out to be in
the left half of the picture!  The computation for a given point is
iterative, and if a point is not in the set, it tends to take only a few
iterations to discover this.  That's why the thread handling the right
half of the picture was idle so often.

So Method A would not work well here, and upon reflection one can see
that the problem was that the tasks within a chunk were not independent,
but were instead highly correlated, thus violating our mathematical
assumptions above.  Of course, before doing the computation, Gove didn't
know that it would turn out that most of the set would be in the left
half of the picture.  But, one could certainly anticipate the correlated
nature of the points; if one point is not in the Mandelbrot set, its
near neighbors are probably not in it either.

But Method A can still be made to work well, via a simple modification:
Simply form the chunks randomly.  In the matrix-multiply example above,
with 10000 rows and chunk size 1000, do NOT assign the chunks
contiguously.  Instead, generate a random permutation of the numbers
0,1,...,9999, naming them $i_0, i_1,...,i_{9999}$.  Then assign thread 0
rows $i_0-i_{999}$, thread 1 rows $i_{1000}-i_{1999}$, etc.

In the Mandelbrot example, we could randomly assign rows of the picture,
in the same way, and avoid load imbalance.

So, actually, Method A, or let's call it Method A', will still typically work
well.

\subsection{Example:  Mutual Web Outlinks}
\label{mutlinks}

Here's an example that we'll use at various points in this book:

\begin{quote}
{\bf Mutual outlinks in a graph:}  

Consider a network graph of some kind, such as Web links. For any two
vertices, say any two Web sites, we might be interested in mutual
outlinks, i.e. outbound links that are common to two Web sites. 
Say we want to find the number of mutual outlinks, averaged over
all pairs of Web sites.

Let A be the {\bf adjacency matrix} of the graph.  Then the mean of
interest would be found as follows:

\begin{lstlisting}[numbers=left]
sum = 0
for i = 0...n-2
   for j = i+1...n-1
      count = 0
      for k = 0...n-1 count += a[i][k] * a[j][k]
mean = sum / (n*(n-1)/2)
\end{lstlisting}

Say again n = 10000 and we have 10 threads.  We should not simply assign
work to the threads by dividing up the i loop, with thread 0 taking the
cases i = 0,...,999, thread 1 the cases 1000,...,1999 and so on.  This
would give us a real load balance problem.  Thread 8 would have much
less work to do than thread 3, say.

We could randomize as discussed earlier, but there is a much better
solution:  Just \underline{pair} the rows of A.  Thread 0 would handle
rows 0,...,499 and 9500,...,9999, thread 1 would handle rows 500,999 and
9000,...,9499 etc.  This approach is taken in our OpenMP implementation,
Section \ref{ompmutlinks}.

In other words, Method A still works well.

\end{quote}

In the mutual outlinks problem, we have a good idea beforehand as to how
much time each task needs, but this may not be true in general.  An
alternative would be to do {\it random} pre-assignment of tasks to
processors.

On the other hand, if we know beforehand that all of the tasks should
take about the same time, we should use static scheduling, as it might
yield better cache and virtual memory performance.

\subsection{Work Stealing}

There is another variation to Method A that is of interest today, called
{\bf work stealing}.  Here a thread that finishes its assigned work and
has thus no work left to do will ``raid'' the work queue of some other
thread.  This is the approach taken, for example, by the elegant Cilk
language.  Needless to say, accessing the other work queue is going to
be expensive in terms of time and memory contention overhead.

\subsection{Timing Example}
\label{mandelbrottiming}

I ran the Mandelbrot example on a shared memory machine with four cores,
two threads per core, with the following results for eight threads, on
an 8000x8000 grid:

\begin{tabular}{|r|r|}
\hline
policy &amp; time \\ \hline 
\hline
static &amp; 47.8 \\ \hline
dynamic &amp; 21.4 \\ \hline
guided &amp; 29.6 \\ \hline
random &amp; 15.7 \\ \hline
\end{tabular}

Default values were used for chunk size in the first three cases.  I did
try other chunk sizes for the {\bf dynamic} policy, but it didn't make
much difference.  See Section \ref{mandelbrotcode} for the code.

Needless to say, one shouldn't overly extrapolate from the above
timings, but it does illustrate the issues.

\subsection{Example:  Extracting a Submatrix}

OpenMP implements the various scheduling algorithms via its {\bf for}
pragma, e.g.

\begin{lstlisting}
#pragma omp for schedule(guided)
\end{lstlisting}

We'll cover this in more detail in Chapter \ref{chap:omp}, but here is
an example of the default, static scheduling with chunk size 1.

\begin{lstlisting}

// creates a new matrix from a group of {\bf nColsSubM}
// consecutive columns of an old one, starting at column 
// firstCol} of the old matrix

#include&lt;omp.h&gt;
#include&lt;stdio.h&gt;
#include&lt;stdlib.h&gt;
#include&lt;string.h&gt;

int onedim(int row, int col, int nCols) { return row * nCols + col; }

int *getCols(int *m, int nRowsM, int nColsM, int firstCol, int nColsSubM) {
   int *subMat;
   #pragma omp parallel
   {  int *startFrom,*startTo;

      #pragma omp single
      subMat = malloc(nRowsM * nColsSubM * sizeof(int));

      #pragma omp for
      for (int row = 0; row &lt; nRowsM; row++) {
         startFrom = m + onedim(row,firstCol,nColsM);
         startTo = subMat + onedim(row,0,nColsSubM);
         memcpy(startTo, startFrom, nColsSubM * sizeof(int));
      }
   }
   return subMat;
}

int main() {
   // intended as a 4x3 matrix
   int z[12] = {5,12,13,6,2,22,15,3,1,2,3,4};
   omp_set_num_threads(2);
   // extract the last 2 cols; should be
   // 12 13
   // 2 22
   // 3 1
   // 3 4
   int *outM = getCols(z,4,3,1,2);
   for (int i = 0; i &lt; 4; i++) {
      printf("%d %d\n",outM[2*i],outM[2*i+1]);
   }
}



\end{lstlisting}

\section{Latency and Bandwidth}
\label{latencybandwidth}

We've been speaking of communications delays so far as being monolithic,
but they are actually (at least) two-dimensional.  The key measures are
{\bf latency} and {\bf bandwidth}:

\begin{itemize}

\item Latency is the time it takes for one bit to travel for source to
destination, e.g. from a CPU to memory in a shared memory system, or
from one computer to another in a cluster.

\item Bandwidth is the number of bits per unit time that can be input
into the communications channel.  This can be affected by factors such
as bus width in a shared memory system and number of parallel network
paths in a message passing system, and also by the speed of the links.

\end{itemize}

It's helpful to think of a bridge, with toll booths at its entrance.
Latency is the time needed for one car to get from one end of the bridge
to the other.  Bandwidth is the number of cars that can enter the bridge
per unit time.   We can reduce latency by increasing the speed limit,
and can increase bandwidth by improving the speed by which toll takers
can collect tolls, and increasing the number of toll booths.

{\bf Latency hiding:}

One way of dealing with long latencies is known as {\bf latency hiding}.
The idea is to do a long-latency operation in parallel with something
else.

For example, GPUs tend to have very long memory access times, but this
is solved by having many pending memory accesses at the same time.
During the latency of some accesses, earlier ones that have now
completed can now be acted upon (Section \ref{coal}).

\section{Relative Merits:  Performance of Shared-Memory Vs. Message-Passing}
\label{sharedbetter}

My own preference is shared-memory, but there are pros and cons to each
paradigm.

It is generally believed in the parallel processing community that the
shared-memory paradigm produces code that is easier to write, debug and
maintain than message-passing.  See for instance R. Chandra, {\it
Parallel Programming in OpenMP}, MKP, 2001, pp.10ff (especially
Table 1.1), and M. Hess {\it et al}, Experiences Using OpenMP Based on
Compiler Directive Software DSM on a PC Cluster, in {\it OpenMP Shared
Memory Parallel Programming: International Workshop on OpenMP
Applications and Tools}, Michael Voss (ed.), Springer, 2003, p.216.

On the other hand, in some cases message-passing can produce faster
code.  Consider the Odd/Even Transposition Sort algorithm, for instance.
Here pairs of processes repeatedly swap sorted arrays with each other.
In a shared-memory setting, this might produce a bottleneck at the
shared memory, slowing down the code.  Of course, the obvious
solution is that if you are using a shared-memory machine, you should
just choose some other sorting algorithm, one tailored to the
shared-memory setting.

There used to be a belief that message-passing was more {\bf scalable},
i.e. amenable to very large systems.  However, GPU has demonstrated that
one can achieve extremely good scalability with shared-memory.

As will be seen, though, GPU is hardly a panacea.  Where, then, are
people to get access to large-scale parallel systems?  Most people do
not (currently) have access to large-scale multicore machines, while
most do have access to large-scale message-passing machines, say in
cloud computing venues.  Thus message-passing plays a role even for
those of us who preferred the shared-memory paradigm.

Also, hybrid systems are common, in which a number of shared-memory
systems are tied together by, say, MPI.

\section{Memory Allocation Issues}

Many algorithms require large amounts of memory for intermediate storage
of data.  It may be prohibitive to allocate this memory statically, i.e.
at compile time.  Yet dynamic allocation, say via {\bf malloc()} or
C++'s {\bf new} (which probably produces a call to {\bf malloc()
anyway}, is very expensive in time.

Using large amounts of memory also can be a major source of overhead due
to cache misses and page faults.

One way to avoid {\bf malloc()}, of course, is to set up static arrays
whenever possible.

There are no magic solutions here.  One must simply be aware of the
problem, and tweak one's code accordingly, say by adjusting calls to
{\bf malloc()} so that one achieves a balance between allocating too
much memory and making too many calls.

\section{Issues Particular to Shared-Memory Systems}

This topic is covered in detail in Chapter \ref{chap:shared}, but is so
important that the main points should be mentioned here.

\begin{itemize}

\item Memory is typically divided into {\bf banks}.  If more than one
thread attempts to access the same bank at the same time, that
effectively serializes the program.

\item There is typically a cache at each processor.  Keeping the
contents of these caches consistent with each other, and with the memory
itself, adds a lot of overhead, causing slowdown.

\end{itemize}

In both cases, awareness of these issues should impact how you write
your code.

See Sections \ref{banks} and \ref{cacheissues}.

</pre></body></html>