\chapter{MapReduce Computation} 
\label{chap:mapreduce}

As the world emerged into an era of Big Data, demand grew for a
computing paradigm that (a) is generally applicable and (b) works on
{\it distributed} data.The latter term means that data is physically
distributed over many chunks, possibly on different disks and maybe even
different geographical locations.  Having the data stored in a
distributed manner facilitates parallel computation --- different chunks
can be read simultaneously --- and also enables us to work with data
sets that are too large to fit into the memory of a single machine.
Demand for such computational capability led to the development of
various systems using the {\it MapReduce} paradigm.

MapReduce is really a form of the scatter-gather pattern we've seen
frequently in this book, with the added feature of a sorting operation
added in the middle.  In rough form, it works like this.  The input is
in a (distributed) file, fed into the following process: 

\begin{itemize}

\item {\bf Map phase:}  There are various parallel processes known as
{\it mappers}, all running the same code.  For each line of the input
file, the mapper handling that chunk of the file reads the line,
processes it in some way, and then emits an output line, consisting of a
key-value pair.

\item {\bf Shuffle/sort phase:}  All the mapper output lines that share
the same key are gathered together.

\item {\bf Reduce phase:} There are various parallel processes known as
{\it reducers}, all running the same code.  Each reducer will work on
its own set of keys, i.e.\ for any given key, all mapper output lines
having the same key will go to the same reducer.  Moreover, the lines
fed into any given reducer will be sorted by key.

\end{itemize}

\section{Apache Hadoop}

At this writing, the most popular MapReduce software package is Hadoop.
It is written in Java, and is most efficiently used in that language (or
C++), but it includes a {\it streaming} feature that enables one to use
Hadoop from essentially any language, including R.  Hadoop includes is
into distributed file system, unsurprisingly called the Hadoop
Distributed File System (HDFS).  Note that one advantage of HDFS is that
it is replicated, thus achieving some degree of fault tolerance.

\subsection{Hadoop Streaming}

As noted, Hadoop is really written for Java or C++ applications.
However, Hadoop can work with programs in any language under Hadoop's
streaming option, by reading from {\bf stdin} and writing to {\bf
stdout}, in text, line-oriented form in both cases.  

Input to the mappers is from an HDFS file, and output from the reducers
is again to an HDFS file, one chunk per reducer.  The file line format is 

\begin{verbatim}
key \t data
\end{verbatim}

where \lstinline{\t} is the Tab character.

The usage of text format does cause some slowdown in numeric programs,
for the conversion of strings to numbers and vice versa, but again,
Hadoop is not designed for maximum efficiency.

\subsection{Example:  Word Count}
\label{hadoopwordcount}

The typical ``Hello World,'' introductory example is word count for a
text file.  The mapper program breaks a line into words, and emits
(key,value) pairs in the form of (word,1).  (If a word appears several
times in a line, there would be several pairs emitted for that word.) In
the Reduce stage, all those 1s for a given word are summed, giving a
frequency count for that word.  In this way, we get counts for all
words.

Here is the mapper code:

\begin{lstlisting}
#! /usr/bin/env Rscript

# wordmapper.R

si <- file("stdin",open="r")
while (length(inln <- 
      scan(si,what="",nlines=1,quiet=TRUE,blank.lines.skip=FALSE))) {
   for (w in inln) {
      cat(w,"\t 1\n")
   }
}
\end{lstlisting}

And here is the reducer:

\begin{lstlisting}
#! /usr/bin/env Rscript

# wordreducer.R

si <- file("stdin",open="r")
oldword <- ""

while (length(inln <- scan(si,what="",nlines=1,quiet=TRUE))) {
   word <- inln[1]
   if (word != oldword) {
      if (oldword != "") 
         cat(oldword,"\t ",count,"\n")
      oldword <- word
      count <- 1
   } else {
      count <- count + as.integer(inln[2])
   }
}

\end{lstlisting}

The above code is not very refined, for instance treating {\it The} as
different from {\it the}.  The main point, though, is just to illustrate
the principles.

\subsection{Running the Code}

I ran the following code from the top level of the Hadoop directory
tree, with obvious modifications possible for other run points.  First,
I needed to place my data file, {\bf rnyt},\footnote{The file was the contents
of the article ``Data Analysts Captivated by R’s Power,'' {\it New York
Times}, January 6, 2009.}  

\begin{lstlisting}
$ bin/hadoop fs -put ../rnyt rnyt
$ bin/hadoop jar contrib/streaming/*.jar \
     -input rnyt \
     -output wordcountsnyt \
     -mapper ../wordmapper.R 
     -reducer ../wordreducer.R
\end{lstlisting}

That first command specifies that it will be for the file system ({\bf
fs}, and that I am placing a file in that system.  My ordinary version
of the file was in my home directory.

Hadoop, being Java-based, runs Java archive, {\bf .jar} files.  So, the
second command above specifies that I want to run in streaming mode.  It
also states that I wan the output to go to a file {\bf wordcountsnyt} in
my HDFS system.  Finally, I specify my mapper and reducer code files.
in my HDFS system, after which I ran the program.  I allowed Hadoop to
use its default values for the number of mappers and reducers, and could
have specified them above if desired.

Recall that the final output comes in chunks in the HDFS.  Here's how to
check (some material not shown), and to view the actual file contents:

\begin{lstlisting}
$ bin/hadoop fs -ls wordcountsnyt

Found 3 items
-rw-r--r--   1 ... /user/matloff/wordcountsnyt/_SUCCESS
drwxr-xr-x   - ... /user/matloff/wordcountsnyt/_logs
-rw-r--r--   1 ... /user/matloff/wordcountsnyt/part-00000
$ bin/hadoop fs -cat wordcountsnyt1/part-00000
1         NA 
$2        1 
1,600     1 
18th      1 
1991,     1 
1996,     1 
2009      1 
2009,     1 
250,000           1 
6,        1       
7,        1 
A         1 
ASHLEE    1 
According         1 
America,          1 
Analysts          2 
Anne      1       
Another           1 
Apache,           1 
Are       1       
At        1 
...
\end{lstlisting}

So, in HDFS, one distributed file is stored as a directory, with the
chunks in {\bf part-00000}, {\bf part-00001} and so on.  We only had
enough data to fill one chunk here.

\subsection{Analysis of the Code}

The first thing to notice is that these two R files are not executed
directly by R, but instead under Rscript.  This is standard for
running R in {\it batch}, i.e.\ noninteractive, mode.

Next, as noted earlier, input to the mappers is from {\bf stdin}, in
this case from the redirected file {\bf rnyt} in my HDFS, seen here in
the identifier {\bf si} in the mapper.  Final output is to {\bf stdout},
redirected to the specified HDFS file, hence the call to {\bf cat()} in
the reducer.  The mapper output goes to the shuffle and then to the
reducers, again using STDOUT, visible here in the call to {\bf cat()} in
the mapper code.  

The reader might be wondering here about the line

\begin{lstlisting}
count <- count + as.integer(inln[2])
\end{lstlisting}

in the reducer.  The way things have been described so far, it would
seem that the expression {\bf as.integer(inln[2])} --- a word count
output from a mapper --- should always be 1.  However, there is more to
the story, as Hadoop also allows one to specify {\it combiner} code, as
follows.

Remember, all the communication, e.g.\ from the mappers to the shuffler,
is via our network, with one (key,value) pair per network message.  So,
we may have an enormous number of short messages, thus incurring the
network latency penalty many times, as well as huge network congestion
due to, for instance, many mappers trying to use the network at once.
The solution is to have each mapper try to coalesce its messages before
sending to the shuffler.  

The coalescing is done by a {\it combiner} specified by the user.  Often
the combiner will be the same as the reducer.  So, what occurs is that
each mapper will run the reducer on its own mapper output, then send the
combiner output to the shuffler, after which it goes to the reducers as
usual.

Thus in our word count example here, when a line arrives at a reducer,
its count field may already have a value greater than 1.  The combiner
code, by the way, is specified via the {\bf -combiner} field in the run
command, like {\bf -mapper} and {\bf -reducer}.

\subsection{Role of Disk Files}

As ntoed, Hadoop has its own file system, HDFS, which is built on top of
the native OS' file system of the machines.  It is replicated for the
sake of reliability, with each HDFS block existing in at least 3 copies,
i.e.  on at least 3 separate disks.  Very large files are possible, in
some cases spanning more than one disk/machine.

Disk files play a major role in Hadoop programs:

\begin{itemize}

\item Input is from a file in the HDFS system.

\item The output of the mappers goes to temporary files in the native
OS' file system.

\item Final output is to a file in the HDFS system.  As noted earlier,
that file may be distributed across several disks/machines.

\end{itemize}

Note that by having the input and output files in HDFS, we minimize
communications costs in shipping the data between nodes of a cluster.
The slogan used is ``Moving computation is cheaper than moving data.''
However, all that disk activity can be quite costly in terms of run
time.

\section{Other MapReduce Systems}

As of late 2014, there has been increasing concern regarding Hadoop's
performance.  One of the problems is that one cannot keep
intermediate results in memory between Hadoop runs.  This is a serious
problem, for instance, with iterative or even multi-pass algorithms.

The Spark package now being developed aims to remedy many of Hadoop's
shortcomings.  Early reports indicate some drastic speed improvements,
while retaining the ability to read HDFS files, and continuing to have
fault tolerance features.  

\section{R Interfaces to MapReduce Systems}

Given the widespread usage of Hadoop, a number of R interfaces have been
developed.  The most popular is probably {\bf rmr}, developed by
Revolution Analytics, and RHIPE, written by Saptarshi Guha as part of
his PhD dissertation.  An R interface package, {\bf sparkr}, is
available.

\section{An Alternative:  ``Snowdoop''}
\label{snowdoop}

So, what does Hadoop really give us?  The two main features are (a)
distributed data access and (b) an efficient distributed file sort.
Hadoop works well for many applications, but a realization developed
that Hadoop can be very  slow, and very limited in available data
operations.

Both of those shortcomings are addressed to a large extent by the new
kid on the block, Spark.  Spark is apparently much faster than Hadoop,
sometimes dramatically so, due to strong caching ability and a wider
variety of available operations.  Recently {\bf distributedR} has also
been released, again with the goal of using R on voluminous data sets,
and there is also the more established {\bf pbdR}.

But even Spark suffers a very practical problem, shared by the others
mentioned above.  All of these systems are complicated.  There is a
considerable amount of configuration to do, worsened by dependence on
infrastructure software such as Java or MPI, and in some cases by
interface software such as rJava.  Some of this requires systems
knowledge that many R users may lack.  And once they do get these
systems set up, they may be required to design algorithms with world
views quite different from R, even though they are coding in R.

So, do we really need all that complicated machinery?   Hadoop and Spark
provide efficient distributed sort operations, but if one's application
does not depend on sorting, we have a cost-benefit issue here.

Here is an alternative, a general approach rather than a package, which
I call ``Snowdoop'':  One simply does one's own chunking of files into
distributed mini-files, and then uses Snow or some other general R tool
on those files.

\subsection{Example:  Snowdoop Word Count}

Let's use as our example word count, the ``Hello World'' of MapReduce.
It determines which words are in a text file, and calculates frequency
counts for each distinct word:

\begin{lstlisting}[numbers=left]
# each node executes this function
wordcensus <- function(basename,ndigs) {
   fname <- filechunkname(basename,ndigs)
   words <- scan(fname,what="")
   tapply(words,words,length, simplify=FALSE)
}

# manager
fullwordcount <- function(cls,basename,ndigs) {
   setclsinfo(cls)  # give workers ID numbers, etc.
   counts <- clusterCall(cls,wordcensus,basename,ndigs)
   addlistssum <- function(lst1,lst2)
      addlists(lst1,lst2,sum)
   Reduce(addlistssum,counts)
}
\end{lstlisting}

The above code makes use of the following routines, which are general
and are used in many ``Snowdoop'' applications.  These and other
utilities are included in my {\bf partools} package.  Here are the call
forms:

\begin{lstlisting}
# give each cluster node an ID, etc.
setclsinfo(cls)

# "add" lists lst1, lst2, applying the operation 'add' to elements in
# common, copying non-null others
addlists(lst1,lst2,add)

# form the file name basename.i, where i is the ID of this node unless
# nodenum is specified
filechunkname(basename,ndigs,nodenum=NULL)
\end{lstlisting}

All pure R!  No Java, no configuration.  Indeed, it's worthwhile
comparing to the word count example in the {\bf sparkr} distribution.
There we see calls to {\bf sparkr} functions such as {\bf flatMap()},
{\bf reduceByKey()} and {\bf collect()}.  But the {\bf reduceByKey()}
function is pretty much the same as R's tried and true {\bf tapply()}.
The {\bf collect()} function is more or less our Snowdoop library
function {\bf addlists()}.  So, again, there is no need to resort to
Spark, Hadoop, Java and so on; we just use ordinary R.


We are achieving the parallel-read advantage of Hadoop and
Spark,\footnote{Note that neither Hadoop, Spark nor Snowdoop will achieve
full parallel reading if the file chunks are all on the same disk.}
{\it while avoiding the Hadoop/Spark configuration headaches and while
staying with the familiar R programming paradigm}.  In many cases, this
should be a highly beneficial tradeoff for us.  

Of course, this approach lacks the fault tolerance feature of Hadhoop
and Spark can, which can be quite advantageous.  And as of this writing,
it is not yet clear how well this scales, e.g.\ how well the {\bf
parallel} package works with very large numbers of nodes.  But Snowdoop
is an attractive approach for many applications.

\subsection{Example:  Snowdoop k-Means Clustering}

The k-means clustering problem is discussed in Section \ref{cluster}.
Read ahead, and then see how we can implement it here:

\begin{lstlisting}[numbers=left]
# k-means clustering, using Snowdoop

# chunked data with name xname, nitrs iterations, nclus clusters;
# assumes for simplicity that a cluster will never become empty; ctrs is
# the matrix of initial centroids

# assumes setclsinfo already called

kmeans <- function(cls,xname,nitrs,ctrs) {
   # will tell everyone to read their chunks; first, find cluster size
   # and compute number of digits in file suffixes
   addlistssum <- function(lst1,lst2) addlists(lst1,lst2,sum)
   for (i in 1:nitrs) {
      # for each data point, find the nearest centroid, and tabulate; at
      # each worker and for each centroid, we compute a vector whose
      # first component is the count of the number of data points whose
      # nearest centroid is that centroid, and whose remainining portion 
      # is the sum of all such data points
      tmp <- clusterCall(cls,findnrst,xname,ctrs)
      # sum over all workers
      tmp <- Reduce(addlistssum,tmp)
      # compute new centroids
      for (i in 1:nrow(ctrs)) {
         tmp1 <- tmp[[as.character(i)]]
         ctrs[i,] <- (1/tmp1[1]) * tmp1[-1]
      }
   }
   ctrs
}

findnrst <- function(xname,ctrs) {
   require(pdist)
   x <- get(xname)
   dsts <- matrix(pdist(x,ctrs)@dist,ncol=nrow(x))
   # dsts[,i] now has the distances from row i of x to the centroids 
   nrst <- apply(dsts,2,which.min)
   # nrst[i] tells us the index of the centroid closest to row i of x
   mysum <- function(idxs,myx) 
      c(length(idxs),colSums(x[idxs,,drop=F]))
   tmp <- tapply(1:nrow(x),nrst,mysum,x)
}

test <- function(cls) {
   m <- matrix(c(4,1,4,6,3,2,6,6),ncol=2)
   formrowchunks(cls,m,"m")
   initc <- rbind(c(2,2),c(3,5))
   kmeans(cls,"m",1,initc) 
}
\end{lstlisting}

So again, we are using chunked files as in Hadoop, but writing ordinary
R code, e.g. {\bf tapply()} and {\bf Reduce()}.  But most important, the
data at each worker persists across iterations.  In Hadoop, it would be
reread from disk at each iteration, and in Spark, we'd need to request
caching, but here it comes for free, no special effort needed.

