
\documentclass[11pt]{article}

\setlength{\oddsidemargin}{0in}
\setlength{\evensidemargin}{0in}
\setlength{\topmargin}{0.0in}
\setlength{\headheight}{0in}
\setlength{\headsep}{0.5in}
\setlength{\textwidth}{6.5in}
\setlength{\textheight}{8.5in}
\setlength{\parindent}{0in}
\setlength{\parskip}{0.1in}

\usepackage{times}     
\usepackage{psfig}     
\usepackage{fancyheadings}     

\begin{document}

\pagestyle{fancy}
\cfoot{Pipelining: \thepage}   

\title{Pipelining}

\author{N.S. Matloff \\
University of California at Davis}

\date{February 23, 1999}

\maketitle

\tableofcontents

\section{General Issues}

In spite of advances in chip technology, there are still real limits to
the degree of speed increase one can attain in a CPU, due to physical
constraints, e.g.  the speed of light, heat dissipation, and so on.
So, a crucial source of speed in modern systems---and even more
so for systems of the future---is to do operations in parallel.
One type of parallel processing is {\bf pipelining}.  

Pipelining is fairly analogous to an assembly line, say in an
automobile factory.  The process of building a car is broken down
into a number of stages, each one of approximately equal duration,
and at any given time different cars are at different stages of
production---but all simultaneously, the key to parallelism.

As a quick example, imagine a widget factory, in which the construction
of a widget is broken down into three separate stages, called stage A,
stage B and stage C, with each stage taking one minute of time.  Suppose
we have a continuing supply of raw materials for widget-making entering
the factory, and that at time 0, our first widget goes into production. 
It will be completely finished at time 3, but the point is that once 
first widget leaves stage A at time 1, the second widget goes into 
production.  While the first widget is at stage B, the second widget 
is at stage A.  When the first widget is at stage C, the second one 
is at B, and a third one now starts at A.

So, what is the rate of widget production?  Without pipelining, the
rate would be one widget produced every 3 minutes, but with pipelining
the rate is one widget produced every 1 minute!  For example, the second
widget will exit stage C one minute after the first widget, the third
widget will exit one minute after the second one, etc.  Every minute a
newly-completed widget will exit the pipe.

But all of this depends on having the continuing supply of widget-making
raw materials to enter the pipe, with one raw widget entering stage A per
minute.  If, for example, purchase orders for widgets come in only
sporadically, then there will rarely be more than one widget in production
at any given time, and the benefits of pipelining will be lost.

\subsection{Example:  Instruction Prefetch}

This is an example of pipelining with which you are already familiar, 
at least to some degree.  The processor will try to fetch the next 
instruction while processing the current one.

As a simple example consider a 3-stage instruction pipeline, with
stage A being the instruction fetch, stage B being the decode and
stage C being the execution.  Assume that the three stages have equal 
times.\footnote{Typically the instruction fetch will take more time,
but we might have an instruction cache to make this assumption
reasonable.  Note, by the way, that if the stage times \underline
{are} unequal, we have to set the clock to the time of the longest
stage.}  Consider instructions at addresses 100, 101 and 102.  Then
we could have the following sequence of events:

\begin{verbatim}
time 0:  start fetch of I100

time 1:  start decode of I100, start fetch of I101

time 2:  start execute of I100, start decode of I101, start fetch of I102
\end{verbatim}

and so on.

But what is the instruction-fetch analog of a ``lack of widget-making
materials''?  The answer is branch (jump) instructions!  Suppose, for
example, that I100 is ``if negative then jump to 132.''  We won't
know whether the jump is actually taken until the end of time slot 2, 
in which case the remainder of the pipe (I101, I102) will be useless.  
The instruction fetch for I132 won't be started until time 3, and its 
execution thus won't finish until the end of time 5.  In other words,
instead of ``widgets'' exiting the pipe on every clock, i.e. at the
ends of clock 2, clock 3, clock 4, clock 5, and so on, we will have 
them exiting at clocks 2 and 5, missing clocks 3 and 4.  We have
two wasted time slots.

\subsection{A More Elaborate Pipeline Example}

Consider a polynomial of degree 3:

\begin{equation}
a x ^ 3 + b x ^2 + c x + d
\end{equation}

Suppose that for fixed a, b, c and d, we wish to repeatedly evaluate
this polynomial, for different values of x, say because we are graphing
the polynomial.  

Say we are designing a special-purpose chip to evaluate third-degree
polynomials.  Since speed is undoubtedly an issue (otherwise, we would
not be designing a special-purpose chip), we should investigate
pipelined designs.

\subsubsection{A First Try}

We could set up the following pipeline:

\vspace{0.2in}
\par
\psfig{file=PolyPipe.eps}
\par

Here the A's and M's are adders and multipliers, respectively. 
Though not shown, there are also registers below each M, which hold
the value of x, which is passed in from the previous stage of the
pipeline, and registers just to the left of the M's, to hold the
value passed on by the previous stage.\footnote{We would not need a
register to the left of the M in the first stage, since its input
never changes. Make sure you understand this.  Ask if you do not
completely grasp it.}  Also not shown is the clock, which would
feed into each register.

The way the system works is as follows.  First, rewrite the polynomial
in the form

\begin{equation}
x \{ x [ x(a) + b ] + c \} + d
\end{equation}

Then at stage 0, M inputs a and x, and outputs x(a) to A, which then
forms x(a)+b and sends this value to M at stage 1.  The latter then
forms x[x(a)+b] and sends it to A, which forms x[x(a)+b]+c, and so on.

Note carefully that at any given time, there will be three values of x
in this circuit, a different x at each of the three stages.  Say for
example that a = 2, b = 5 and c = 1, and that the stream of x values
entering the pipe is 6, 10, 4, 29, ... If the 6 enters at time 0, then
at time 2, then the x at the third (i.e. rightmost) stage will be 6; at
the second stage the x value will be 10; and at the first stage the x
value will be 4.  Moreover, at that time, the other input to the M unit
at the second stage will be $2 \times 10 + 5$, which is the output of
the first stage for x = 10.  In other words, both inputs to M at the
second stage will correspond to x = 10.  Similarly, at that time both
inputs to M at the third stage will correspond to x = 6. 

Note that the clock here will have to be set to the maximum M time plus
the maximum A time.  This combined time might be long.

\subsubsection{A More Refined Approach}

We could split each stage above into two stages:

\vspace{0.2in}
\par
\psfig{file=PolyPipe2.eps}
\par

Here again there would be registers: to the left of each M (except for
the first); to the left of each A, to hold incoming results from the
previous stage; one beneath each M; and one \underline{between} each
pair of consecutive Ms, in order to force each x value to take two clock
cycles (instead of one) to get from one M to the next.  Again, the clock
lines are not shown either.

How much faster would this new pipe be?  The answer depends on the
relative times of the M and A units.  

An M unit is likely to need much more time than an A unit, since the M unit
itself probably does some additions.  Thus the M's will be the ``weak
link in the chain,'' and the clock cycle time will have to be set to
the maximum time needed to do a multiplication.  If that time is, say,
4 times longer than the maximum time for an addition, then, this new
pipe will have a cycle time only 20\% shorter than the original 
one, and thus a throughput rate only about 20\% higher.\footnote{Again,
make sure you understand this.}

Note that even though this new pipe has twice as many stages as the
original design, for any given x, the computation time will not
double.  It will increase somewhat, though, owing to the fact
that the A time is less than the M time.

\subsubsection{A Feedback Version}

If we were short on space but willing to sacrifice some speed,
we could have a feedback version of the polynomial pipe:

\vspace{0.2in}
\par
\psfig{file=PolyPipe3.eps}
\par

Again the registers and clock lines are not shown.  Note, though,
that the clock line will feed into not only the registers but
also the box labeled ``mod 3'' and to the bottom MUX.

Here we save space by having only one M and only one A.  The
intermediate values are repeatedly fed back into the M/A
unit.  The clock time, mod 3, controls the multiplexer, so
that the proper one of b, c and d is input to the adder.

In this design, the computation for only one x can be handled
at any given time.  However, a refinement could be made in
which we handled two x computations at a time; here, the M
unit would repeatedly input the two x's in alternative clock
cycles, requiring another MUX.  When the addition for one x
is being done, the multiplication for the other x is being done. 

\subsection{Example:  Floating-Point Addition}

\subsubsection{Review}

{\bf Floating-point} numbers, as in for example the type {\bf float} 
in the C language, store numbers corresponding to ``scientific 
notation'' such as $7.2 \times 10^5$.  In the latter number, 7.2 is 
called the {\bf mantissa} and 5 is the {\bf exponent}.  If for 
instance we have the C code
 
\begin{verbatim}
float X;
\end{verbatim}

Then on a 32-bit machine, the compiler would allocate one 32-bit
word for X (just as it would if X had been of type {\bf int}).  Within
that word, a certain set of the 32 bits would be used for the mantissa
and the rest for the exponent.  (Both of these values would be signed,
i.e. could take on either positive or negative values.)

To make this concrete, here is a simplified example.  Suppose our word
size is 6 bits, and that we only wish to store nonnegative numbers.  The
leftmost (i.e. most significant) 4 bits form the mantissa m and the last
2 bits comprise the exponent e, which is going to serve on a base of 2.
The m field is to be considered to be an unsigned integer, while e is a
2s-complement signed integer.  The number stored is $m \times 2^e$.

As an example, verify for yourself that

\begin{verbatim}
001110 + 010111 = 110110
\end{verbatim}

\subsubsection{The Pipeline}

A floating-point adder could be implemented as a four-stage pipe
as follows:

\begin{itemize}
\item [S1.]  Compare exponents.

\item [S2.]  Shift the mantissa paired with the lesser exponent so that
two exponents match, adjusting the paired exponent accordingly.

\item [S3.]  Add mantissas (updating exponents, if carry-out occurs).

\item [S4.]  Normalize to standard form.
\end{itemize}

The standard form referred to in S4 involves making sure that the first
bit in the mantissa is a 1, except if the mantissa is 0.  (We did not
do this in our simplified example above.)  To normalize to standard
form, we shift the mantissa left and subtract from the exponent.

Once again, pipelining will be useful only if we have a continuing
supply of tasks entering the pipe.  The example in the next section will
provide such a situation.

\subsection{Memory Considerations}

As stressed earlier, the key to the success of a pipelined design
is to keep the pipe filled.  If the pipe is part of a CPU, this
may be difficult, as it may be hard to bring in numbers from memory
quickly enough.

Consider the case of {\it vector processors}, so called because
their instruction sets include operations whose operands are
vectors rather than scalars.  As our running example here, consider
an instruction VADD X,Y,N where X and Y are arrays of type {\bf float}, 
and N is an {\bf int} denoting the array length.  The instruction will
calculate X[0]+Y[0] and put the result back in X[0], put X[1]+Y[1] in
X[1], and so on.  

Please note carefully that VADD would be one single instruction,
doing the work normally done by a whole {\bf for} loop of instructions.
It should run faster than the loop, for at least two reasons :

\begin{itemize}
\item  Fewer instruction fetches.

\item  Pipelined operation.
\end{itemize}

For the ALU, we will assume that the floating-point addition unit
is the four-stage pipe described earlier.  So, in the first time 
period, the exponents of X[0] and Y[0] will be compared.  In the 
second time period, one of the mantissas of X[0] and Y[0] will be 
shifted, \underline{and} the exponents of X[1] and Y[1] compared, 
etc.  The problem is fetching the components of X and Y fast enough 
so that one pair can enter the pipe in each time period.

In order to do this, we can use {\it low-order memory interleaving}.
Consider the following setup:

\vspace{0.2in}
\par
\psfig{file=Interleaving.eps}
\par
       
For simplicity, we are assuming 16-bit word and address widths.  The
notation A[m:n] means address bits m through n (15 being the most
significant), with a similar notation D[m:n] for data bits.  Again
for simplicity, suppose that each Mj stores 16K words; that would
mean that each chip has 14 address pins, attached to the lines
labeled A[15:2], and 16 data pins, leading to the MUX.\footnote{We 
might have all the Mj hanging from a single data bus leading to the 
CPU, with M0 sending a value first, then M1 immediately afterward, 
then M2, etc.  In that case, we could even describe the memory as
being pipelined.}

Suppose that individual bytes within a word are not addressable, so
that consecutive words have consecutive addresses.  Using low-order 
interleaving, word i will be stored in Mj, where j = i mod 4.  In
other words, word 0 is in M0, word 1 is in M1, word 2 is in M2,
word 3 is in M3, word 4 is back in M0, word 5 is in M1, and so on.
In other words, word m within Mn plays the role of word 4m+n
in the CPU's address space.

(Make absolutely sure that you understand all of these numbers and
relations before continuing.  You may wish to review the material on
memory chips and systems in the digital logic handout, {\it Digital
Logic---the Bare Minimum}.)

Let us see how this setup can be used to read a vector, say X, from 
memory in a very rapid manner.\footnote{Our example is of course not 
quite that simple.  We are actually reading two vectors, X and Y, and 
in addition are writing to one of them, X.  One possible solution
would be to use {\it multiport} memory, which allows more than one
access to a memory chip at a time.  But to keep things simple, suppose
we are just reading X.}

Suppose the CPU needs to read word 26, which is 0000000000011010 in
binary.  Then it puts 00000000000110 on the upper address lines in
the picture, and all four Mj respond:  M0, M1, M2 and M3 send words
24, 25, 26 and 27, respectively, which will be saved in latches at
the entrance to the MUX (not shown).  The CPU then puts the lower
two bits of the address, 10, on the lower address lines in the picture,
which are attached to the selector inputs of the MUX, with the result
that the MUX selects whatever M2 sent it (since 10 binary is 2 
decimal); this is word 26, just what the CPU wanted.

The point is that the CPU can now pick up words 24, 25 and 27 
``for free,'' i.e. with almost no extra time spent, if it needs 
them.\footnote{The word ``almost'' refers to the fact that although
there is some amount of gate delay in the MUX, that delay should be 
much smaller than the delay in an Mj.}  For instance, if N = 2 above 
and X starts at word 26, the CPU will now put 11 on the lower address 
lines which lead to the MUX, and thus pick up word 27.  The result is 
that the CPU has read two words in just a little more time than it
takes to read one word.

In fact, if it so happens that the application program running on
the CPU needed to read a four-element array beginning at word 24,
we would be able to read four words in the time normally needed to
read just one.  In other words, the setup here can increase memory
access speed by a factor of nearly four.

Of course, the speedup would occur because we happen to need to
read four consecutive words of memory.  But even that condition
is not sufficient for achieving a four-fold speedup.  For example,
suppose we need to read the four words 26, 27, 28 and 29.  We
would be able to quickly get the first two of these, words 26 and
27.  But after getting them we would have to issue a separate 
``major memory request'' for words 28 and 29, by putting a new
value, 00000000000111, on the upper address lines in the picture
(and then issuing the values 00 and 01 to the MUX).  So, here we
would have a speedup of only about two, not four.

The applications which run on vector machines, tend to be heavily
mathematical, such as matrix operations.  In this setting, to write
efficient software, one must pay attention to the internal storage
of arrays.  One-dimensional arrays are stored contiguously, but
higher-dimensional arrays are more complicated.

In C/C++, two-dimensional arrays are stored in {\it row-major order}.
What this means is that the first part of the storage consists of all
of row 0, right after which is stored all of row 1, and so on.

For example, suppose we have a matrix declared as 

\begin{verbatim}
int z[5][5];
\end{verbatim}

and again for simplicity, assume that individual bytes are not
addressable on our machine.  Say that z begins at word 520.  Then
z[0][0] is in word 520, z[0][1] in 521, ..., z[0][4] in 524,
z[1][0] is in word 525, z[1][1] in 526, and so on.

Thus, while a matrix row will be stored in contiguous locations,
a matrix column will not.  In the case of z here, for instance,
consecutive elements within a matrix column will be spaced exactly
five words apart.  Thus we will be able to get the benefit of
memory ``pipelining'' for rows but not columns.  Accordingly, our
program might run much faster if we work with matrix transposes
in some cases.

\section{Scheduling of Feedback Pipes}

As noted earlier, in some pipes a task may need to visit some pipe
stages more than once, forming a feedback structure.  However, in
most such cases, not all stages are in use during all time slots
while a task is being processed.\footnote{I am using the term
{\it time slot} here to mean one stage time.}  This suggests the 
possibility of having more than one task in the pipe at the same time.  

Note that in a nonfeedback pipe, we always have more than one task in 
the pipe at one time (assuming we have a ready supply of tasks 
waiting to be processed), but this may not be possible in a feedback 
pipe, since separate tasks may need to use the same stage during the
same time slot.

Consider an example in {\it Computer Architecture and Organization},
by John Hayes (McGraw-Hill, 1988).  We start with the same four-stage 
pipe for floating-point addition in our examples above, but with a 
different type of task.  In the VADD example, the task was to add
a pair of numbers, and the pipe was nonfeedback.  In our example
here, though, our task will be to find the sum of eight numbers,
$b_1+b_2+...+b_8$.   The question then becomes, Can we have
more than one eight-number set in the pipe at the same time, and if
so, at what times should the different sets be started?

Let us first look in more detail at how \underline{one} task,
i.e. the summing of eight numbers, is accomplished:  Roughly
speaking, first the sums $b_1+b_5$, $b_2+b_6$, $b_3+b_7$ and 
$b_4+b_8$ will be computed by running the corresponding pairs
of numbers through the pipeline.  Then the sums $b_1+b_2+b_5+b_6$
and $b_3+b_4+b_7+b_8$ will be computed, and so.

Here is how it will work:  Say there are ``left'' and ``right'' inputs
at the entrance to the pipe, with MUXs at each.  The MUXs will select
either a new input or the feedback from the exit of the pipe.  In time
slot 1, $b_1$ and 0.0 enter the pipe, in preparation to compute
$b_1+0.0$.  It is assumed here that the $b_i$ are arriving one at a time
from memory.  Thus $b_5$ is not available yet, so we cannot start
computing $b_1+b_5$ yet, and we have to settle for computing the ``sum''
$b_1+0.0$.  Say $b_1$ enters the left input of the pipe, and the
constant 0.0 enters at the right. 

In time slot 2, $b_2$ enters the left input from memory, and 0.0
enters the right input.  In time slot 3, $b_3$ and 0.0 enter, and
so on.

At the end of time slot 4, $b_1+0.0$ finishes at S4, and is recycled
back to S1, say to the left MUX.  It is joined by $b_5$ entering the
right input from memory, and the computation of $b_1+b_5$ begins in time
slot 5.  That sum exits S4 at the end of time slot 8, and is held in 
a register for one more time slot, after which $b_2+b_6$ exits that 
stage at the end of time slot 9.  Both of these sums are then 
recycled back to S1, so that the computation of $b_1+b_2+b_5+b_6$ can 
be begun in time slot 10, and so on.
  
The entire process turns out to require 19 time slots, i.e. 
$b_1+b_2+b_3+b_4+b_5+b_6+b_7+b_8$ is finally ready at the end of
time slot 19.  That is already a big improvement, because without
pipelining the sum of eight numbers would take (4+2+1)x4 = 28
time slots, much longer than 19.\footnote{Each sum operation
takes 4 time slots to perform.  If you look at which sums are
performed to get the final grand total of eight numbers above, you
will see that there are 4+2+1 such sums to be done.}  But we want
to be even greedier, and have more than one set of eight numbers
being in the pipe at the same time.  In other words, we want
to start the computation of some sum $c_1+c_2+...+c_8$ while
the computation of $b_1+b_2+...+b_8$ is still in progress.

Then the following {\it reservation table} shows which stages are busy
at which time slots for the computation of $b_1+b_2+...+b_8$:

\newpage

\begin{verbatim}
     1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
S1   x x x x x x x x    x     x           x         
S2     x x x x x x x x     x     x           x      
S3       x x x x x x x  x     x     x           x   
S4         x x x x x x  x  x     x     x           x
\end{verbatim}

Now consider what happens if we try to have more than one eight-number
set in the pipe at a time.  Suppose that at time slot, say, 527, the
pipe is empty and we start an eight-number set's computation.  Let us
call this set Set I.  Note that all times will be relative.  For that 
set, ``time slot 1'' in the reservation table for this first set is 
absolute time 527, time slot 2 is absolute time 528, and so on.  This 
set will finish at the end of absolute time slot 545 (527+19-1).

Now, when can we start the computation of a second set of eight numbers,
which we will call Set II?  For instance, could we do so, say, in 
absolute time slot 530?  The answer is no, for the following reason:

If we were to start Set II at 530, then its ``time slot 1'' would be
at time 530, at which time it would use S1, as can be seen from the
reservation table.  But absolute time slot 530 is relative time slot 
4 for Set I, and from column 4 in the reservation table we can see
that Set I is using S1 at that time.  In other words, if we were to
start Set II at 530, it would immediately {\it collide} with Set I.

The reservation table does show that S1 is idle at relative time
slot 9.  So, how about starting Set II at absolute time 527+9-1=535?
Set I will not be using S1 at that time (again, 535 would be relative 
time slot 9 for Set I), so it seems that this might be a feasible 
time to start Set II.  But it turns out that collisions would occur 
later on, for instance at time 536:  That time is relative time slot 
10 for Set I, and would be relative time slot 2 for Set II.  As can
be seen from the reservation table, S1 is in use both in time slots
2 and 10.

The same reasoning shows that if two x's in some row of the
reservation table are a distance k apart, we cannot start Set II
at time 527+k.  In the S1 row of our reservation table here, there
are x's in time slots 2 and 10, corresponding to k=8, so we cannot
start Set II at time 527+8 = 535, as we discovered in the last 
paragraph.

We define the {\it forbidden list} to be the set of all such values
of k, i.e. all the values of k such that in some row of the
reservation table there are two x's a distance k apart.  In this
case, the forbidden list will be

\begin{verbatim}
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
\end{verbatim}

We can start Set II at any time 527+j for which j is not in the
forbidden list.  Clearly, that would be true for any j larger than
19, because Set I will have left the pipe after 19 time slots, but
we can do better than that, such as j=16.  In other words, we could
start Set II at time 527+16=543, without causing a collision.

However, what about Set III?  In determining times at which we can
start Set III, we now have to worry about possible collisions with
both Set I and Set II, so the forbidden list is not enough.  We
need a more sophisticated analytical method.

To explain this method, let us look at a simpler example, a four-stage
pipe.  Say it makes widgets.\footnote{By the way, though most of
the example pipes we have seen here have four stages, the reader should
of course not conclude that most pipes have that number.} with 
reservation table as follows:
 
\begin{verbatim}
    1 2 3 4 5 6 7
S1  x           x
S2    x   x     
S3      x     x 
S4          x   
\end{verbatim} 

This yields a forbidden list of 

\begin{verbatim}
2,3,6
\end{verbatim}

Now suppose that the pipe is empty before time, say, 34, and that at
that time we start Widget I.  Based on the numbers in the forbidden list,
we can't start Widget II at 2, 3 or 6 time slots later (i.e. at
absolute times 36, 37 or 40).  But we can start Widget II at any other
time, i.e. 1, 4 or 5 slots after Widget I (or 7 or more time slots,
since Widget I will be out of the system by then).

We can summarize the findings of the previous paragraph by saying that
the {\it collision vector} for deciding when to start Widget II is
$(c_1,c_2,c_3,c_4,c_5,c_6)$ = (0,1,1,0,0,1), where $c_i = 1$ means
that we cannot start Widget II in the time slot which is i slots
after the one in which Widget I was started.  In other words, the
collision vector (0,1,1,0,0,1) is just a compact way of saying that
we can't start Widget II 2, 3 or 6 slots after Widget I, but all
other slots are all right.

Assume we choose to start Widget II 5 slots after Widget I, i.e.
in absolute time slot 39.

Now then, what about Widget III?  For the same reasons, in order to 
avoid colliding with Widget II, we cannot start Widget III 2, 3 or 6
slots past the slot in which Widget II started (i.e. at absolute
time slots 41, 42 or 45).  So, this again gives us a collision vector
of (0,1,1,0,0,1) for Widget III, at least in terms of avoiding
collisions with Widget II.

But we also have to make sure Widget III doesn't collide with Widget I.
For instance, Widget II had to avoid starting 6 slots after Widget I,
because otherwise they would collide at S1.  So now, 5 time slots 
later, Widget III has to refrain from starting 6-5=1 slots from now.
In other words, the 1 which was in the $c_6$ position of Widget II's
collision vector now becomes a 1 in the $c_1$ position of Widget III's
collision vector.

To reflect this, we take Widget II's collision vector and shift it to
the left 5 positions.  That is, we shift (0,1,1,0,0,1) left five 
positions, yielding (1,0,0,0,0,0).  What that 1 means in the first 
component position is that we cannot start Widget III in the first 
time slot after Widget II, because to do so would collide with Widget I.  
Starting in any other time slot, though, will not produce a collision
with Widget I.

The point is that whatever restrictions Widget I caused for Widget II
now must be updated to list the restrictions Widget I is causing for
Widget III.  Since 5 time units have passed since Widget II listed
its restrictions, we must shift that list left by five positions.

To illustrate the principle another way, suppose Mary wants to avoid
bumping into Jim at the student union.  Say 2 hours ago, Mary sent John
e-mail saying, "John will be there 6 hours from now, so I won't go there
then.''  Suppose John wants to avoid Jim too.  That means that John
should avoid going to the student union 6-2 = 4 hours from now.

To summarize what we have found, Widget III's collision vector for
avoiding collisions with Widget II is (0,1,1,0,0,1), and Widget III's 
collision vector for avoiding collisions with Widget I is (1,0,0,0,0,0). 
So, the full collision vector for Widget III is obtained by taking
the ``bitwise'' (i.e. componentwise) logical-OR of these two vectors,
which is (1,1,1,0,0,1).  We see, then, that Widget III cannot be
started 1, 2, 3 or 6 time slots after Widget II (i.e. in absolute
slots 40, 41, 42 or 45).  Any other time is all right.

Here, then, is the general procedure:  We first find the {\it initial
collision vector}, CV0, from the forbidden list, as we did for Widget II.
This will consist of an m-component vector, where m is the largest
value in the forbidden list.  CV0 will be the collision vector for 
the second task in the system, i.e. it will say how long after the 
first task we can initiate the second one.  For each subsequent task, 
the collision vector is obtained by OR-ing the \underline{shifted} 
collision vector for the previous task with CV0, where the amount of 
shifting corresponds to our choice of waiting time for that previous task.

For example, we obtained Widget III's collision vector by shifting
Widget II's collision vector by 5 positions, since we had decided to
start Widget II 5 slots after Widget I, and then OR-ing that shifted
vector with CV0.

This gives rise to the {\it state diagram} for this pipe:

\vspace{0.2in}
\par
\psfig{file=States.eps}
\par

The numbers which label the arcs denote the waiting time before
starting the task.  For instance, look at the collision vector 
(1,1,1,0,0,1).  Since there are 0's at positions 4 and 5, we can
start our next task either 4 or 5 slots after the last task.  If
we choose to wait 4 slots, the collision vector for the following
task will be (0,1,1,0,0,1).  (Make sure to compute this yourself
to check.)  If on the other hand we wait 5 slots, our new 
collision vector will be (1,1,1,0,0,1) again.

Now, what is the optimal schedule?  What we want is to minimize
the \underline{average} time between successive task initiations;
this is equivalent to maximizing the throughput rate of widgets
in this pipe.\footnote{This is actually a {\it long-run} average,
since we may visit a few other states when the pipe first starts
up, after which we settle down to a set pattern of state visits.}
So, what schedule will have minimal average latency
(MAL)?  Inspection of the state diagram reveals the MAL schedule
to be the one that cycles repeatedly as follows:  (0,1,1,0,0,1), 
(1,1,1,0,1,1), (1,1,1,0,0,1), (0,1,1,0,0,1), (1,1,1,0,1,1), ...
In other words, we schedule Widget II 1 slot after Widget I,
Widget III 4 slots after Widget II, Widget IV 4 slots after
Widget III, Widget V 1 slot after Widget IV, etc.  In terms of
absolute times, Widgets I, II, III, IV, V, ... will start at
slots 39, 40, 44, 48, 49, ...  The MAL will be (1+4+4)/3 = 3,
i.e. on the average we will start a task every 3 slots; this
corresponds to a throughput of 1/3 task per slot.

By the way, at any state we can always wait 7 or more slots, since
all previous tasks will have left the system by then (one task in
this pipe takes 7 time units to complete), but for clarity we have
not drawn arcs for them.   If we did, all such arcs would lead to
CV0, (0,1,1,0,0,1).

The throughput rate for a pipe can sometimes be increased by
adding {\bf delay stages}, that is, extra stages that do nothing.
This seems strange at first, but these stages can help avoid
collisions, and thus improve throughput.  We will not pursue
this further here, though.

\section{Pipelining and RISC Architecture}

\subsection{History}

The term {\bf RISC} is an acronym for \underline{r}educed 
\underline{i}nstruction \underline{s}et \underline{c}omputer, the
antonym being {\bf CISC}, for \underline{c}omplex 
\underline{i}nstruction \underline{s}et \underline{c}omputer. 

During the 1970s, computers became more and more CISC-like, with
richer and richer instruction sets.  The most cited example is that 
of the VAX family, whose architecture includes 304 instructions, 
including such highly specialized instructions as POLY, which 
evaluates polynomials.  By contrast, the UC Berkeley RISC I 
architecture, which later became the ``rough draft'' for what is 
now the commercial SPARC chip, which is used in the ACS SPARCstations, 
had only 32 instructions.

At the same time, the single-chip CPU was starting out on its path
to dominance of the market.  The more components a computer has, the
more expensive it is to manufacture, so single-chip CPUs became very
attractive.\footnote{And there is a performance gain as well.  Multi-chip
CPUs suffer from the problem that off-chip communication is slower than
within-chip.}  But the problem then became that the very limited space 
on a chip---this space is colloquially called ``real estate''---made
it very difficult, if not impossible, to fit a CISC architecture onto a
single chip.\footnote{It would seem that the solution is to simply make
larger chips.  However, this idea has not yet worked.  For example,
larger chips have lower {\bf yield} rates, i.e. lower rates of flaw-free
chips.  Of course, one can try to make chips denser as well, but there
are limits here too.}

The VAX was a clear casualty of this.  When the Digital Equipment 
Corporation (DEC) tried to produce a ``VAX-on-a-chip,'' they found that
they could not fit the entire instruction set on one chip.  They were
then forced to implement the remaining instructions in software.  For 
example, when the CPU encountered a POLY instruction, that would 
cause an ``illegal op code'' condition, which triggered a {\bf trap} 
(i.e. internal interrupt); the trap-service routine would then be a 
procedure consisting of ordinary ADD and MUL instructions which
compute the polynomial.

At this point, some researchers at IBM\footnote{Thoug Professor Patterson
at UC Berkeley is the one who really popularized the RISC idea, it 
was actually invented at IBM first.  Actually, the old CDC machines 
designed by Seymour Cray had many RISC characteristics, i.e. small 
instruction sets and load/store architecture.} decided to take a 
fresh look at the whole question of instruction set formulation.  The 
main point is optimal use of real estate.  In a CISC CPU chip, a 
large portion of the chip is devoted to the microcode---and yet much 
of that microcode is rarely, if ever used.  For example, most users 
of VAX machines do not make use of the POLY instruction, and yet
the microcode for that instruction would occupy valuable-but-wasted 
real estate in the chips comprising a VAX CPU.  Maybe, for example, an
on-chip cache would make better use of that space; at least the cache would
adapt itself to the characteristics of whatever program is currently running,
quite unlike the the case of the POLY microcode.  

True, for those who do use the POLY instruction, execution speed will 
increase somewhat,\footnote{You should think this.  First, POLY, being a
single instruction, would require only one instruction fetch, compared to
many fetches which would be needed if one wrote a function consisting of
many instructions which would collectively perform the work that POLY does. 
Second, on the microcode level, we might be able to perform more than one
register transfer at once.}  but for most programs the POLY instruction is
just getting in the way of fast performance.

Other factors came into play.  For example, the richer the instruction
set, the longer the decode portion of the instruction cycle would take
(whether for a microprogrammed \underline{or} a hardwired machine).  Even
more important is the issue of pipelining.  Unless we are willing to make
pipeline stage times to be of variable length, we then have to make each
stage have time equal to that of the worst possible case (e.g. the longest
possible decode time in the example above), or add stages to the pipe.
The former of course slows things down a lot whenever we don't have the 
worst-case situations, and the latter has real problems when branches 
occur. 

For this reason, the IBM people asked, ``Why do we need the microcode at
all?  Why not just make the microengine itself our machine?''  They 
didn't literally mean a microengine, with different bit fields 
directly controlling various CPU components, but instead meant a CPU
with a very simple, limited instruction set, eliminating the need for
all that real-estate-consuming microcode, and one in which most
instructions would---like microengine instructions---execute in a 
single clock cycle, so that a smooth pipeline could be built.

Unfortunately, IBM scrapped their RISC project, but later David Patterson
of the Division of Computer Science at UC Berkeley became interested 
in the idea, and developed two RISC CPU chips, RISC I and RISC II.  
These two chips will be our focus here.  

Patterson's team did not invent the RISC concept, nor did they invent
some of the features of their design such as delayed branch, register
windows, and so on.  Nevertheless, these people singlehandedly changed
the thinking of the entire computer industry, by showing that RISC
could really work well.  The team considered the problem as a whole,
investigating all aspects, ranging from the questions of chip layout
and VLSI electronics technology to issues of compiler writing, and
through this integrated approach were able to make a good case for
the RISC concept.  Manolis Katevenis, a Ph.D. student who wrote his
dissertation on RISC under Professor Patterson, worked on everything
from the software analysis to the actual internal design of the CPU
chip, and won the annual ACM for the best dissertation in Computer
Science.\footnote{Published as
{\it Reduced Instruction Set Computer Architectures for VLSI}, M. 
Katevenis, MIT Press, 1984.}

Today virtually every computer manufacturer has placed a major emphasis
on RISC.  Sun Microsystems has the SPARC chip, DEC uses MIPS and also
their own new Alpha chip, IBM and Motorola have the Power PC chip
series, and so on.  (On the other hand, this does not imply that
RISC is inherently ``superior,'' and as electronic technologies
advance in the future, there may well be some resurgence of CISC
ideas, at least to some degree.)

The Patterson effort is often cited today as an example of the 
contributions which can be made by university Computer Science 
Departments to the computer industry.

\subsection{The Role of the RISC Concept in Achieving a Smooth
Pipeline}

It is of absolutely central importance that you keep in mind when
reading the material that follows that the ``RISC-ness'' of the RISC
I/II instruction set is of tremendous help in achieving a smooth
pipeline.  

First, all RISC I/II instructions are four bytes in length.  This
compares, for instance, to the CISC architecture in the Intel
80x86 family, in which instruction length can range from one to
six bytes.  An 80x86 pipeline would have to either allow for the
worst case in the instruction-fetch stage, thus possibly making
the common stage time for all states longer than necessary, or
would have to undergo a vast increase in pipeline complexity,
having to allow for many different special cases.  

Similarly, the Intel family also has variable-length op codes, so
the decoding portion of the pipe\footnote{Which in the case of
RISC I/II is so simple that we have incorporated it into the
instruction-fetch stages in the pictures above.} becomes more complex 
too, with the same kinds of negative implications for the pipe.

Second, except for load/store instructions, all RISC I/II instructions
execute in a single clock cycle.  By contrast, Intel instruction times
have a wide range.  For instance, an ADD instruction could execute in
as few as 2 clock cycles, while a DIV could take as many as 25 cycles.
Again, there are profound implications of this on the complexity of
the pipes in the Intel chips.

\subsection{A Definition of RISC}

There is no universally-agreed-upon definition of RISC, but most people
would agree at least to the following characteristics:

\begin{itemize}
\item Most instructions execute in a single clock cycle.

\item ``Load/store architecture''---the only instructions which access
memory are of the LOAD and STORE type; e.g. memory-to-register adds like

\begin{verbatim}
ADD R2,X
\end{verbatim}

are not possible.

\item Every instruction has the same length, e.g. 4 bytes.  By contrast,
Intel instructions range in length from 1 to 6 bytes.

\item ``Orthogonal architecture''---any operation can use any operand.
This is in contrast, for example, to the Intel 80x86-family architecture,
in which for example, MUL and DIV require that one of their operands be
in a specific register, AX.

\item The instruction set is limited to only those instructions which
are truly justified in terms of performance/space/pipeline tradeoffs.
The SPARC chip, for instance, does not even have a MUL instruction.
And the ``VAX on a chip'' ended up doing this kind of thing anyway, albeit 
not intentionally.

\item Hardwired implementation, i.e. not microcoded.
\end{itemize}

All but the last of the traits listed above deal with pipelining.

Concerning hardwired implementation, this actually follows from the others.
In the old days, the rationale for microcoded implementation of a CPU was
that it made the CPU development faster, and the CPU debugging faster.
It simplified a very complex job.  But if the CPU is very simple anyway,
that rationale disappears.

\subsection{Effects on Compiler Writers}

Computer architects used to partly justify including instructions like POLY 
in the instruction set by saying that this made the job of compiler writers
easier.  However, it was discovered that this was not the case, for several
reasons.

\begin{itemize}
\item Compiler writers found it difficult to write compilers which
would automatically recognize situations in (say) C source code in which
instructions like POLY could be used.

\item Compiler writers found that some specialized instructions that were
actually motivated by high-level language constructs did not match the
latter well.  For example, the VAX had a CASE instruction, obviously
motivated by the Pascal {\bf case} and C {\bf switch} constructs; yet
it turned out to be too restrictive to use well for compiling those
constructs.

\item CISC generally led to very nonorthogonal architectures, which made
the job of compiler writers quite difficult.  For instance, the requirement
that the AX register be used in MUL and DIV, \underline{plus} the
requirement that it be used in other instructions, implies that compiler
writers ``don't have any spare registers to work with,'' and thus they
must have the compiler generate code to save and restore ``popular''
registers such as AX; this is a headache for the compiler writer.
\end{itemize}

Most RISC machines do ask that the compiler writers do additional work
of other kinds, though, in that they have to try to fill {\bf delay
slots}; more on this below.

\subsection{Dealing With the Branch Problem}

As you know, a major problem with pipelined CPUs is that the pipeline
must be drained when a branch is taken.  If for example there are n
stages to the pipe, and the instruction is a conditional branch (so
that we won't know until the last stage whether the branch will be
taken), we will have n-1 empty clock cycles in which no instruction is
executed.

\subsubsection{Branch Delay Slots}

Many RISC processors use something called a {\bf delayed branch} to
deal with the branch problem.  The term means that the execution of 
the {\bf branch target}, i.e. the place to which we jump on the branch, 
is delayed by one cycle.  This is accomplished in an odd but 
actually quite clever way:  Regardless of whether a branch is taken 
or not, we still execute the instruction following the branch.

For example, suppose we have a 2-stage pipe, with Stage 0 being the
instruction fetch and Stage 1 being decode and execute; while one
instruction is executing, the next one is being fetched.  Note that
this implicitly means that we are assuming that it takes one pipe-stage
time to read memory.\footnote{Or possibly to read from a cache.}
Now consider the following generic (not RISC I/II) code:

\begin{verbatim}
ADD R5,R3
ADD R2,R4
JNEG T
n.s.i.
\end{verbatim}

Here n.s.i. stands for ``next sequential instruction,'' i.e. the
instruction following JNEG in memory, which will be executed only if the
branch is not taken.  Using the normal scheme, if the condition in JNEG
is satisfied and we jump to T, we will have one empty clock cycle in
which no instruction is executed, because we will have to fetch the
instruction at T (which had not been prefetched).

Under the delayed-branch scheme, the n.s.i. will be executed 
regardless of whether we jump to T!  By the time we start execution 
of the n.s.i., we will know whether the condition in the JNEG was 
satisfied, and so during this time we either fetch from T (if it was 
satisfied) or from the location following the n.s.i. (if it was not 
satisfied).  In either case, we will have no empty clock cycles.

A potential problem arises in that we may not want to have the n.s.i.
executed.  Thus the compiler (or assembly-language programmer) would 
first tentatively produce code like this

\begin{verbatim}
ADD R5,R3
ADD R2,R4
JNEG T
NOP
n.s.i.
\end{verbatim}

so that the delayed-branch mechanism will not harm correct execution
of the program.  (Recall that most machines have an NOP instruction, 
pronounced ``no-opp,'' which does literally that, i.e. no operation at all.)

This code would work properly, but it would not exploit the parallelism
idea of delayed branches.  Instead, what we would like to do is 
change the order of the instructions:

\begin{verbatim}
ADD R2,R4
JNEG T
ADD R5,R3
n.s.i.
\end{verbatim}

Here the ADD to R5 has been moved to follow the JNEG, and this ADD will
be done right after the JNEG.  The change in order won't affect the correct
execution of the code,\footnote{If we had moved the other ADD, there would
have been an effect.}, but the key point is that in the case the 
branch is taken, we save one cycle!  The time that would have been 
wasted fetching the instruction at T is now usefully employed by 
execution of the ADD at R5 at the same time.  And of course if the 
branch is \underline{not} taken, no harm is done.

The instruction position following a branch is called a {\bf delay
slot}.  Machines such as MIPS and SPARC have just one delay slot,
because their design fulfills an instruction-fetch request in just one
clock cycle.  (This assumes a cache hit; in the case of a miss, the pipe
needs to be suspended for quite some time.)  But some other machines
have more than one delay slot.  Clearly, the more delay slots there are,
the harder it is to ``fill'' them, i.e. to find instructions to move to
put in the slots, and thus avoid simply putting wasteful NOPs in them.

How hard is it to fill even one delay slot?  The Katevenis dissertation
on RISC says:

\begin{quote}
Measurements have shown [citation given] that the [compiler] optimizer is
able to remove about 90\% of the no-ops following unconditional transfers
and 40\% to 60\% of those following conditional branches.  The unconditional
and conditional transfer-instructions each represent approximately 10\%
of all executed instructions (20\% total).  Thus, while a conventional
pipeline would lose $\approx$ 20\% of the cycles, optimized RISC code
only loses about 6\% of them.
\end{quote}

\subsubsection{Branch Prediction}

Most modern CPUs use {\bf branch prediction} as a way to avoid pipeline
delays due to branches (either in addition to delayed branch or instead
of it).  Here is an overview of how it might work.

In the decoding stage of the pipeline, suppose the CPU discovers an
instruction like

\begin{verbatim}
JNEG T
\end{verbatim}

and suppose that T is a location with a lower memory address than
that of the instruction, i.e. this is a backward branch.  Usually
this will mean that the instruction is the last one in a loop. 
Since it has been found empirically that most loops iterate more
than once, that means that backward branches are usually taken.

So, the CPU circuitry can be built so that the branch target, say
T above, is always prefetched.  The reasoning above says that we
will usually be correct, and thus the branch will usually cause us
no delays.  Of course, sometimes this guess will be incorrect, in
which case we do incur a delay.

Another approach to branch prediction is to maintain a {\bf branch history
table}.  This is a small cache inside the CPU that records the last
action (taken/not taken) for each of several recently-executed branch
instructions.\footnote{More complicated histories could be maintained
instead.}  We would predict a given branch to be taken or not, depending
on what the last action of that instruction was.  The table entry for a
given instruction might consist of two fields, first the instruction's
address and second the history bit.  In a more ambitious version, the
second field might actually be the target instruction itself (i.e.  the
instruction at T in the above example), so that the prefetching would
actually be done on the table entry.

Note that we might incur more of a penalty for incorrectly guessing a
branch to be taken than for wrongly guessing it to not be taken.  This
is because the target T might be far enough away from the current
location that prefetching the target causes a cache miss, or even
worse a virtual memory page fault.

\subsection{Register Windows}

Though register windows are not considered to be part of the definition
of RISC, the RISC I/II machines did use them, and a number of the earlier
commercial RISC processors---notably SPARC---followed suit.

The motivation underlying the register window concept is to reduce traffic
from the CPU to memory, since such access is much slower than CPU speeds.
Let us first recall the notion of a stack frame.

\subsubsection{Stack Frames}

In most modern machine/compiler combinations, function calls are 
handled via the stack.  Not only does a machine CALL instruction save
the return program counter value on the stack, but also the compiler
produces code to push function parameters on the stack, and also to
allocate stack space for local variables in the called function.

For example, suppose we have a C function whose first few lines are

\begin{verbatim}
x(a,b)
   int a,b;

{  int c,d,e;
\end{verbatim}

and that there is a call to the function like this:

\begin{verbatim}
x(m,n);
\end{verbatim}

The compiler will translate the call to something like

\begin{verbatim}
PUSH N
PUSH M
CALL X
\end{verbatim}

and will translate the beginning of the function itself to something 
like

\begin{verbatim}
SUB SP,3
\end{verbatim}

(assuming successive words differ in address by 1, and assuming that the
stack grows toward 0), in order to allocate three words on the stack 
for x's local variables c, d and e.

In other words, for a given function call, the portion of the stack will
be devoted to that call will look like this:

\begin{verbatim}
local variables
return address
parameters
\end{verbatim}

This is called the {\bf stack frame} for that call.

\subsubsection{Stack Frames on RISC I and II}

RISC I and II store the top few frames of the stack in registers within
the CPU.  When the stack grows to the point at which all these registers
are occupied, a stack overflow occurs, and some of the lower part of the
stack is copied to memory, to make room for the newly-pushed part of the
stack.  Later, when the stack is popped enough to reach the part of 
the stack which was shunted out to memory, a stack underflow occurs, and
that data will be copied back to these registers in the CPU.

Remember, due to top-down software design, typically we will have 
function f1 call function f2, f2 then in turn calling f3, and so on.
This leads to the notion of {\bf parent} and {\bf child} in the
calling chain.  For example, in the chain above, f2 would have f1 as
its parent and f3 as its child.

This structure is then reflected in the register windows.  What is 
normally called a stack frame for other machines is called a {\bf 
register window} here.  Each window consists of 32 registers,
called r31 through r0.  Of these, r15-r10 are used by the function
to receive parameters from the parent, r25-r16 are used to store 
the function's local variables, r31-r26 are used to send parameters
to the child, and r9-r0 are used to store global variables (but r0 
has the value 0 hardwired into it).

The RISC II CPU chip contains a total of 138 registers, which I will 
denote here by R137-R0.  These registers then form 8 windows, 
numbered 0 through 7.  Each window register ri ($0 <= i <= 31$) maps 
to some Rj, in the following way.

\begin{itemize}
\item [1.]
R9-R0 are held in common by all windows, so that a reference to ri,
$0 <= i <= 9$, will map to Ri.  Again, except for r0/R0, which stores
the constant 0, these registers store global variables, accessible to
all functions.

\item [2.]
Registers r31-r26 of a window will be physically identical to 
r15-r10 of the child.  It is this overlap which allows parameter 
passing between functions, as well as passing the return program 
counter value if this function in turns calls a child.

\item [3.]
Registers r25-r16 are used to store local variables. 
\end{itemize}

Below are the correspondences between the ``virtual'' register
numbers ri and the ``physical'' register numbers Rj.  Note that
1r11, for instance, means r11 in window 1.

\begin{verbatim}
...                                                 ...       ...
R122                                                6r26      7r10
                                          ...
...                           ...
R47                 1r31      2r15
...                 ...       ...
R43                 1r27      2r11
R42                 1r26      2r10
R41                 1r25
...                 ...
R32                 1r16
R31       0r31      1r15
...       ...       ...
R27       0r27      1r11
R26       0r26      1r10
R25       0r25
...       ...
R16       0r16
R15       0r15
...       ...
R11       0r11
R10       0r10     
R9        0r9       1r9       2r9      ...          6r9       7r9
...       ...       ...       ...                   ...       ...
R1        0r1       1r1       2r1      ...          6r1       7r1
R0        0r0       1r0       2r0      ..1          6r0       7r0
\end{verbatim}

The second column, labled `0', corresponds to main().  If main()
then, say, calls x() which in turns calls y(), then the column labeled
`1' would be for x(), column `2' would be for y() and so on.

A special Current Window Pointer register points to r16 of the 
current window.

As an example, consider the C code presented above.  Assuming that
m and n were global variables, the compiler might have them stored
in r1 and r2, and might translate the call to the following RISC II 
instructions:\footnote{This is somewhat simplified from the
real RISC II.}

\begin{verbatim}
add r0,r1,r26
add r0,r2,r27
call x,r28
\end{verbatim}

Since r0 contains the constant 0, the add instructions here
simply copy r1 and r2 to r26 and r26, respectively.\footnote{By this
device, there is no need to have a ``move'' instruction in the RISC
I/II architecture, and there is none.}  Note that the return address
is saved in the window too, in this case in r28.

The code produced by the compiler for the function x itself would
access the formal parameters m and n via registers r10 and r11;
remember, these registers are physically the same as r26 and r27
of the parent.  The local variables c, d and e would go in r16-r18.
Finally, the last instruction in x would be

\begin{verbatim}
ret r12
\end{verbatim}

(Again, you should verify all of these statements.)

If a function has more than six parameters, there is not enough
room in a window to pass them.  Thus even though the idea of register
windows is to replace the concept of a stack, the excess parameters
in this situation must be passed through a stack after all, and
compilers for machines having register windows do set up stacks.

The stacks are needed for another reason as well.  Consider the
following code:

\begin{verbatim}

w(s)
   int *s;

{...}

main()

{  int z;

   ...
   w(&z);
   ...
}
\end{verbatim}

Since z is local to main(), we would normally store it in a window,
somewhere in r25-r16.  But here that would be impossible, since the
expression

\begin{verbatim}
&z
\end{verbatim}

would be meaningless.  So here too z would need to be stored in a
stack.
 

\subsubsection{Performance}

The Katevenis thesis reports the following findings, in a comparison
of words of memory access for data in two sample programs:

\begin{verbatim}
machine          puzzle          quicksort
-------          ------          ---------
VAX-11/780         440K               700K
RISC                 8K                 4K
\end{verbatim}

So, the use of register windows does appear to achieve a dramatic
reduction in memory traffic, extremely valuable in light of the
slowness of memory.  Anti-register-window advocates argue that a
cache might do as well or better (the VAX did have a cache, but
not as big a cache as it would have today).  Katevenis, Patterson
{\it et al} disagree, but the question would be hard to resolve
completely to everyone's satisfaction.

\subsection{SPARC Example:  Delayed Branch}

\begin{verbatim}
     1	Script started on Sun Mar 10 16:19:06 1996
     2	taco 1: m a.c
     3	
     4	int a,b;
     5	
     6	main(argc,argv)
     7	   int argc; char **argv;
     8	
     9	{  int x,y;
    10	
    11	
    12	   a = atoi(argv[1]);
    13	   b = atoi(argv[2]);
    14	   if (a < b) x = a;
    15	   else y = a+b;
    16	}
    17	
    18	taco 2: 
    19	taco 2: 
    20	taco 2: 
    21	taco 2: cc -S a.c
    22	taco 3: m a.s
    23	LL0:
    24	        .seg    "data"
    25	        .common _a,0x4,"data"
    26	        .common _b,0x4,"data"
    27	        .seg    "text"
    28	        .proc 04
    29	        .global _main
    30	_main:
    31	        !#PROLOGUE# 0
    32	        sethi   %hi(LF14),%g1
    33	        add     %g1,%lo(LF14),%g1
    34	        save    %sp,%g1,%sp
    35	        !#PROLOGUE# 1
    36	        st      %i0,[%fp+0x44]
    37	        st      %i1,[%fp+0x48]
    38	        ld      [%fp+0x48],%o0
    39	        ld      [%o0+0x4],%o0
    40	        call    _atoi,1
    41	        nop
    42	        sethi   %hi(_a),%o1
    43	        st      %o0,[%o1+%lo(_a)]
    44	        ld      [%fp+0x48],%o2
    45	        ld      [%o2+0x8],%o0
    46	        call    _atoi,1
    47	        nop
    48	        sethi   %hi(_b),%o3
    49	        st      %o0,[%o3+%lo(_b)]
    50	        sethi   %hi(_a),%o4
    51	        ld      [%o4+%lo(_a)],%o4
    52	        sethi   %hi(_b),%o5
    53	        ld      [%o5+%lo(_b)],%o5
    54	        cmp     %o4,%o5
    55	        bge     L17
    56	        nop
    57	        sethi   %hi(_a),%o7
    58	        ld      [%o7+%lo(_a)],%o7
    59	        st      %o7,[%fp+-0x4]
    60	        b       L18
    61	        nop
    62	L17:
    63	        sethi   %hi(_a),%l0
    64	        ld      [%l0+%lo(_a)],%l0
    65	        sethi   %hi(_b),%l1
    66	        ld      [%l1+%lo(_b)],%l1
    67	        add     %l0,%l1,%l2
    68	        st      %l2,[%fp+-0x8]
    69	L18:
    70	LE14:
    71	        ret
    72	        restore
    73	       LF14 = -104
    74	        LP14 = 96
    75	        LST14 = 96
    76	        LT14 = 96
    77	        .seg    "data"
    78	taco 4: 
    79	
    80	
    81	taco 4: 
    82	taco 4: 
    83	taco 4: cc -S -O1 a.c
    84	taco 5: m a.s
    85	        .seg    "text"                  ! [internal]
    86	        .proc   4
    87	        .global _main
    88	_main:
    89	!#PROLOGUE# 0
    90	!#PROLOGUE# 1
    91	        save    %sp,-104,%sp
    92	        st      %i0,[%fp+68]
    93	        st      %i1,[%fp+72]
    94	        call    _atoi,1
    95	        ld      [%i1+4],%o0
    96	        sethi   %hi(_a),%l0             ! [internal]
    97	        st      %o0,[%l0+%lo(_a)]
    98	        ld      [%fp+72],%o0
    99	        call    _atoi,1
   100	        ld      [%o0+8],%o0
   101	        sethi   %hi(_b),%o1             ! [internal]
   102	        st      %o0,[%o1+%lo(_b)]
   103	        ld      [%l0+%lo(_a)],%l0
   104	        ld      [%o1+%lo(_b)],%o1
   105	        cmp     %l0,%o1
   106	        bge,a   LY1
   107	        sethi   %hi(_a),%l0
   108	        sethi   %hi(_a),%o7
   109	        ld      [%o7+%lo(_a)],%o7
   110	        b       LE14
   111	        st      %o7,[%fp-4]
   112	LY1:                                    ! [internal]
   113	        ld      [%l0+%lo(_a)],%l0
   114	        sethi   %hi(_b),%l1
   115	        ld      [%l1+%lo(_b)],%l1
   116	        add     %l0,%l1,%l0
   117	        st      %l0,[%fp-8]
   118	LE14:
   119	        ret
   120	        restore
   121	        .seg    "data"                  ! [internal]
   122	        .common _a,4,"data"
   123	        .common _b,4,"data"
   124	taco 6: 
   125	taco 6: 
   126	taco 6: e
   127	exit
   128	
   129	script done on Sun Mar 10 16:20:11 1996
\end{verbatim}

In lines 3-17, we have a simple C source file a.c, which, in line 21 we
compile, using the -S option.  This option produces a file a.s, which
shows us the assembly language produced in the compilation.  We look
at that file starting at line 22 (in my .cshrc file, I have aliased
`m' to `more').

There are a number of assembler directives which have been inserted by
the compiler, such as those seen in lines 24-29, but they do not concern
us here.  

Recall that the SPARC was modeled on RISC I/II.  In particular, it uses
register windows.  Remember that registers r31-r26 in RISC I/II were
used to send parameters to the child function, i.e. as ``output'' to the
called function.  Similarly, in SPARC the ``output'' registers are
denoted \%o0, \%o1,..., \%o5, and have virtual register numbers r8-r13.

The registers r15-r10 in RISC I/II are for receiving parameters
(``input'') from the function which called the current one, and the
analogs in SPARC are denoted \%i0, \%i1,..., \%i5, and in virtual terms
are r24-r29.  

The SPARC registers r16-r23, denoted \%l0-\%l7, are intended for
storing local variables.

The SPARC registers named \%gn are analogous to RISC I/II's global
registers r9-r0; \%g1-\%g7 are virtual registers r1-r7.  

As in RISC I/II, register r0 (denoted \%g0) has the value 0 hardwired
into it.  Here are the rest:  r14 is the stack pointer, denoted by sp;
r15 is for temporary use, denoted \%o7; r30 is the frame pointer, fp;
and r31, \%i7, is for the return address.

Registers r8-r15 of a parent window are physically identical to r24-r31
of the child.

The \%fp register is the frame pointer, which serves a second copy of the
stack pointer.  So the expression [\%fp+0x48] in line 38 refers to the
location 0x48 bytes into the stack frame, for instance.  The brackets
notation ``[x]'' means ``contents of memory location x.''  Note that the
addressing here is similar to local addressing in Mac-1 (though on
SPARC any register can be used, not just the stack pointer).

The assembly-language notation lists the destination operand last.
For example, in line 39, the destination operand is \%o0.

Let's go right to line 36, which (together with the lines following 
it) is the result of compiling line 12, the call to atoi().
The call occurs at line 40, but first we have to get the parameter
ready.  This occurs in lines 36-39:

Keep in mind that main() is {\it itself} a function, whose parameters 
are argc and argv.  So, argc and argv will be in some \%i registers,
apparently \%i0 and \%i1, which are copied to the stack frame in lines
36 and 37.

Recall (line 7) that argv is a pointer-to-a-pointer.  After line 38
is executed, \%o0 now points to argv[0], \%o0+4 points to argv[1],
\%o0+8 points to argv[2], and so on.  This is reflected in lines
39 and 45.

After line 39 sets up the parameter, line 40 then is the actual call to
the function.  (The operand 1 indicates that there is 1 parameter in
the call.)  Of particular interest to us here is that the compiler has
placed a {\bf nop} in line 41, because after all, the
{\bf call} in line 40 is a form of branch.  Since we did not request
any optimization in our command line (line 21), the compiler has made
no attempt to fill this branch-delay slot.

The function's return value will be in \%o0.  We now have to store
it in the variable `a', which is done in lines 42 and 43:

The {\bf sethi} (``set high'') instruction (line 42) puts a value 
into the high 22 bits of the destination register.   Typically the
value consists of the high bits of an address, and {\bf sethi} is
executed in preparation for a load or store to that address,
in this case the {\bf st} in line 43.

Here are the details of line 43.  The high 22 bits of the address of 
`a' already have been copied (by line 42) to the high 22 bits of
the register \%o1.  By adding the low 10 bits of the address of `a',
we then get the full address of `a'.  Line 43 therefore stores to `a'.

Note, by the way, that the reason we must resort to this piecemeal
way to set up the address of `a' is that under the RISC philosophy
all instructions ought to be the same length, in this case 32 bits.
Since the address of `a' is already 32 bits, we could not fit both
it \underline{and} an op code into one 32-bit instruction!

Now, in line 83, we have recompiled the program, using first-level
optimization (-O1), to force the compiler to attempt to fill the
branch-delay slots.  

To see an example of how the code has changed, note that the {\bf ld}
which used to be just before the call, in line 39, now appears just
after the call, in line 95.  (It also now has slightly different form,
because the compiler has done some other optimizations in addition to
trying to fill delay slots.)

Now look at lines 106-107.  The {\bf bge} instruction in line 55 has
been changed to {\bf bge,a,} which means the following:  If the branch
in line 55 is taken, do execute line 56; if the branch is not taken,
then do not execute line 56.  The basic motivation is this:  Suppose we
have a branch to a target, T, but cannot find an instruction to move
and replace the {\bf nop} which had been tentatively placed in the
delay slot.  An alternative would be to move T itself into the delay
slot (assuming T is not also the target for some other branch).  If the
branch is taken, this saves us a clock cycle.  But if the branch is not
taken, we don't want T executed.  The SPARC architecture is set up to
deal with this by the ``,a'' appendage, where `a' stands for ``annul.''
It says that the instruction in the delay slot should be executed if the
branch is taken but is skipped otherwise.  We still lose a cycle in the
``not taken'' case, since we will not have prefetched the instruction
following the delay slot, but in the ``taken'' case we do save a cycle.

\subsection{Analysis of the UCB RISC I/II Pipelines}

\subsubsection{RISC I}

RISC I had a two-stage pipeline, and three internal buses, the first two
of which serve to copy register values into the ALU, with the third
copying the output of the ALU back to a register.  Here are diagrams
of the RISC I timing, both in its pipeline and its within-cycle
performance:

\vspace{0.2in}
\par
\psfig{file=IPipe.eps}
\par

The upper half of the picture gives an overview of the pipeline.  The
first stage is the instruction fetch, and the second is the execution
of the instruction.  Here I1, I2 and I3 are three instructions which
are executed consecutively.\footnote{As opposed to being consecutive
in memory.  E.g. I1 might be a conditional branch, in which case I2
would be consecutive to I1 in memory, but I3 would be consecutive in
memory to neither.}  The fetch of I2 is done during execution of I1,
etc.

The second half of the picture gives the details of the breakdown of one 
cycle into three subcycles, for each of the major components, the three 
buses and the ALU (as well as the outputs of the latter).\footnote{The
term {\bf precharge} here refers to a process necessitated by the 
NMOS electronics technology, and the modified six-transistor register
cell design, used in the RISC I and II chips.  The details may be 
found in Katevenis' thesis and the references it cites, but do not 
worry about this for the purposes of our course.}

Consider, for example, the instruction

\begin{verbatim}
add r2,r12,r13
\end{verbatim}

which puts the sum of registers r2 and r12 into r13.  In RISC I, registers
r2 and r12 would be put onto buses 1 and 2 during subcycle 1, the sum
would be formed during subcycle 2, and the sum would be written back to
bus 3 and register r13 during subcycle 3.

\subsubsection{RISC II}

RISC II's design is more complicated, but more efficient, using only
two internal buses:

\vspace{0.2in}
\par
\psfig{file=IIPipe.eps}
\par

Here the instruction add r2,r12,r13 would execute in the following way:
As before registers r2 and r12 would be read onto buses 1 and 2 during
subcycle 1.  The sum would be computed in subcycles 2 and 3, and then
the sum would be written back to bus 1 and r13 during subcyle 2 of the
\underline{next} cycle.\footnote{And thus subcycle 2 of the \underline{current}
cycle would do the write-back of the result from the \underline{previous}
cycle.}

This design is clearly more efficient.  The register file is kept busy
all the time (as opposed to being idle 1/3 of the time in the earlier
design), and precious chip real estate is saved:  With this two-bus design,
they were able to put 75\% more registers into a 25\% smaller area.

But what if the following instruction were, say,

\begin{verbatim}
add r4,r13,r1
\end{verbatim}

This wouldn't work.  During subcycle 1 for this instruction, r13 won't
be ready for copying onto bus 2, since the previous instruction, which
changes r13, has not done its write-back yet.  To solve this problem,
the designers included an {\bf internal forwarding} feature, which sends
to result of the previous instruction directly to the ALU, even though
the write-back for that instruction has not been done yet.  Indicator
bits in the ALU will be set or cleared, depending on whether the 
instruction that follows needs to use the internally-forwarded value.

Note again that a three-stage pipe is no slower than a two-stage, as long
as we keep the pipe filled.  Of course, whenever we have a taken branch,
that condition fails, but if the compiler optimizer has been successful
in filling most delay slots, the problem will arise only rarely.

\subsubsection{Taking Memory Access Into Account}

RISC I/II were designed on the assumption that memory (or cache) 
access time would be approximately the same as the time to do one
ALU computation.  The pipelines described above work that way, with
an instruction fetch being done during an ALU operation.

But what happens when a load or store instruction is encountered?  If 
we only have one path to memory---i.e. we have a cacheless system 
with only one system bus, or we have a combined instruction/data 
cache---then a load/store can't be done at the same time as an 
instruction fetch.  When a load/store instruction is encountered, the 
pipeline must be suspended.  This is how RISC I/II are actually 
implemented.

Another alternative, though, would be to have two paths to memory.
For example, there could be separate instruction and data caches, or
two system buses, both connected to a {\bf dual-port} memory.  In 
such case, a four-stage pipe such as the following would be appropriate:

\vspace{0.2in}
\par 
\psfig{file=DualPipe.eps}
\par

The boxes labeled `comp' indicate a computation, either that as in

\begin{verbatim}
add r4,r13,r1
\end{verbatim}

or possibly in a memory load/store instruction such as

\begin{verbatim}
ldxw r12,30000,r13
\end{verbatim}

The latter instruction is of interest to us now.  It uses indexed 
addressing mode; it loads the contents of the memory word which is
c(r12)\footnote{The notation c( ) means ``contents of.''} bytes past 
location 30000 into r13.  Thus the addition 30000+c(r12) must be
performed, so that even in the case of load/store instructions, the
labels `comp' in the boxes above do make sense.

The labels `mem' mean that \underline{if} a data memory access is needed
for this instruction---i.e. if the instruction type is load or
store---then the memory access will be done at this stage.  Otherwise,
this stage of the pipe becomes a dummy stage.

Note that double internal forwarding is used here, since the
register write-back occurs two time slots after the computation,
rather than one.  (In the picture the internal forwarding from
memI1 to compI3 genuinely comes after a memory access if I1 is
a store instruction.  Otherwise, the forwarding is done from this
dummy stage, and the forwarded value came from compI1.)

In addition, no dependencies are allowed between a load and the 
instruction which follows it.  If for example I1 is the ldxw 
instruction shown above, I2 is not allowed to use r13 as a source 
register, since compI2 occurs at the same time as memI1.  The 
compiler must avoid writing such code, resorting to inserting a nop 
instruction if necessary.

As before, the addition of one more pipe stage (from three to four),
even a stage for which non-load/store instructions do nothing, does
not hurt us at all---until we encounter a branch instruction.  So
again, the efficiency of the delayed-branch mechanism is vital.

\subsection{So, How Well Does RISC Work?}

Some statistics from Katevenis:

\begin{itemize}
\item Due to the simple, orthogonal instruction set, the load/store
architecture, etc., the C compiler for the machine was easy to write, 
taking only 6 person-months, including the optimization components.

\item Though RISC's simpler instruction set means we sometimes need
to use more instructions to accomplish a given task, the RISC I/II 
code was typically no more than 50\% larger than that of the VAX.

\item The ratio of execution times of hand-coded assembly language to
C code was about 90\% for RISC I/II, but around 45\% for the VAX.  In
other words, on a VAX you pay a much higher speed penalty for the
convenience of coding in a high-level language.

\item The {\bf control area} of the RISC II chip consists just of the
opcode decoder, and occupied only 0.5\% of the chip.  By contrast, in
the Motorola 68000,\footnote{The chip family formerly used in 
Macintoshes.} the control area, in this case meaning the microcode, 
occupied 68\% of the chip.

\item The chip layout design for RISC II was done by students in 12
person-months.  For the 68000 the process was done by experienced
professionals, and took 70 person-months.

\item RISC II and a PDP-11/70, having the same clock speed and the
same register-to-register addition time, were compared over 11 programs.
RISC II was on the average 2.1 times faster.
\end{itemize}

\subsection{What to Do With All That Extra Space?}

Now that RISC has liberated all this real estate, what might we do with
it?

\begin{itemize}
\item cache

\item more registers

\item on-chip stack

\item another CPU!
\end{itemize}

\end{document}


