Title

ECS 120 Theory of Computation
Complexity Theory
Julian Panetta
University of California, Davis

Measuring Run Time

3Sum Problem:
Given a set \(A\) of \(n\) integers (\(A \subset \Z\), \(|A| = n\)), check if there are three elements that sum to zero (i.e., \(a + b + c = 0\) for \(a, b, c \in A\)).

  • Two competing algorithms:

    def three_sum_1(A):
    for i in range(len(A)):
        for j in range(i, len(A)):
            for k in range(j, len(A)):
                if A[i] + A[j] + A[k] == 0:
                    return True
    return False
    def three_sum_2(A):
    two_sums = set()
    for i in range(len(A)):
        for j in range(i, len(A)):
            two_sums.add(A[i] + A[j])
    for i in range(len(A)):
        if -A[i] in two_sums:
            return True
    return False

  • How do we know which one is faster?

  • Simple idea: measure the “wall-clock time” it takes to run each on the same input.

Wall-clock Time Experiments

$ python 3sum.py 1 50.txt
Result: True
Elapsed time: 0.0003481250000000047
$ python 3sum.py 2 50.txt
Result: True
Elapsed time: 0.00020370899999999637
$ python 3sum.py 1 50.txt
Result: True
Elapsed time: 0.00013012499999999483
$ python 3sum.py 2 50.txt
Result: True
Elapsed time: 0.0005745000000000056
  • One problem: this can be noisy and non-repeatable
    • Other processes running on the computer at the same time.
    • Thermal state, cache state, etc.
    • We could work around these fluctuations by running the code many times and taking the average.

1000 runs in 0.21506016700000002

1000 runs in 0.26039516700000004

1000 runs in 0.20773370800000002

1000 runs in 0.27380958299999997

def three_sum_1(A):
  for i in range(len(A)):
      for j in range(i, len(A)):
          for k in range(j, len(A)):
              if A[i] + A[j] + A[k] == 0:
                  return True
  return False
def three_sum_2(A):
    two_sums = set()
    for i in range(len(A)):
        for j in range(i, len(A)):
            two_sums.add(A[i] + A[j])
    for i in range(len(A)):
        if -A[i] in two_sums:
            return True
    return False

Wall-clock Time Experiments

  • Another problem: performance is strongly impacted by low-level implementation details
    • Different programming languages.
    • Different interpreter/compiler versions.
    • Compilation flags when building the code.
    • Different hardware capabilities (CPU vs GPU, etc.)
$ g++ 3sum.cc -o 3sum
$ ./3sum 1 50.txt
Result: True
1000 runs in 0.00394083 seconds
$ ./3sum 1 50.txt
Result: True
1000 runs in 0.13204 seconds
$ g++ 3sum.cc -O3 -o 3sum
$ ./3sum 1 50.txt
Result: True
1000 runs in 0.000488875 seconds
$ ./3sum 1 50.txt
Result: True
1000 runs in 0.0416488 seconds
def three_sum_1(A):
  for i in range(len(A)):
      for j in range(i, len(A)):
          for k in range(j, len(A)):
              if A[i] + A[j] + A[k] == 0:
                  return True
  return False
def three_sum_2(A):
    two_sums = set()
    for i in range(len(A)):
        for j in range(i, len(A)):
            two_sums.add(A[i] + A[j])
    for i in range(len(A)):
        if -A[i] in two_sums:
            return True
    return False

The Turing Machine as a Standardized Environment

  • To ensure we’re making a fair comparison of the algorithms themselves, we consider a “standard environment” when analyzing performance.
  • Specifically, we can use a Turing machine as a precisely defined environment.
  • “Time” = “number of steps taken by the Turing machine.”

Let \(M\) be a TM, and \(x \in \binary^*\).
Define \(\texttt{time}_M(x)\) to be the number of configurations that \(M\) visits on input \(x\).

What is \(\texttt{time}_M(x)\) when \(M\) immediately halts on \(x\)?

  • 0
  • 1

Factoring out Input Dependence

  • “Difficult” inputs can take longer to run than “easy” inputs.
    In the following, inputs num_x.txt and num_y.txt both contain 500 numbers.
$ ./3sum 1 num_x.txt
Result: False
1000 runs in 7.61071 seconds
$ ./3sum 2 num_x.txt
Result: False
1000 runs in 0.373348 seconds
$ ./3sum 1 num_y.txt
Result: True
1000 runs in 7.5e-07 seconds
$ ./3sum 2 num_x.txt
Result: True
1000 runs in 0.371461 seconds

What happened!?

num_x.txt: \(\{373, 351, 694, 389, 300, \cdots \}\)

num_y.txt: \(\{0, 351, 694, 389, 300, \cdots \}\)

Factoring out Input Dependence

  • “Difficult” inputs can take longer to run than “easy” inputs.
  • How do we compare two algorithms \(A, B\) if \(\texttt{time}_A(x) < \texttt{time}_B(x)\) but \(\texttt{time}_A(y) > \texttt{time}_B(y)\) even when \(|x| = |y|\)?
  • We address this issue by considering only the size \(n\) of the input and not the contents.
  • We then perform a worst-case analysis:
    we determine the longest possible time \(A\) and \(B\) can run on an input of size \(n\).

If \(M\) is total, define the (worst-case) running time or time complexity of \(M\) to be the function \(t : \mathbb{N} \to \mathbb{N}^+\) such that \[ t(n) = \max_{x \in \binary^n} \texttt{time}_M(x) \]

Why must \(M\) be total?

We call such function \(t(n)\) a time bound.

Finally, we ask: “How quickly does the running time \(t(n)\) grow as \(n\) increases?”

Asymptotic Analysis

  • Rather than worring about specific values of \(n\) and \(t(n)\), we consider the growth rate of \(t(n)\) as \(n\) increases.
  • We’ll say an algorithm is faster if it has a smaller growth rate.

    • We have experimental evidence that three_sum_2 is faster than three_sum_1 in this sense:

      Input Size Algorithm 1 Time (s) Algorithm 2 Time (s)
      50 0.0118399 0.0447408
      100 0.0749655 0.166695
      500 7.68075 0.377361

      These times were collected for a “worst-case input”!

    • To formally define and compare the growths of time bound functions \(t(n)\), we use asymptotic analysis.

  • Simplifications:
    • Ignore constant factors (e.g., \(2n^2\) considered equivalent to \(3n^2\)).
    • Consider only the highest-order term (e.g., \(2 n^2 + n + 2\) is equivalent to \(n^2\)).
  • These not only ease analysis but also help us ignore low-level implementation details.

Asymptotic Analysis

Given nondecreasing \(f, g : \N \to \R^+\), we write \(f = O(g)\) if there exists \(c \in \N\) such that \[ f(n) \le c \cdot g(n) \quad \text{for all } n \] We call \(g\) an asymptotic upper bound for \(f\).

  • This is “Big-O” notation: \(f = O(g)\) means \(f\) grows no faster than \(g\).
  • Important cases:
    • Polynomial bounds: \(f = O(n^c)\) for \(c > 0\) (“\(f\) is polynomially bounded”)
      • Examples: \(n^2, n^3, n^{2.2}, n^{1000}\)
      • Counterexamples: \(n^{\log(n)}\)
    • Exponential bounds: \(f = O(2^{c n^\delta})\) for some \(c, \delta > 0\)
      • Examples: \(2^n, (2^n)^2, 2^{100n}, 2^{0.01 n}, 2^{n^2}, 2^\sqrt{n}, e^{n^2}\)
      • Also: \(a^{n^\delta}\) for any \(a > 1\) and \(\delta > 0\)

Asymptotic Analysis

Given nondecreasing \(f, g : \N \to \R^+\), we write \(f = o(g)\) if \[ \lim_{n \to \infty} \frac{f(n)}{g(n)} = 0 \]

  • This is “Little-O” notation: \(f = o(g)\) means \(f\) grows strictly slower than \(g\).
  • We can say algorithm \(A\) is (asymptotically) faster than algorithm \(B\) if \(t_A(n) = o(t_B(n))\).
  • Warning, just because \(f = O(g)\) and \(g \ne O(f)\) does not mean \(f = o(g)\)
    (see pathalogical counterexample in Figure 9.1 of the lecture notes)

Strategies for Comparing Growth Rates

  • To show \(f = o(g)\) you can always apply the definition and prove: \(\lim_{n \to \infty} \frac{f(n)}{g(n)} = 0\).
  • But usually there is an easier shortcut that you can apply.
    1. First, simplify by removing constants and lower-order terms (without changing the growth rate):
      • Examples: \[10 n^7 + 100 n^4 + n^2 + 10n = O(n^7), \quad\quad \quad\quad 2^n + n^{100} + 2^n = O(2^n) \]
      • This is helpful for the common case of an algorithm with multiple stages of different complexities.
    2. Second, you might notice that \(g(n)\) is of the form \(f(n) \cdot h(n)\) with an unbounded \(h(n)\).
      • Example: \(f(n) = n, \quad g(n) = n \log(n), \fragment{\quad g(n) = f(n) \cdot \log(n)}\)
      • Since \(\lim_{n\to\infty} \log(n) = \infty\), we conclude \(n = o(n \log(n))\).

      Proof?

    3. Remember that applying a log or raising to a power less than \(1\) shrinks the growth rate.
      \(\log(n) = o(n), \quad \fragment{\sqrt{n} = o(n),} \quad \fragment{\log(n^4) = o(n^4)}\) (actually \(\log(n^4) = 4 \log(n) = O(\log(n))\))
    4. Try taking a log of both functions since \(\log(f(n)) = o(\log(g(n)))\) implies \(f(n) = o(g(n))\).
      • Example: compare \(2^{n^2}\) to \(n^n\) \(\log(2^{n^2}) = \fragment{n^2},\ \log(n^n) = \fragment{n \log(n)} \fragment{\implies n^n = o(2^{n^2})}\)
      • Warning: \(f(n) = o(g(n))\) does not imply \(\log(f(n)) = o(\log(g(n)))\) (Counterexample: \(2^n = o(2^{2n})\), but \(n \ne o(2n)\))

Asymptotic Analysis Facts to Remember

  • \(1 = o(\log \log n)\) (any unbounded function outgrows a constant)
  • \(\log \log n = o(\log n)\)
  • \(\log n = o(n^c)\) for any \(c > 0\)
  • \(\sqrt{n} = o(n)\)
  • \(n^c = o(n^k)\) if \(c < k\)
  • \(n^c = o(2^{n^\delta})\) for any \(c > 0\) and any \(\delta > 0\)