CSC 103 Lecture Notes Week 1
Introduction to the Course
Review of Math and Java Background
Introduction to Algorithm Analysis



  1. Math review
    1. Exponentiation
      XAXB = XA+B

      XA / XB = XA-B

      (XA)B = XAB

      XN + XN = 2XN != X2N

      2N + 2N = 2N+1
    2. Logarithms
      1. Basic definition:
        XA = B iff logX B = A
      2. In computer science, logarithms are almost always base 2, which means the fundamental definition we care about is
        log 2N = N
      3. Useful theorems are:
        logA B = logC B / logC A , for A > 1, B,C > 0

        log AB = log A + log B , for A,B > 0

        log(AB) = B log A , for A,B > 0

        log X < X, for all X > 0

        The first of these is useful for computing log2 on a calculator that only has log10 or loge (ln).
    3. Series

      N

      sum 2i = 2N+1 - 1

      i=1



      N

      sum Ai = (AN+1 - 1) / (A - 1)

      i=1



      N

      sum i = N (N+1) / 2

      i=1
    4. Modular arithmetic
      1. Basic definition:
        A mod N = the remainder after division of A by N.
      2. Useful theorems are:
        if A = B mod N then A + C = B + C mod N

        if A = B mod N then AD = BD mod N
    5. Proofs
      1. By induction:
        1. The goal is to prove that a statement is true for all N, where N is the measure of something.
        2. Most typically in computer science induction proofs, N will be the number of elements in some data structure or the number of steps that an algorithm takes to execute.
        3. An induction proof has two steps:
          1. The proof of a base case, typically when N = 0 or 1.
          2. The proof of the inductive step, where we assume what we're trying to prove is true for N, and prove it's true for N + 1.
        4. Your CSC 141 discrete math book has examples of induction proofs; we'll see one a little later in these notes, and more as we progress through the quarter.
      2. Proof by counter example.
        1. This is used to refute a statement that is claimed true for all N.
        2. It works by providing a single value of 0 t;= N < where the statement is false.
      3. Proof by contradiction.
        1. Assume that what we're trying to prove is false.
        2. Then show that this assumption leads to an erroneous result.
        3. Therefore, the assumption of falseness cannot be correct, which means what we're trying to prove must be true.

  2. A brief introduction to recursion.
    1. The use of recursion will be key to a number of the algorithms we will study this quarter.
    2. The basic structure of a recursive algorithm is the following:
      1. Base case: Define what the algorithm computes in the simplest possible case.
      2. Recursive case: Define what the algorithm computes in a single step through the algorithm, such that the single step can be applied recursively until the base case is reached.
    3. Consider the following recursive algorithm for summing the elements in a list of numbers:
      1. Base case: If there are no elements in the list, compute the sum as zero.
      2. Recursive case: Compute the sum by adding the first element of the list to the recursive sum of the rest of the list.
    4. Here is a Java method that implements this recursive summing algorithm:
      /**
       * Return the sum of the elements in the given numbers array, starting at
       * the given index position.  If the position is past the end of the array,
       * return 0.
       */
      public int sumArray(int[] numbers, int position) {
      
          /*
           * Base case: Return 0 if the position is past the end of the array.
           */
          if (position >= numbers.length)
              return 0;
      
          /*
           * Recursive step: Add the element at the current position to the
           * recursive sum of the elements at position+1 to the end.
           */
          else
              return numbers[position] + sumArray(numbers, position+1);
      }
      
    5. This example of recursion is very simple, and could actually be accomplished more efficiently with a for-loop.
      1. The example does however illustrate the basic idea of a recursive algorithm, and we will use it next week as part of our lab work on stacks.
      2. We will definitely see more sophisticated uses of recursion as the quarter progresses, particularly when we get to the study of tree data structures.

  3. Generic objects in Java data structures.
    1. An important concept we will use in our study of data structures is that of generic objects, and in particular generic collections.
    2. A generic collection is a data structure that can hold values of any type, rather than a specific type.
    3. Different programming languages provide different means to define generic collections.
      1. In C++, we use templates to define generic structures.
      2. In Java, the key to defining generic collections is the use of generic class and interface definitions, including the most generic Object class.
    4. To motivate the use of generics, consider the following Java method to find the largest element in an array of integers.
      /**
       * Return the largest element in the given array of integers.  Return
       * Integer.MIN_VALUE if the array is null or empty.
       */
      public int findLargest(int[] numbers) {
      
          int largestValue;                       // Largest value so far
          int i;                                  // Array index
      
          /*
           * Return Integer.MIN_VALUE if the array is null or empty.
           */
          if ((numbers == null) || (numbers.length == 0)) {
              return Integer.MIN_VALUE;
          }
      
          /*
           * Initialize the largest value to the zeroth array element, then loop
           * through the remaining elements, reassigning the largest value
           * whenever one of the elements is greater than the largest so far.
           */
          for (largestValue = numbers[0], i = 1; i < numbers.length; i++) {
              if (largestValue < numbers[i]) {
                  largestValue = numbers[i];
              }
          }
          return largestValue;
      }
      
    5. The problem with this implementation is that it only works for arrays that contain integer values.
      1. Suppose we wanted to find the largest element in an array of some other type of values, say an array of arrays?
      2. If we try to send findLargest such an array, the Java compiler will complain that an array of arrays is not compatible with an array of ints.
    6. The solution to the problem is to define a findLargest method that takes an array of more generic elements than just ints; here's the Java code for such a method:
      /**
       * Return the largest element in the given array of comparable objects.
       * Return null if the array is null or empty.
       */
      public Object findLargest(Comparable[] objects) {
      
          Comparable largestValue;                // Largest value so far
          int i;                                  // Array index
      
          /*
           * Return null if the array is null or empty.
           */
          if ((objects == null) || (objects.length == 0)) {
              return null;
          }
      
          /*
           * Initialize the largest value to the zeroth array element, then loop
           * through the remaining elements, reassigning the largest value
           * whenever one of the elements is greater than the largest so far.
           */
          for (largestValue = objects[0], i = 1; i < objects.length; i++) {
              if (largestValue.compareTo(objects[i]) < 0) {
                  largestValue = objects[i];
              }
          }
          return largestValue;
      }
      
    7. The key differences between this new version of findLargest and the original integer-only version are the following:
      1. The return value is of type Object, which is the most generic type of all Java classes.
        1. Object is at the root of the Java class hierarchy.
        2. Every Java class has Object as a superclass.
      2. The input to findLargest is an array of type Comparable.
        1. Comparable is a Java interface that requires its extending classes to implement the compareTo method.
        2. compareTo returns a negative integer, zero, or a positive integer, based on whether the object it is applied to is less than, equal to, or greater than the input parameter object.
    8. We will see continued uses of Object, Comparable, and other generic Java classes as we discuss various data structures.

  4. Java exceptions.
    1. Exceptions in Java are objects that are used to indicate that a method has failed in some way.
    2. Exceptions are tied to the Java try, catch, and throw statements.
    3. An upcoming example introduces the use of exceptions.
    4. We will study the subject of exception handling further as we introduce new data structures.

  5. Java input/output.
    1. The Java language has been designed to write programs that talk to the user primarily through a GUI -- graphical user interface.
      1. As a result, input/output on a standard terminal interface is not all that well-supported in Java.
      2. You are undoubtedly aware of this from your programming work in CSC 101 and 102.
    2. The following program provides a quick review of terminal i/o in Java, as well as the use of exception handling.
      1. As the comment indicates, it is a test driver program for the recursiveSum method presented earlier in the notes.
      2. We will dissect this program during the first week's lab.
    import java.io.*;
    import java.util.*;
    
    /****
     *
     * Class BasicRecursionTest illustrates the use of a very basic recursive
     * function to sum the elements in an array.  The elements are read from stdin
     * and stored in the array.  The resulting sum is printed to stdout.
     *                                                                          <p>
     * The summing computation is performed by the sumArray method in the <a href=
     * BasicRecursion.html> BasicRecursion </a> class.
     *
     * @author Gene Fisher (gfisher@calpoly.edu)
     * @version 2apr01
     *
     */
    
    public class BasicRecursionTest {
    
        /**
         * Call the recursiveSum method and print the result to stdout.
         */
        public static void main(String args[]) throws IOException {
    
            BasicRecursion basic;           // The class with the method
            BufferedReader reader;          // Input stream reader
            StringTokenizer toker;          // Input stream tokenizer
            final int NUM_ELEMENTS = 10;    // Number of inputs allowed
            int[] numbers                   // Input array
                = new int[NUM_ELEMENTS];
            int i;                          // Working array index
    
            /*
             * Instantiate a BasicRecursion class object.
             */
            basic = new BasicRecursion();
    
            /*
             * Prompt for the input.
             */
            System.out.print("0nput up to " +
                Integer.toString(NUM_ELEMENTS) + " numbers: ");
    
            /*
             * Initialize the input reader and tokenizer.
             */
            reader = new BufferedReader(new InputStreamReader(System.in));
            toker = new StringTokenizer(reader.readLine());
    
            /*
             * Read each input from the terminal.
             */
            for (i = 0; toker.hasMoreElements() && i < 10; ) {
    
                /*
                 * Store the next input in the array, ignoring any non-numeric
                 * input.
                 */
                try {
                    numbers[i] = Integer.parseInt(toker.nextToken());
                }
                catch (NumberFormatException e) {
                    continue;
                }
    
                /*
                 * Increment the input index if the read was a legal number.
                 */
                i++;
    
            }
    
            /*
             * Output the array sum to stdout.
             */
            System.out.print("The sum is: ");
            System.out.println(basic.sumArray(numbers, 0));
            System.out.println();
    
        }
    
    }
    

    End of Text Chapter 1
    On to Chapter 2 on Algorithm Analysis




  6. What is algorithm analysis?
    1. It's the process of determining how fast an algorithm runs and how much space it uses.
    2. The goal of analyzing algorithms is to determine which ones run the fastest and use smallest of storage under some particular set of conditions.
    3. For example, some algorithms work well when a collection of data is randomly organized, whereas other algorithms are only efficient when data are in sorted order.
    4. The major objective for CSC 103 is to develop your analysis skills so you can choose the appropriate data structures and algorithms to meet particular program specifications.

  7. O-notation.
    1. Algorithm analysis is mathematically based, and uses a mathematical notation.
    2. The common name for it is O-notation, where the "O" stands for "order of magnitude".
      1. As the name suggests, O-notation is used to measure the broad performance of an algorithm -- i.e., the general order of magnitude of its execution time.
      2. This measure leaves out low-level details of exactly how long individual program statements take to run on particular kinds of computers.
    3. Here are the formal definitions for an algorithm timing function T, for a problem of size N:
      1. T(N) = O(f(N)) if there are positive constants c and n0 such that T(N) t;= cf(N) when N t;= n0
      2. T(N) = (g(N)) if there are positive constants c and n0 such that T(N) t;= cg(N) when N t;= n0
      3. T(N) = (h(N)) iff T(N) = O(h(N)) and T(N) = (h(N))
      4. T(N) = o(p(N)) if T(N) = O(p(N)) and T(N) != (h(N))
    4. What this all means is the following:
      1. O(f(N)) is an upper bound on how much time an algorithm takes.
        1. The name for this bound is "big-Oh".
        2. The function f(N) defines an upper bound on the growth rate of the algorithm timing function.
        3. For example, if the time it takes an algorithm to run is linearly proportional to the problem size, then f(N) = N, and we say that the algorithm is O(N), or linear.
        4. In contrast, if the time it takes an algorithm to run is proportional to the square of the problem size, then f(N) = N2, and we say that the algorithm is O(N2), or quadratic.
      2. Similarly, the "theta" notation (g(N)) is a lower bound on how much time an algorithm takes.
      3. The "omega" notation (h(N)) is an exact bound on how much time an algorithm takes.
      4. Finally, the "little-Oh" notation o(N)) is a strict upper bound on how much time an algorithm takes.

  8. What is the "problem size"?
    1. The variable N used in all of the notations refers to the size of the problem an algorithm is designed to solve.
    2. Almost always in what we'll study, N is the size of the data structure the algorithm works with.
    3. For example, if an algorithm adds the N elements of an array in a time that is linearly proportional to N, then the algorithm is O(N).
    4. If an algorithm takes time proportional to N2 to sort an N-element array, the algorithm is O(N2).

  9. Growth rates for typical O-notation timing functions.
    Function Name Goodness
    O(c) or O(1) Constant Best
    O(log N) Logarithmic Very good
    O(log2 N) or O(log log N) Log-squared Very good
    O(N) Linear Generally good
    O(N log N) Decent, particularly for sorting
    O(N2) Quadratic OK, but we'd prefer something better
    O(N3) Cubic Ditto
    O(Nk) Polynomial Not so good, but we can live with it if we have to
    O(2N), O(XN), O(N!) Exponential Bad news
    The plot graphs on page 34 of the book provide helpful visualizations of how fast these functions grow in relation to one another.

  10. Best, worst, and average cases.
    1. Most frequently when we measure algorithm running times, we are concerned with the worst case performance.
      1. We care about this because we want to know how bad things can possibly get for all possible inputs to an algorithm.
      2. In general, when a big-Oh bound is given for an algorithm, it means the worst case behavior, unless specifically noted otherwise.
    2. Average-case performance is also frequently of interest, since it measures the typical or regular behavior of an algorithm.
      1. Average case performance is of particular interest for certain algorithms where the worst-case behavior is very rare, or can be isolated to known data configurations.
      2. Unfortunately, average case analysis is often considerably more difficult to produce than worst case analysis.
    3. Best-case is only occasionally of interest.

  11. General running time calculations.
    1. As we have been discussing, we are concerned overall with the broad measure of algorithm performance, not low-level coding details.
    2. Hence, when calculating an algorithm timing function, we will make some general assumptions about how long it takes to execute specific elements of an algorithm.
    3. First off, individual expression operations are all assumed to take some constant amount of time.
      1. For example, assignment statements, arithmetic operations, and relational comparisons are all constant-time.
      2. Even though it may take longer to add two numbers than to compare them, we don't really care because the time for both operations is constant.
    4. The time to execute an if-statement is the constant time needed to evaluate its test, plus the maximum of the times for its if-clause and else-clause.
    5. The running time for loops is generally where the O(f(N)) time is consumed.
      1. The mechanics of the loop involve some constant time for the initialization, test, and increment expressions.
      2. It's the body of the loop that is executed repetitively where the O(N) behavior occurs.
      3. Specifically, any loop that executes N times, is O(N).
      4. E.g.,
        for (i = 0; i < N; i++) { ... }
        
        is O(N), assuming that the body does not break out of the loop.
    6. Nested loops are O(N2), e.g.,
      for (i = 0; i < N; i++) {
          for (j = 0; j < N; i++) { ... }
      }
      
      is O(N2); (think about why this is the case).
    7. Consecutive statements are the sum of their individual running times.
      1. Since we're dealing with orders of magnitude, the running time of multiple consecutive assignment statements may be a large constant value when they're all summed up, but it's still just a constant value; i.e., O(3C) = O(100C) = O(C).
      2. Consecutive statements of larger orders of magnitude always dominate the running time, e.g.,
        for (i = 0; i < N; i++) { ... }
        for (i = 0; i < N; i++) { ... }
        for (i = 0; i < N; i++) {
            for (j = 0; j < N; i++) { ... }
        }
        
        is O(N2) overall, since O(N) + O(N) + O(N2) = O(N2)
    8. The running time for recursive algorithms will generally take individual analysis.
      1. In a simple case such as the sumArray algorithm above, the recursion is performing the same form of computation as a for-loop, so it's O(N).
      2. We will see cases later in the quarter where recursive "divide and conquer" algorithms run in logarithmic time.

  12. An example analysis -- linear search of a list.
    1. Consider the following algorithm that performs a simple linear search through an array:
      /**
       * Return the position in the given numbers array of the first occurrence of
       * the given numeric value.  Return -1 if the element is not in the array
       * or if the array is null.
       */
      public int searchArray(int[] numbers, int value) {
      
          int i;                                      // Array index
      
          /*
           * Return -1 if the array is null.
           */
          if (numbers == null) {
              return -1;
          }
      
          /*
           * Loop through the array, comparing the input value with each
           * element.  Return the index of the first element that equals the
           * value, if any.
           */
          for (i = 0; i < numbers.length; i++) {
              if (numbers[i] == value) {
                  return i;
              }
          }
          return -1;
      }
      
    2. As outlined above, N here is the length of the numbers array.
    3. The best case for this search is O(c).
      1. In the best case we get lucky, when what we're searching for is the first element of the array.
      2. In this case we only execute one iteration of the loop, and all other computations take constant time.
      3. Hence, the overall performance is O(c), or we might even say O(1) since we do exactly one iteration of the loop.
    4. The worst case for the search is O(N).
      1. This is the case where we're unlucky, and what we're searching for is all the way at the end, or not there at all.
      2. In this case we must execute all N loop iterations, which makes the running time O(N).
    5. The average case is also O(N), but requires more analysis than the other cases.
      1. First we must make some assumptions about order of the array elements in the average case.
      2. If we assume that elements are in random order in the array, than statistical analysis says that each element is equally likely to occur at any give array position.
      3. Thus on average, if we search the array N times, we'll end up searching each position once.
      4. Given this, we can define the average running time in terms of the number of loop iterations as follows:
        (1 + 2 + 3 + ... + N) / N
        
        which equals

        N


        ( sum i ) / N

        i=1

      5. If we recall that the closed-form formula for the summation is (N (N+1)) / 2, then we have the average time as ((N (N+1)) / 2) / N, which is (N + 1) / 2, which is O(N).
      6. To be entirely rigorous, we should in fact prove the closed form, i.e., that

        N

        sum i = (N (N+1)) / 2

        i=1
      7. This can be done by the following inductive proof (which appears in chapter 1 of the CSC 141 text book):
        1. The base case for N = 1 is

          1

          sum i = (1 (1+1)) / 2

          i=1
          which reduces to 1 = 1.

        2. For the inductive step, we assume

          N

          sum i = (N (N+1)) / 2

          i=1
          and prove

          N+1

          sum i = ((N+1) (N+1+1)) / 2

          i=1
          Working from the summation side of the equation, we have

          N

          sum i + N+1 , by the definition of summation

          i=1
          = (N (N+1)) / 2 + N + 1 , by the inductive hypothesis
          = (N (N+1)) / 2 + 2N/2 + 2/2 , by the identity rule
          = (N (N+1) + 2N + 2) / 2 , by common denominator
          = (N2 + N + 2N + 2) / 2 , by distributive rule
          = (N2 + 3N + 2) / 2 , by term addition
          = ((N + 1) (N + 2)) / 2 , by factoring
          q.e.d.
      8. The point of all of this is to demonstrate that average case analysis can be, and often is, non-trivial.

  13. Empirical timing analysis.
    1. All of the timing analysis so far discussed has been analytic; that is, we have discussed general formulas for computing how long an algorithm runs.
    2. To confirm such theoretical analysis, we generally want to code up the algorithm and get some actual, i.e., empirical, running times.
    3. Using UNIX and Java, there are two specific empirical timing techniques we'll employ this quarter:
      1. The UNIX time command.
      2. The Java System.currentTimeMillis method.
    4. We'll examine these methods in upcoming labs.



index | lectures | labs | handouts | examples | assignments | solutions | grades | help