|
Foundations of Computer Science, Rensselaer
Online Course A foundation for computing, focusing on discrete math and the theory of computing. Suitable for a sophomore in computer science or related area. For maximum benefit, one year of programming and calculus is helpful. Instructor: Malik Magdon-Ismail Text Book: Discrete Mathematics and Computing by Magdon-Ismail. Part Ⅰ: Discrete Mathematics and Probability. (lectures 1-21) Part Ⅱ: Theory of Computing. (lectures 22-28) |
|
Copyright for all material remains with the original copyright holder (in most cases the authors of the "Discrete Mathematics and Computing" book).
Ⅰ. Discrete Mathematics and Probability
|
|
1: A Taste. What is discrete mathematics? What is computing? Let us whet the appetite with sample problems and a few challenge problems. Modules: (10min) Introduction, the storyline and background. (23min) Virus spreading, speed dating, online advertising. (07min) Domino puzzle. (05min) Matching subset sum problem. (16min) A little wisdom from some of the greats. (03min) 3 Challenge problems. Find the matching subsets. Download the numbers here. Reading: DMC Chapter 1,2 Slides: (full) (handout) (compact) Full video on youtube (63min) |
|
|
2: Discrete Objects + Proof. Meet the three main discrete objects. Using them, let us build an informal intuition of proofs. What does it take to convince? Modules: (04min) Intro (38min) Sets (N,Z,Q) , sequences and graphs. (19min) Proof is convincing: speed-dating, even squares? (09min) Extremist parties: 3-cliques or 3-wars. (17min) Hippasus, well-ordering and root-2 is irrational (05min) Making and proving claims. Reading: DMC Chapter 1,2 Slides: (full) (handout) (compact) Full video on youtube (92min) |
|
PAUSE. Reinforce and Practice: Assignment 1.
| |
|
|
3: Making
Precise Statements. Rigorous proof is useless if the point is not clear. We all need to agree on what is being said. We need a precise framework for making claims and understanding their logical meaning. Modules: (02min) Recap of discrete objects. (09min) Claims and connectors: NOT, AND, OR, IF-THEN. (22min) Complex claims, truth tables and IF-THEN. (10min) Equivalence, if-and-only-if. (10min) Order is important. Inferences. (29min) Predicates, quantifiers and negated quantifiers. Reading: DMC Chapter 3 Slides: (full) (handout) (compact) Full video on youtube (82min) |
|
|
4: Proof. A proof is golden. You can take it to the bank. What are the main proof patterns? Our focus is on proving and disproving implications. The goal of a proof is always to convince the reader of something. If you fail to convince, your proof failed. Correct is important. Well written is just as important. Modules: (36min) Proving and disproving implications. (03min) Implications and the relationship to sets. (09min) Proving equivalence. (16min) Proof by contradiction. (08min) Proofs about sets. (05min) Writing good proofs (mathematical essays). Reading: DMC Chapter 4 Slides: (full) (handout) (compact) Full video on youtube (76min) |
|
PAUSE. Reinforce and Practice: Assignment 2.
| |
|
|
5: Induction. How do you prove something is always true? It's a miracle that we have blockbuster way to prove "for all". It is based on the simple idea of start simple and only then get complex. Modules: (03min) Recap of proof patterns. (10min) The essence of induction: postage stamps. (12min) The predicate and "for all n". Why induction? (25min) A first proof and infinite domino chains. (08min) Gauss' great feat. Induction and integer sums. (05min) A BAD "proof" of the induction step. (05min) The challenge. You must know what to prove. (11min) Induction and well ordering (06min) Induction is more than just proofs of formulas. Reading: DMC Chapter 5 Slides: (full) (handout) (compact) Full video on youtube (85min) |
|
|
6: Strong Induction. A stronger claim can be easier to prove. All inductions are equivalent but some variants can be more convenient, depending on the problem. Strong induction is always easier than ordinary induction and has many uses. Modules: (13min) Recap of induction, and a harder example. (47min) Strengthening the induction hypothesis: L-tile land. (09min) Leaping induction and the postage stamp machine. (26min) Strong induction and prime factorization. (07min) Binary expansions and strategic games (Nim). (26min) Supplement: Binary expansions and games of strategy (Nim) Reading: DMC Chapter 6 Slides: (full (handout) (compact) Full video on youtube (102min) (Supplement, 26min) |
|
PAUSE. Reinforce and Practice: Assignment 3.
| |
|
|
7: Recursion. To reason backwards is key. Recursive functions, sets, sequences and structures are fundamental to representing and manipulating data. The link between induction (small to big) and recursion (big to small) is unbreakable. Modules: (02min) Recap on induction (14min) Basics of recursion and some examples. (12min) The bond between recursion and induction. (13min) Induction proofs about recursions. (11min) Fibonacci numbers and recurrences. (09min) Recursive programs and runtimes. (34min) Recursive sets/structures: numbers, strings, trees. Reading: DMC Chapter 7 Slides: (full) (handout) (compact) Full video on youtube (94min) |
|
|
8: Proofs with Recursion. Recursion is powerful but dangerous. Always make sure to prove your recursive functions and recursive objects have the desired properties. If you want to be good at recursion, you need to be good at induction. And you do need to be good at recursion. Modules: (02min) Recap of recursion: functions, sets, structures. (07min) What is in a recursive set? (22min) Structural induction, inheritance and blue-eyes. (36min) Matched parentheses, N, palindromes, arimethic. (18min) Rooted binary trees (RBT): (04min) Summary of structural induction. Reading: DMC Chapter 8 Slides: (full) (handout) (compact) Full video on youtube (90min) |
|
PAUSE.
Assess: Take Quiz 1
|
Fall 2020
(solution)
Fall 2019 (solution) Fall 2017 (solution) Fall 2016 (solution) Fall 2015 (solution) |
|
|
9: Sums, Asymptotics & Order Notation. We compute sums to analyze algorithms. But, we don't need exact runtimes to get at the high-level differences between algorithms. We need simple tools to compute and compare runtime of algorithms. Modules: (11min) Why do we need sums in computer science? (22min) Notation, basic rules, common sums and practice. (12min) Nested sums. (36min) Comparing runtimes: asymptotics and O-notation. (15min) The integration method for approximating sums. Reading: DMC Chapter 9 Slides: (full) (handout) (compact) Full video on youtube (96min) |
|
|
10: Number Theory. Numbers have attracted the purest of mathematicians, and we're grateful. Everything we do on computers involves numbers. No number theory means no secure email, social media, online banking, ... Modules: (09min) Basics: division, quotients, remainders and primes (18min) Greatest common divisor (GCD). Euclid's algorithm. (25min) Bezout's identity: an almost formula for GCD. (08min) Bezout and GCD in the movies: Die Hard. (11min) Unique prime factorization (Fund. Thm. Aritmetic) (26min) Modular arithmetic and cryptography. (06min) RSA public key cryptography. Reading: DMC Chapter 10 Slides: (full) (handout) (compact) Full video on youtube (104min) |
|
PAUSE. Reinforce and Practice: Assignment 4.
| |
|
|
11: Graphs I, Notation and Basics. Relationships are everywhere. Graphs are the tool for problems that involve relationships. Almost all problems in the real world in some way use graphs, you just need to tease out the graph. Become one with graphs. No dying person ever said "I wish I knew less about graphs." Modules: (13min) Notation/basics: paths, connectivity, isomorphism. (26min) Graph representation & degree sequences. (12min) The handshaking theorem. (18min) Trees, planar graphs, maps and Euler's invariant, (09min) Weighted, directed and multi-graphs. Euler paths. (10min) Problem solving with graphs. Reading: DMC Chapter 11 Slides: (full) (handout) (compact) Full video on youtube (88min) |
|
|
12: Graphs II, Matching and Coloring. An important skill is to take a real world situation and represent it as a graph problem. Master this approach and you have arrived. We will consider several examples from resource allocation to dating to conflict resolution. Modules: (02min) Recap of graphs and outline for today. (36min) Bipartite matching. Sex in America. Hall's Theorem. (19min) Dating and stable marriage. Be the aggressor. (13min) Coloring and conflicts: radio frequencies/exams. (12min) Greedy coloring. Trees are 2-colorable. (11min) Connected components. Euler/Hamiltonian paths. Travelling salesperson. Flow. Min. spanning tree. Vertex cover. Dominating set. Facility location. Reading: DMC Chapter 12 Slides: (full) (handout) (compact) Full video on youtube (93min) |
|
PAUSE. Reinforce and Practice: Assignment 5.
| |
|
|
13: Counting. Counting is subtle but crucial in algorithms, e.g. to count the operations. Master the two basic tools and you can count anything. Nevertheless, it is useful to have quick handy tools and systematic approaches, because key basic scenarios recur often. Modules: (05min) Recap of graphs. Let's count goody-bags. (27min) Basics: menus; NY-plates; races; passwds; subsets. (06min) Some challenges in NLP and scheduling exams. (25min) Build-up: bin. strings; Pascal's triangle; goody-bags. (16min) Bijection: subsets, goody-bags and bin. strings. (20min) Permutations & combinations. Binomial Theorem. Reading: DMC Chapter 13 Slides: (full) (handout) (compact) Full video on youtube (98min) |
|
|
14: Advanced counting. We extend the basic tools to more powerful tools for repetition and counting overlapping sets. Surprisingly, counting also gives a powerful tool for non-constructive existence proofs, and we will see these tools in full flight in the theory of computing. Modules: (03min) Recap of counting basics and the plan for today. (28min) Anagrams and repetition. A cheat sheet. (29min) Overlap and inclusion-exclusion with examples. (05min) Non-constructive proof, I-spy and the black cat. (15min) Pigeonhole principle. Social twins must exist! (10min) The $100 matching subset problem is solvable! Reading: DMC Chapter 14 Slides: (full) (handout) (compact) Full video on youtube (91min) |
|
PAUSE. Reinforce and Practice: Assignment 6.
| |
|
MIDTERM.
Assess: make a double sided 8.5x11 crib sheet and take the MIDTERM Lectures 1-14 DMC Chapters 1-14 Assignments 1-6 |
MIDTERM Fall 2020
(solution)
MIDTERM Fall 2019 (solution) MIDTERM Fall 2017 (solution) MIDTERM Fall 2016 (solution) MIDTERM Fall 2015 (solution) MIDTERM Fall 2014 (solution) |
|
|
15: Probability. Real algorithms operate in a random world. We even inject randomness into algorithms to make them go. How do we analyze such randomized algorithms in a random world? Modules: (13min) What is probability: weather; coin flips; gambling? (34min) Computing a probability: coins and dice. (24min) Practice. The Monty Hall Game; non-transitive dice. (05min) Probability space: probability and sets. (11min) Uniform probability space. (08min) Infinite experiments. Flip a coin until a heads. Reading: DMC Chapter 15 Slides: (full) (handout) (compact) Full video on youtube (95min) |
|
|
16: Conditional Probability. When new information arrives, you have to update your probabilities. Conditional probability also leads us to a powerful tool for computing regular probabilities, the Law of total probability. Modules: (11min) What happens when you get new information? (16min) Conditional probability: updating the probability. (15min) Examples: Clouds and rain. Dice. Summary. (12min) Monty prefers door 3. The chances of two boys. (19min) Poll bias, transposed conditional and med. tests. (20min) Law of total probability. Fair coin from biased. Reading: DMC Chapter 16 Slides: (full) (handout) (compact) Full video on youtube (94min) |
|
|
17: Independence. For unrelated events you can multiply probabilities, which can considerably simplify computing probabilities. We'll see examples in estimation, birthday coincidences, hashing, random walks and gambling. Modules: (20min) Defining independence. An example with coins. (14min) Estimation (Fermi method): dateable partners. (19min) Coincidence, FOCS-twins and the birthday paradox. (18min) Application to hashing. (16min) Random walks and gambler's ruin. Reading: DMC Chapter 17 Slides: (full) (handout) (compact) Full video on youtube (87min) |
|
|
18: Random Variables. In complex experiments, we aren't really interested in the complex outcome. Rather we are interested in some property or measurement of the outcome - the observable. Random variables are not random or variables, however, they are instrumental in simplifying complex experiments to the measurable quantities we want to focus on. Modules: (16min) Measurements and random variables. (24min) Probability distribution function (PDF). (15min) Bernoulli and Uniform random variables. (24min) Binomial: number of successes in n trials. (12min) Exponential waiting time to success. Reading: DMC Chapter 18 Slides: (full) (handout) (compact) Full video on youtube (95 min) |
|
PAUSE. Reinforce and Practice: Assignment 7.
| |
|
|
19: Expected Value. Can we summarize a complex experiment by a single number that captures the essence of the random outcome. We are after what happens "on average". How do we update this summary when new information arrives? Modules: (35min) Defining and interpretting the expected value. (04min) Expectations of a Bernoulli and the Uniform. (15min) Expected number of successes: Binomial. (07min) Expected waiting time. (16min) Conditional expectation & total expectation. (13min) Total expectation and expected waiting time. Reading: DMC Chapter 19 Slides: (full) (handout) (compact) Full video on youtube (90min) |
|
|
20: Expected Value of a Sum. Complex outcomes can often be represented as a sum of simpler outcomes. In such cases, computing expected values simplifies. We develop powerful tools for expectations that leverages sums. One such tool is a sum of indicators. Modules: (26min) Two lottery tickets and linearity of expectation. (04min) Expected sum of dice. (03min) Number of successes in n trials (Binomial). (10min) Waiting time to k successes (Exponential). (16min) Coupon collecting. How long to get all coupons? (05min) Expected value of a product. (11min) Build-up expectation. Wait to 3-boys, 3-girls. (18min) Indicators, derangements and record breaking. Reading: DMC Chapter 20 Slides: (full) (handout) (compact) Full video on youtube (92min) |
|
|
21: Deviation from the Mean. The expected value (mean) summarizes the random outcome of a complex experiment. How good a summary is it? To quantify the quality of the expected value, we introduce variance and derive the 3-sigma rule. Modules: (09min) The mean versus the actual outcome. (18min) Defining the variance. (11min) An easier way to compute variance (21min) Linearity of variance. (15min) Markov & Chebyshev Inequalities. 3-sigma rule. Reading: DMC Chapter 21 Slides: (full) (handout) (compact) Full video on youtube (min) |
|
PAUSE. Reinforce and Practice: Assignment 8.
| |
|
PAUSE.
Assess: Take Quiz 2
|
Fall 2020
(solution)
Fall 2019 (solution) Fall 2017 (solution) Fall 2016 (solution) Fall 2015 (solution) |
Ⅱ. Theory of Computing
|
|
22:
Infinity. To step into the abyss of Infinity, we must relinquish the absolute size of sets and resort to comparing sets. What is countable and what is uncountable, and what does it have to do with computing? The connection is stunning: Infinity says there are many things we cannot compute. Modules: (07min) Recap of discrete math and probability. (20min) Counting versus comparing: Cantors great insight. (24min) Countable. Integers are countable. (17min) The list-method: rationals are countable. (04min) Finite binary strings (programs) are countable. (21min) Infinite binary strings (computing problems) are uncountable. Cantor diagonalization. Reading: DMC Chapter 22 Slides: (full) (handout) (compact) Full video on youtube (93min) |
|
|
23: Languages. What Is Computing? Someone on the street asks you, "What is a computing problem?" What do you say? Decision problems have withstood the test of time and can be defined by a set of finite binary strings. The complexity of a problem is the complexity of its language. Modules: (34min) Computing problem = decision problem = YES-set. (13min) Language = computing problem. (25min) Regular expressions (vs. recursive definitions). (07min) How hard is a computing problem? (14min) Complexity: a simple state transitioning machine. Reading: DMC Chapter 23 Slides: (full) (handout) (compact) Full video on youtube (93min) |
|
|
24:
Deterministic Finite Automata (DFA). Let us build our first baby computing device. A simple state transitioning machine that is easy to build mechanically. What can it solve? What can't it solve? Modules: (13min) Practice with regular expressions. (27min) Basics of Automata: Finite State Machines. (16min) Constructing a DFA to solve a problem (language). (21min) DFA and set operations. (06min) DFA solves regular expressions. (19min) DFA can't solve equality! How to fix it? Reading: DMC Chapter 24 Slides: (full) (handout) (compact) Full video on youtube (103min) |
|
PAUSE. Reinforce and Practice: Assignment 9.
| |
|
|
25: Context Free Grammers (CFG). Being able to generate the strings in a language is also useful. CFGs are more powerful than DFA and have many uses in computer science, e.g. parsers and compilers. What are the capabilities of grammars? Modules: (08min) DFA with stack memory solves equality. (28min) CFGs for equality and balanced strings. (19min) CFGs for baby English and baby programing. (10min) Finding a CFG for a language: balanced strings. (11min) Power of CFGs: regular operations and more. (11min) Parse trees and interpretting strings. (09min) CFG cannot solve repetition, triple equality, ... Reading: DMC Chapter 25 Slides: (full) (handout) (compact) Full video on youtube (96min) |
|
|
26: Turing Machines. The! model of computation that captures the intuitive notion of an algorithm, the Church-Turing Thesis. With power comes pitfalls, and we know them all too well: infinite looping programs. This leads us to the distinction between deciders versus recognizers. Modules: (46min) Building a Turing Machine for repetition. (18min) Multiplication. Deciders versus transducers. (15min) Infinite loops. Deciders versus recognizers. (08min) Turing Machine Encodings. TMs can be listed. Reading: DMC Chapter 26 Slides: (full) (handout) (compact) Full video on youtube (87min) |
|
|
27: Unsolvable Problems. The Turing Machine (TM) is incredibly powerful. It is the computer we know and love, and even more. Is there anything we cannot solve? We'll examine a variety of love-to-have tools from auto-graders (program verification) to ultimate-debuggers, and also the domino challenge problem (PCP). The punchline is that we can't solve these problems, and we'll have to live with that. Modules: (12min) Deciders, recognizers and TM-sketches. (19min) Programmable Turing Machines: the computer. (45min) Unsolvable: PCP, halting and verification. (06min) The state of affairs in computability. Reading: DMC Chapter 27 Slides: (full) (handout) (compact) Full video on youtube (82min) |
|
PAUSE. Reinforce and Practice: Assignment 10.
| |
|
PAUSE.
Assess: Take Quiz 3
|
Fall 2020
(solution)
Fall 2019 (solution) Fall 2017 (solution) Fall 2016 (solution) Fall 2015 (solution) |
|
|
28: Efficiency, P and NP. The Fast (P) the Slow (EXP) and the verifiable (NP). Fast means polynomial and it's architecture independent. We'll touch on the interplay between solvable and verifiable. The details involve circuits, 3-SAT (a hardest verifiable problem) and NP-completeness (chapters 28 & 29). Modules: (07min) What we did and the places you'll go. (16min) Fast vs. slow is architecture independent. (03min) Fast, slow and the fuzzy region NP. (05min) The fuzzy region NP: solutions are easy to verify. (04min) Verifiabile fast equals nondeterministic polynomial. (06min) Hardest NP-problems: NP-completeness. (06min) Prove your program is fully correct all the time. Reading: DMC Chapter 28 & 29 Slides: (full) (handout) (compact) Full video on youtube (47min) | ||
|
FINAL Fall 2020
(solution)
FINAL Fall 2019 (solution) FINAL Fall 2017 (solution) FINAL Fall 2016 (solution) FINAL Fall 2015 (solution) FINAL Fall 2014 (solution) | ||
