Class 28: Entropy

A giant python swallows an alligator in Everglades National Park, Fla..
Paul Zenk/PBS Nature “Invasion of the Giant Pythons” (yesterday’s New York Times)

cs1120 Fall 2011
David Evans
Class 28:
28 October 2011 Entropy

Plan
Story so Far: Liberal Arts Recap
Reversing in Python
Shannon’s Information Theory: Entropy
Returning Quiz 3, PS5, etc.

2

Ch 1: Computing
Ch 2: Language

Analysis
Synthesis

Ch 3: Programming

Ch 4: Procedures

Ch 5: Data
Ch 6: Machines

Ch 7: Cost

Ch 8: Sorting and Searching
PS5, Ch 9: State
You are PS6, Ch 10: Objects
Course Roadmap

here
PS7, Ch 11: Interpreters

Ch 12: Computability
PS8, 9
Intractability

cs1120 Main Topics
Language: How to describe information processes by
defining procedures (Chapters 3, 4, 5, 9, 10)
Programming with procedures, lists, recursion (PS1-4)
Programming with state (PS5), objects (PS6), languages (PS7)
Logic: How to predict properties about information
processes (Chapter 6, 7, 8)
Predicting how running time grows with input size
Are there problems which can’t be solved by algorithms? (Ch 12)
Machines: How to efficiently implement information
processes (Chapter 6) (not much on this)
Machines, Digital Logic
How to implement a Scheme interpreter (Ch 11, PS7)

Grammar: study of meaning in BNF, RTN, rules of
written expression evaluation for meaning
Trivium

Rhetoric: comprehension of verbal Interfaces between
components
and written discourse (PS6), evaluation rules

Logic: argumentative discourse for Rules of evaluation, if,
recursive definitions
discovering truth
Arithmetic: understanding adding with logic,
representing numbers
numbers
Quadrivium

Curves as procedures,
Geometry: quantification of space fractals (PS3)

Music: number in time Ada, GEB Chapter

Astronomy Nothing!
But…read the Neil DeGrasse Tyson essay!

Lists (Tuples) in Python
Scheme (list) Python (tuple)
> (define p (list 1 2 3)) >>> p = (1, 2, 3)
> (car p) >>> p[0]
1 1
> (cdr p) >>> p[1:]
(2 3) (2, 3)
> (cdr (cdr (cdr p))) >>> p[1:][1:][1:]
null ()

7

Python Shortcuts
>>> p = (1, 2, 3)
>>> p[2]
3 list[i]
>>> p[3:] Evaluates to ith (counting from
0) element of tuple/list
()
Runs in (approximately)
>>> p[:2] constant time!
(1, 2)

Mutable Lists (Lists) in Python
Scheme (mutable list) Python [List]
> (define p (mlist 1 2 3)) >>> p = [1, 2, 3]
> (mcar p) >>> p[0]
1 1
> (mcdr p) >>> p[1:]
(2 3) (2, 3)
> (mcdr (mcdr (mcdr p))) >>> p[1:][1:][1:]
null ()

9

Using Mutable Lists
>>> p = [1, 2, 3] Scheme Application:
>>> p[0] (<verb> <operands>)
1 Python procedure call:
>>> p[0] = 2 <verb>(<operand>)
>>> p Method call:
[2, 2, 3] <object>.<verb>(<parameters>)
>>> p.append(4) Note: this is different from mappend!
>>> p It take a single element, not a list.
[2, 2, 3, 4]

Trick Question

>>> p = (1, 2, 3)
>>> p[0] = 2
Traceback (most recent call last):
File "<pyshell#23>", line 1, in <module>
p[0] = 2
TypeError: 'tuple' object does not
support item assignment

Reversing a Mutable List

Scheme: p:

1 2 3

Python:
p:

1 2 3

mlist-reverse!
(define (mlist-reverse! p)
(if (null? (mcdr p))
(void)
(let ((rest (mcdr p))
(carp (mcar p)))
(mlist-reverse! rest)
(set-mcar! p (mcar rest))
(set-mcdr! p (mcdr rest))
(mlist-append! p (mcons carp null)))))

Literal Translation
(define (mlist-reverse! p)
(if (null? (mcdr p))
(void)
(carp (mcar p)))
(mlist-reverse! rest)
(set-mcar! p (mcar rest))
(set-mcdr! p (mcdr rest))
(mlist-append!
p
(mcons carp null)))))

Literal Translation
(define (mlist-reverse! p) def sreverse(p):
(if (null? (mcdr p)) if len(p) == 1:
(void)
return
(carp (mcar p))) else:
(mlist-reverse! rest) rest = p[1:]
(set-mcar! p (mcar rest)) first = p[0]
(set-mcdr! p (mcdr rest)) sreverse(rest)
(mlist-append!
p[0] = rest[0]
p
(mcons carp null))))) p[1:] = rest[1:]
p.append(first)
>>> p=[1,2,3]
This is correct, but no sane Python >>> sreverse(p)
programmer would do it this way! >>> p
[3, 2, 1]

Python for
Statement ::= (define (loop index result test update proc)
for Varible in Expression: (if (test index)
(loop (update index)
Block (proc index result)
test update proc)
result))
def gaussSum (n):
sum = 0 (define (gauss-sum n)
(loop 1 0 (lambda (i) (<= i n))
for i in range(1, n+1): (lambda (i) (+ i 1))
(lambda (i sum) (+ i sum))))
sum = sum + i
return sum

Reverse in Python
p:

1 2 3

Note: there is a built-in list method, p.reverse().

int(len(p) / 2) len(p) >> 1
def reverse(p):
for i in range(0, len(p) >> 1):
p[i], p[len(p) – 1 - i] = p[len(p) – 1 - i], p[i]

Information Theory

19

Entropy

H = - pi log2 pi
20

flickr cc:Daniel Dale

(Ideal) Coin Toss

H = - pi log2 pi

21

Nucleotide
N {A, C, G, T}
Intuition: need two bits to distinguish four possible values
00 = A, 01 = C, 10 = G, 11 = T

H = - pi log2 pi

22

Real
Coin
Toss?

Claude Shannon’s
Juggling W. C. Fields
24

Real Coin Toss
According to Susan Holmes, 51% of the time, a
coin will land same way it started. [link]

H = - pi log2 pi

25

Are Randall Munroe’s entropy estimates correct?
Left for course blog discussion.
27

0.9880135796356116381290864023063602427958811193759308857790...

Class Sorting using “Odette”
Assuming alphabet
randomly
distributed: O is
15th out of 26
letters in alphabet

= 0.988…
28

0.9880135796356116381290864023063602427958811193759308857790...

Class Sorting using “Odette”
The real class: 41 out
of 49 students have
names before
“Odette”!

-((41/49) * log_2 (41/49) + (7/49) * log_2 (7/49))
= 0.6162235892609252

29

Better Class Sorting
Front of Class

crm Caroline Hannah – Li-
- Gregory Chang
Anthony - mdc Michael
Casey - Yuan
30

Charge
Office hours:
Peter – right now
Sunday, 1-6pm
Everyone should be making progress on PS6
Monday:
Trick-or-Treat Protocols!
Inheritance

Class 28: Entropy

More Related Content

What's hot (20)

Similar to Class 28: Entropy (20)

More from David Evans (20)

Recently uploaded (20)

Class 28: Entropy