Information Theory & Coding

Analog Signaling x(t +  ) a  x(t ) inputs are functions of time xˆ (t +  ) a  xˆ (t ) xˆ (t ) + yˆ (t ) linear x(t ) + y (t ) A linear time-invariant system is characterized by the eigenfunctions eiwt w = 2f f = frequency t = time i fix w Ae = e ln A e i iwt phase  gain iwt i iwt → e = Ae  e e = e ln A+ i =e i ( − i ln A+ ) =e A(w) = amplitude response – limits bandwidth Θ(w) = phase shift – distorts pulse shape iF ( w ) transfer function limits signaling capacity Sampling Theorem: A band-limited signal of duration T and of bandwidth W can be reconstructed perfectly by 2WT samples, at evenly spaced intervals. The sample vector (x1, …, x2WT) can be viewed as a point in 2WT-dimensional space. A.9

These samples can tell us the signal’s energy and their radius 2W T (distance from origin) 1 2W T 2 2 E= x r = x 2WE  n  n = 2W n =1 n =1 And the signal power (energy per unit time) is S = E  r = 2WST T The noise added to the channel has power N, and a corresponding radius 2WNT . The total power (signal + noise) has radius 2WT (S + N ) . How many spheres of noise can fit in? [2WT ( S + N )] 1 2W T 2 S+N =   N  WT WT S  # of M  = 1 +  . 1 messages 2W T  N 2 (2WTN ) signal- to- noise ratio        ratio of volumes S  I = log M = WT log 1 +  , The amount of information sent is  N I S  And the rate of info is = W log 1 +  E.g. (telephone) T  N Appendix (end.)

Information & Coding Theory Channel (errors) Information Symbols Source s1,…,sq signal Encoding Source/Channel  Information Symbols Destinsignal Decoding + noise Channel/Source ation s1,…,sq Noise Example: Morse Code transmitter A, …, Z Encoding keyer telegraph wire dots, dashes spaces receiver ∙ ─ _ Decoding shortwave radio A, …, Z recognizer Example: ASCII Code Character keyboard seven-bit Telephone seven-bit terminal modem modem blocks wire blocks screen character

Information Source – the symbols are undefined, and the “meaning” of the information being sent is not dealt with – only an abstract measure of the “amount” or “quantity. Examples text of various forms – reports, papers, memos, books, scientific data (numbers) pictures of various forms – diagrams, art, photographic images, scientific data (e.g. from satellites) sound of various forms – music, speech, noises, recorded sound, radio animation of various forms – moving pictures, film, video tape, video camera, television equations representing mathematical ideas or algorithms – two textual representation systems with graphical output: Tex & Mathematica continuous analog waveforms and shapes discrete digital sampled and quantized