Information Theory

• Instructor: Prof. Dr.-Ing. Werner Henkel

Information theory serves as the most important foundation for communication systems. The course provides an analytical framework for modeling and evaluating point-to-point and multi-point communication. After a short rehearsal of probability and random variables and some excursion to random number generation, the key concept of information content of a signal source and information capacity of a transmission medium are precisely defined, and their relationships to data compression algorithms and error control codes are examined in detail. The course aims to install an appreciation for the fundamental capabilities and limitations of information transmission schemes and to provide the mathematical tools for applying these ideas to a broad class of communications systems. Aside from source and channel aspects, an introduction to security is given, including publickey cryptography. Information theory is standard in every communications-oriented Bachelor’s program.

Lecture at Jacobs University [Campusnet link].

• Introduction (sources and channels, probabilities and densities, information, entropy)
• Entropy and mutual information (chain rule, Jensen’s inequality, Fano’s inequality)
• Source coding principles (Kraft’s inequality, optimum codes, Huffman coding, Shannon-Fano codes)
• Practical lossless source coding algorithms (Lempel-Ziv, Arithmetic Coding, PPM, runlength encoding, move-to-front transform, Burrows-Wheeler transform, Context Tree Weighting)
• The discrete channel (channel capacity and channel coding theorem)
• The Gaussian and bandlimited channel (the single and multiple Gaussian channel and its capacities, water filling, capacities for modulation alphabets)
• Random coding bound (error exponent, cutoff rate)
• Rate-distortion theory
• Multiple access and MIMO (capacity region, multiple access, broadcast, relay, capacity of MIMO channels)
• Short introduction to Channel Coding (finite (Galois) fields, linear block codes, Reed-Solomon codes, convolutional codes, Turbo and LDPC codes)
• Cryptology (encryption and authentication, secrecy capacity, wiretap channel, perfect, weak and strong secrecy, one-time pad, public-key encryption)