Information Theory

Syllabus

This course will set out the fundamental concepts of information theory. Expressions for the information generated by discrete memoryless sources and sources with memory will be established and the lossless source coding theorem will be proved and the asymptotic equipartition theorem will be presented. The practical significance of the source coding theorem will be examined and examples of source coding will be given. The concept of channel capacity will be introduced and the calculation of the capacity of important communication channels and systems will be dealt with. The capacity theorem will be proved for various cases and its practical significance will be examined, and simple examples of coding aimed at achieving the results promised by the capacity theorem will be outlined. The concept of source coding, subject to fidelity criteria (rate distortion theory) will be introduced. Finally, the basics of network information theory will be given, which is presently the most dynamic research area in information theory.

Textbooks

• Course textbook: Elements of Information Theory, T. M. Cover and J. A. Thomas, Wiley.
• Reference: Information Theory and Network Coding, Raymond W. Yeung, Springer, 2008. Free at http://iest2.ie.cuhk.edu.hk/~whyeung/book2/
• Reference: Lecture Notes on Network Information Theory, A. E. Gamal and Y.-H. Kim. http://arxiv.org/abs/1001.3404
• Reference: Information Theory, Inference, and Learning Algorithms, D MacKay, CUP, or free at http://www.inference.phy.cam.ac.uk/mackay/itila/
• Shannon’s original paper: A mathematical theory of communication, Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948.

Lecture Slides

Slides

Problem Sheets

Problems

Past Exam Papers

• 2006
• 2007
• 2008
• 2009
• 2010
• 2011

Frequently Asked Questions

• Is topic “xyz” examinable? All topics included in the above lectures are examinable with the exception of the Kuhn Tucker conditions for constrained optimization.
• Do I need to remember long complicated proofs? You will not be asked to reproduce long proofs from the notes. However you will be expected to understand the proofs and to justify the steps they take.
• Do I need to remember long formulas? In general you are not expected to memorize long complicated formulas. However you will be expected to remember some basic and important formulas.