Due on Oct 19 (60 points)

1. Use the two coin setup from Q.3 of the last HW. Denote \(X_i\) as the output of the \(i\)th toss and \(C\) as actual coin type. Compute

   1. \(H(X_1|X_2)\)

   2. \(H(X_1|C)\)

   3. \(H(C|X_1,X_2)\)

   4. \(I(X_1;X_2)\)

   5. \(I(X_1;X_2|C)\)

2. Consider a jar with three types of dices. Dice A is fair. Dice B has probabilities \(0.1,0.1,0.1,0.1,0.3,\) and \(0.3\) for outcomes \(1,2,3,4,5,\) and \(6\), respectively. And Dice C has probabilities \(0.3,0.3,0.1,0.1,0.1,\) and \(0.1\) for outcomes \(1,2,3,4,5,\) and \(6\), respectively. Assume that we have 2 A-dices, 3 B-dices, and 5 C-dices in the jar, and we drew one dice from the jar and threw it three times and getting all ones. What is the estimated probability of getting another one in the next toss if we use a) MLE, b) MAP, and c) Bayesian estimation?

3. Continue with the earlier question. Denote \(D\) as the actual dice we drew, and \(X_1, X_2, X_3\) are the respective outcomes of three tosses. Compute

   1. \(H(X_1)\)

   2. \(H(X_1,X_2)\)

   3. \(H(X_1|D)\)

   4. \(H(D|X_1)\)

   5. \(H(D|X_1,X_2)\)

   6. \(I(X_1;X_2)\)

   7. \(I(X_1,X_2;X_3)\)

   8. \(I(X_1;X_2|D)\)

   9. \(I(X_1;D)\)

   10. \(I(X_1;D|X_2)\)

4. Consider sequences of \(10,000\) coin-flips with the probability of head equal to \(0.8\). Show that a sequence with \(8,000\) heads and \(2,000\) tails is typical.

5. For a group of students with weight \(W\) and height \(H\) are jointly normally distributed with mean of 50 kg and 160 cm and a covariance matrices of \(E\left[\begin{matrix} (W-\bar{W})^2 & (W-\bar{W})(H-\bar{H}) \\ (H-\bar{H})(W-\bar{W}) & (H-\bar{H})^2 \end{matrix} \right]=\begin{pmatrix} 80 & 30 \\ 30 & 140 \end{pmatrix}\).

   1. Compute \(h(W)\)

   2. Compute \(h(W|H)\)

   3. how many bits on average will be needed per sample to store a student's weight with precision of 0.1 if his height is known?