HW3 (Due 10/31)

(5 points) For a discrete variable with more than two outcomes, the Beta prior can be generalized to the Dirchlet prior (please also see slides 87-91). Assuming a Dirchlet prior of \(Dir(\alpha_1=3,\alpha_2=3,\alpha_3=3,\alpha_4=1,\alpha_5=1,\alpha_6=1)\), compute the a) ML, b) MAP, and c) Bayesian estimate of the probability of the third toss of a dice being one when the first two tosses are also ones.

(5 points) Consider two distributions \(p=[0.199, 0.599, 0.199, 0.001, 0.001, 0.001]\) and \(q=[0.1, 0.3, 0.1, 0.1, 0.3, 0.1]\). Compute \(KL(p||q)\) and \(KL(q||p)\). Note that they are significantly different from one another. You may also want to check out this tweet to get some more insight.

(5 points) For a signal with maximum variance of \(5^2\), if we want to store the signal with the precision of 1 decimal place, how many bits will be needed (without any other information)?

(5 points) Use the tropical island example in HW1 again. Let's denote \(Q\) as a random variable that \(Q=1\) if \(T > T_{th}\) and \(Q=0\) otherwise.

1. Compute \(H(W|Q)\) when \(T_{th}=25.5\)

2. Compute \(H(W|Q)\) when \(T_{th}=26.5\)

Extra credit. Write a program to compute the best threshold \(T_{th}\) so as to design a decision tree for the tropical island problem.

1. (10 points) Assume that we sampled the tropical island for 10 days and it is sunny for the first 8 days and rainy for the last 2 days. And the temperatures are 28,27,28,29,33,34,27,26,27,22. Find the best \(T_{th}\) that minimizes \(H(W|Q)\) for the given samples. Please submit your source code in pdf and a screenshot running your code.

2. (5 points) Simulate the tropical island by generating 10,000 samples. Estimate \(H(W|Q)\) with the given samples for \(T_{th}=25.5\) and \(T_{th}=26.5\)

3. (5 points) Find the optimum threshold for your samples.