Table of contents

🎓 Intended learning outcomes

At the end of this lesson, the student is expected to:

• Be able to recall the steps involved in parametric density estimation problems
• Understand the maximum-likelihood principle and how it relates to the field of machine learning
• Demonstrate capability to find maximum-likelihood parameter estimates for simple distributions through mathematical derivations
• Know and understand the theoretical guarantees of maximum likelihood estimation, and be aware of their limitation in practical scenarios
• Define the maximum a posteriori approach to parameter estimation
• Understand the regularising role of prior distributions in maximum a posteriori parameter estimation
• Be aware of the advantages and disadvantages of maximum likelihood and maximum a posteriori parameter estimation

Preliminaries

<aside> 📢 Now would be a good time to quickly review the most common and basic distributions used in probability and statistics, including their multivariate versions. Wikipedia has a big list of probability distributions, many many more that you ever will need. You should be able to find a more compact list in your basic probability textbook.

</aside>

Notation refresher

Introduction

Suppose you toss a coin $n$ times and observe the outcomes $\mathcal{X} = \{\text x_1, \dots, \text x_n\}$, where $\textbf x_i=1$ means heads and $\textbf x_i=0$ means tails. How can we build a probabilistic ****model of this coin using the data? Formally, our goal is to construct a probability distribution represented by the probability density function $p_\mathbf{x}$ that assigns a probability to each of the two possibilities (heads or tails).

Distributions are defined by two things: their model family, and their parameter values.

We will assume that the coin flips follow a Bernoulli distribution: