At the end of this lesson, students are expected to:
In this module, we will look at machine learning through the perspective of optimization. Why? Because since its earliest days, ML (or a sizeable part of it) has relied on optimization methods to find good solutions.
So, what is the relationship between ML and optimization? Are they the same thing? Is one a part of the other? Short answer: it is hard to define exactly because ML is a fast evolving field. But I think a reasonably good answer is that they are two separate fields with a high degree of overlap.
I drew a heart at the intersection, which you can interpret in two ways: 1) optimization is at the heart of ML — a great many ML methods use optimization techniques to solve problems. 2) There is a great deal of exchange — ML has driven the field of optimization forward, posing new challenges as it becomes more advanced and consumes more and more data.
Speaking of data, the inclusion of data into our modeling is an important distinction between Machine Learning and Decision Theory.
One way to describe machine learning is as a statistical approach to function approximation. That is, ML approximates some underlying real-world function that maps input examples to some output (e.g. photo inputs that should be classified as dogs or cats). If we succeed, this reusable mapping function can generalize from specific examples to make predictions on new data.
