News + Publications

The “No Free Lunch Theorem” is often referenced in the context of Machine Learning (ML) algorithms when people talk about there being no one ML model or architecture that performs best on all problems. While this is true, a key element that is often overlooked is what “all problems” really mean.

The original No Free Lunch Theorems state that all optimization problems perform equally well when their performance is averaged across the space of all possible objective functions. Or put another way if we have two algorithms (A and B) there are just as many prior distributions where A beats B and vice-versa.

For industrial applications, however, we are often only concerned with one class of problem and/or a specific type of data.

So, we don’t really care about all possible priors, but only a restricted set of priors defined by the particular problem and data at hand. In this restricted case, the No Free Lunch Theorems do not state any limitations. For these “specific” problems one can have algorithms and models that generally work better than others. This comes about when the biases of the particular model are well suited for the particular problem at hand (i.e., structures/patterns in the data).

A good example of this is how Convolutional Neural Networks perform well at image related tasks. This is due to the bias of convolutions to consider the locality of data (i.e., the proximity of one data point to another) and treat these data similarly, while having less connectivity between data that is far away. This is an advantageous bias for images, where image content is dominated by spatially local patterns. (Whereas for data where the position or ordering is not important, this is not a good bias to have.)

For most industrial ML problems this is where knowledge and experience, such as subject matter expertise, feature engineering, and selection of models/architectures – can be essential for rapid development and successful solutions.

Armed with such insights, clients may not quite get that free lunch, but certainly a hefty discount…