If I build a neural network with enough neurons, can it represent (or approximate) any function I care about?
That’s important because in machine learning we rarely know the true function that maps inputs (e.g., images, signals, data) to outputs (e.g., labels, predictions). We want a flexible model that can “mimic” whatever this unknown function might be.
Here’s the core intuition/mechanism:
- Linear functions are limited.
If you just stack linear operations, you only get other linear functions — you can’t represent curved or jagged shapes. - Nonlinear activation is the key.
Functions like the sigmoid or ReLU allow you to build “pieces” (like step functions or localized bumps). - Building blocks of functions.
- With sigmoids, you can approximate a step function.
- With differences of sigmoids, you can approximate a “bump.”
- By summing enough bumps with the right weights, you can approximate any curve, surface, or function (just like how Fourier series approximate functions using sines and cosines).
So the network becomes a basis function machine. The hidden neurons act like flexible “basis elements,” and with enough of them, you can cover any shape.