Universal approximator

Bounded depth and bounded width case

Here "${\displaystyle \sigma \colon \mathbb {R} \to \mathbb {R} }$ is ${\displaystyle \lambda }$-strictly increasing on some set ${\displaystyle X}$" means that there exists a strictly increasing function ${\displaystyle u\colon X\to \mathbb {R} }$ such that ${\displaystyle |\sigma (x)-u(x)|\leq \lambda }$ for all ${\displaystyle x\in X}$. Clearly, a ${\displaystyle \lambda }$-increasing function behaves like a usual increasing function as ${\displaystyle \lambda }$ gets small. In the "depth-width" terminology, the above theorem says that for certain activation functions depth-${\displaystyle 2}$ width-${\displaystyle 2}$ networks are universal approximators for univariate functions and depth-${\displaystyle 3}$ width-${\displaystyle (2d+2)}$ networks are universal approximators for ${\displaystyle d}$-variable functions (${\displaystyle d>1}$).