Part 2: Systems are maps #
[This is Part II of my earlier post on signals as vectors]
If there is one sentence that I think is most enlightening and important to understand coming out of a signal processing course, it’s
Fourier transforms are useful to signal processing because complex exponentials are eigenfunctions of Linear Time Invariant systems, and Fourier transforms transform signal vectors into a complex exponential basis, diagonalising convolutions.
My hope in this post is to help you understand that sentence. Along the way we’ll talk about complex exponentials, eigenvectors and eigenfunctions, and hopefully build some geometric intuition for what frequency transforms are doing and why they’re useful. The broader goal is to recognise that not only are signals vectors, but our “systems” are maps, like matrices that map from one vector to another, and that we can exploit that structure to make understanding systems much simpler.
Systems as linear operators #
In Part 1 we established that signals/functions are vectors. Now, a system takes an input signal and produces an output signal. From a linear algebra perspective, this is just a map from one vector space to another. In finite dimensions, a linear map is a matrix. In function space, we call it a linear operator.
For a system \(L\) to be a linear operator, it just needs to obey superposition and homogeneity (same conditions as in Part 1). That’s the “linear” in Linear Time Invariant (LTI). The “Time Invariant” part means a time shift in the input produces the same time shift in the output. Shift the input by \(\tau\) seconds and the output shifts by \(\tau\) seconds too, nothing else changes:
$$ \text{if } L{f(t)} = g(t) \text{, then } L{f(t - \tau)} = g(t - \tau) $$
These two properties together, linearity and time invariance, turn out to be very constraining. It means the only operation an LTI system can perform on an input signal is convolution with the system’s impulse response \(h(t)\):
$$ g(t) = (h * f)(t) = \int_{-\infty}^{\infty} h(\tau) f(t - \tau) d\tau $$
If you haven’t seen convolution before, don’t worry too much about the integral, just know that it’s the function-space analogue of multiplying a vector by a matrix. The impulse response \(h(t)\) is to an LTI system what a matrix is to a finite-dimensional linear transformation: it completely characterises how the system transforms any input.
Eigenvectors, refreshed #
Before we go further, let’s remind ourselves of one of the most important ideas in linear algebra: eigenvectors.
Given a matrix \(A\), a vector \(\vec v\) is an eigenvector if multiplying by \(A\) only scales it, without rotating it or mixing it with other directions:
$$ A \vec{v} = \lambda \vec{v} $$
The scalar \(\lambda\) is the corresponding eigenvalue. Geometrically, eigenvectors are the “special” directions in a transformation where the matrix just stretches or squishes, that’s it. Every other vector gets some mixture of rotated and scaled.
Why does this matter? Because if you could express any input \(\vec x\) as a sum of eigenvectors \(\vec v_i\), then applying the transformation becomes easy:
$$ A\vec{x} = A \sum_i c_i \vec v_i = \sum_i c_i \lambda_i \vec v_i $$
Each component just gets multiplied by its own eigenvalue. You’ve turned a matrix multiplication into a bunch of independent scalings. This is the core idea behind diagonalisation: if you write your matrix in an eigenvector basis, it becomes a diagonal matrix. Diagonal matrices are about as simple as it gets.
Eigenfunctions #
Since functions are vectors and LTI systems are linear operators (maps), we can ask: do LTI systems have “eigenvectors”? In function space, these are called eigenfunctions.
A function \(\phi(t)\) is an eigenfunction of an operator \(L\) if:
$$ L{\phi(t)} = \lambda \cdot \phi(t) $$
for some scalar \(\lambda\). Just like eigenvectors, eigenfunctions are signals that pass through the system with their “shape” completely intact, only scaled by \(\lambda\).
So: what are the eigenfunctions of LTI systems?
Complex exponentials are eigenfunctions of LTI systems #
Let’s take an LTI system with impulse response \(h(t)\) and feed in a complex exponential \(e^{st}\), where \(s\) is some complex number. The output is the convolution:
$$ g(t) = (h * e^{st})(t) = \int_{-\infty}^{\infty} h(\tau) e^{s(t - \tau)} d\tau $$
Expanding \(e^{s(t-\tau)}\):
$$ = \int_{-\infty}^{\infty} h(\tau) e^{st} e^{-s\tau} d\tau $$
Since \(e^{st}\) doesn’t depend on \(\tau\), we can pull it out of the integral:
$$ = e^{st} \int_{-\infty}^{\infty} h(\tau) e^{-s\tau} d\tau $$
That integral is just a number, let’s call it \(H(s)\):
$$ H(s) = \int_{-\infty}^{\infty} h(\tau) e^{-s\tau} d\tau $$
So we end up with:
$$ g(t) = H(s) \cdot e^{st} $$
How cool is that! The output is just the input \(e^{st}\), scaled by \(H(s)\). The complex exponential came out the other side as the same complex exponential, just multiplied by a constant. That’s the eigenfunction condition \(L{\phi} = \lambda \phi\), with \(\phi(t) = e^{st}\) and \(\lambda = H(s)\).
And if \(s = j\omega\) (i.e. \(s\) is purely imaginary), this is just a sinusoid, which is why people often explain it as “sinusoids pass through LTI systems unchanged in frequency, only gaining an amplitude and phase change.” That explanation is correct but incomplete. It works because sinusoids are a special case of complex exponentials, and complex exponentials are eigenfunctions of all LTI systems.
Also notice that the number \(H(s)\) we pulled out of the integral is exactly the Laplace transform of \(h(t)\). If \(s = j\omega\), it’s the Fourier transform of \(h(t)\).
Frequency transforms as a basis change #
The Fourier transform projects \(f(t)\) onto each complex exponential basis function:
$$ F(\omega) = \int_{-\infty}^{\infty} f(t) e^{-j\omega t} dt $$
with the inverse reassembling the signal:
$$ f(t) = \frac{1}{2\pi}\int_{-\infty}^{\infty} F(\omega) e^{j\omega t} d\omega $$
This is a basis change from the time basis into the complex exponential basis. In fact, if you use the unitary versions of our transforms, they’re just rotations!
Diagonalising convolution #
In time domain, the output of an LTI system is a convolution:
$$ g(t) = (h * f)(t) $$
Convolution is a complicated integral operation. But we just showed that complex exponentials are eigenfunctions of LTI systems. So if we work in the complex exponential basis (i.e. take the Fourier transform), convolution becomes pointwise multiplication:
$$ G(\omega) = H(\omega) \cdot F(\omega) $$
Convolution in time \(\longleftrightarrow\) multiplication in frequency. This is probably the most important identity in signal processing, and it falls straight out of the eigenfunction argument: every \(\omega\) component of the input gets scaled by \(H(\omega)\) independently.
The Fourier transform diagonalises convolution. Just like \(A = P D P^{-1}\) where \(D\) is diagonal and \(P\) is the eigenvector matrix, we have
$$ h * f = \mathcal{F}^{-1}{ H(\omega) \cdot F(\omega) } $$
where \(\mathcal{F}\) and \(\mathcal{F}^{-1}\) are the basis change (Fourier transform) and its inverse, and the multiplication \(H(\omega) \cdot F(\omega)\) in the middle is the diagonal operation.
So why are frequency transforms useful to signal processing? Not because “it converts time domain to frequency domain” (which is a tautology and not useful on its own), but because complex exponentials happen to be eigenfunctions of LTI systems, so the Fourier transform is a basis change into the eigenfunctions, and in that basis the action of any LTI system is just pointwise scaling. That’s why frequency-domain analysis works so well.
Hopefully the opening sentence makes a bit more sense now:
Fourier transforms are useful to signal processing because complex exponentials are eigenfunctions of Linear Time Invariant systems, and Fourier transforms transform signal vectors into a complex exponential basis, diagonalising convolutions.