Entropy is a notoriously tricky subject. There is a famous anecdote of John von Neumann telling Claude Shannon, the father of information theory, to use the word “entropy” for the concept he had just invented, because “nobody knows what entropy really is, so in a debate you will always have the advantage“.

Entropy means many different things in different contexts, but there is a wonderful notion of entropy which is purely topological. It only requires a space, and a map on it. It is independent of geometry, or any other arbitrary features — it is a purely intrinsic concept. This notion, not surprisingly, is known as topological entropy.

There are a few equivalent definitions; we’ll just discuss one, which is not the most general. As we’ll see, it can be described as the rate of information you gain about the space by applying the function, when you have poor eyesight — in the limit where your eyesight becomes perfect.

Let $$X$$ be a metric space. It could be a surface, it could be a manifold, it could be a Riemannian manifold. Just some space with an idea of distance on it. We’ll write $$d(x,y)$$ for the distance between $$x$$ and $$y$$. So, for instance, $$d(x,x) = 0$$; the distance from a point to itself is zero. Additionally, $$d(x,y) = d(y,x)$$; the distance from $$x$$ to $$y$$ is the same as the distance from $$y$$ to $$x$$; the triangle inequality applies as well. And if $$x \neq y$$ then $$d(x,y) > 0$$; to get from one point to a different point you have to travel over more than zero distance!

We assume $$X$$ is compact, so roughly speaking, it has no holes, it doesn’t go off to infinity, its volume (if it has a volume) is finite.

Now, we will think of $$X$$ as a space we are looking at, but we can’t see precisely. We have myopia. Our eyes are not that good, and we can only tell if two points are different if they are sufficiently far apart. We can only resolve points which have a certain degree of separation. Let this resolution be $$\varepsilon$$. So if two points $$x,y$$ are distance less than $$\varepsilon$$ apart, then our eyes can’t tell them apart.

Rather than thinking of this situation as poor vision, you can alternatively suppose that $$X$$ is quantum mechanical: there is uncertainty in the position of points, so if $$x$$ and $$y$$ are sufficently close, your measurement can’t be guaranteed to distinguish between them. Only when $$x$$ and $$y$$ are sufficiently far apart can your measurement definitely tell them apart.

We suppose that we have a function $$f \colon X \rightarrow X$$. So $$f$$ sends points of $$X$$ to points of $$X$$. We assume $$f$$ is continuous, but nothing more. So, roughly, if $$x$$ and $$y$$ are close then $$f(x)$$ and $$f(y)$$ are close. (Making that rough statement precise is what the beginning of analysis is about.) We do not assume that $$f$$ is injective; it could send many points to the same point. Nor do we assume $$f$$ is surjective; it might send all the points of $$X$$ to a small region of $$X$$. All we know about $$f$$ is that it jumbles up the points of $$f$$, moving them around, in a continuous fashion.

We are going to define the topological entropy of $$f$$, as a measure of the rate of information we can get out of $$f$$, under the constraints of our poor eyesight (or our quantum uncertainty). The topological entropy of $$f$$ is just a real number associated to $$f$$, denoted $$h_{top}(f)$$. In fact it’s a non-negative number. It could be as low as zero, and it can be infinite; and it can be any real number in between.

We ask: what is the maximum number of points can we distinguish, despite our poor eyesight / quantum uncertainty? If the answer is $$N$$, then there exist $$N$$ points $$x_1, \ldots, x_N$$ in $$X$$, such that any two of them are separated by a distance of at least $$\varepsilon$$. In other words, for any two points $$x_i, x_j$$ (with $$i \neq j$$) among these $$N$$ points, we have $$d(x_i, x_j) \geq \varepsilon$$. And if the answer is $$N$$, then this is the maximum number; so there do not exist $$N+1$$ points which are all separated by a distance of at least $$\varepsilon$$.

Call this number $$N(\varepsilon)$$. So $$N(\varepsilon)$$ is the maximum number of points of $$X$$ our poor eyes can tell apart.

(Note that the number of points you can distinguish is necessarily finite, since they all lie in the compact space $$X$$. There’s no way your shoddy eyesight can tell apart infinitely many points in a space of finite volume! So $$N(\varepsilon)$$ is always finite.)

Clearly, if our eyesight deteriorates, then we see less, and we can distinguish fewer points. Similarly, if our eyes improve, then we see more, so we can distinguish more points. Eyesight deterioration means $$\varepsilon$$ increases: we can only distinguish points if they are further apart. Similarly, eyesight improvement means $$\varepsilon$$ decreases: we can tell apart points that are closer together.

Therefore, $$N(\varepsilon)$$ is a decreasing function of $$\varepsilon$$. As $$\varepsilon$$ increases, our eyesight deteriorates, and we can distinguish fewer points.

Now, we haven’t yet used the function $$f$$. Time to bring it into the picture.

So far, we’ve thought of our eyesight as being limited by space — by the spatial resolution it can distinguish. But our eyesight also applies over time.

We can think of the function $$f$$ as describing a “time step”. After each second, say, each point $$x$$ of $$X$$ moves to $$f(x)$$. So a point $$x$$ moves to $$f(x)$$ after 1 second, to $$f(f(x))$$ after 2 seconds, to $$f(f(f(x)))$$ after 3 seconds, and so on. In other words, we iterate the function $$f$$. If $$f$$ is applied $$n$$ times to $$x$$, we denote this by $$f^{(n)}(x)$$. So, for instance, $$f^{(3)}(x) = f(f(f(x)))$$.

The idea is that, if you stare at two moving points for long enough, you might not be able to distinguish them at first, but if eventually you may be able to. If they move apart at some point, then you may be able to distinguish them.

So while your eyes are encumbered by space, the are assisted by time. Your shoddy eyes have a finite spatial resolution they can distinguish, but over time points may move apart enough for you to resolve them.

(You can also think about this in a “quantum” way. The uncertainty principle says that uncertainties in space and time are complementary. If you look over a longer time period, you allow a greater uncertainty in time, which allows for smaller uncertainty in position. But from now on I’ll stick to my non-quantum myopia analogy.)

We can then ask a similar question: what is the maximum number of points we can distinguish, despite our myopia, while viewing the system for $$T$$ seconds? If the answer is $$N$$, then there exist $$N$$ points $$x_1, \ldots, x_N$$ in $$X$$, such that at some point over $$T$$ seconds, i.e. $$T$$ iterations of the function $$f$$, any two of them become separated by a distance of at least $$\varepsilon$$. In other words, for any two points $$x_i, x_j$$ (with $$i \neq j$$) among these $$N$$ points, there exists some time $$t$$, where $$0 \leq t \leq T$$, such that $$d(f^{(t)}(x_i), f^{(t)}(x_j)) \geq \varepsilon$$. And if the answer is $$N$$, then this is again the maximal number, so there do not exist $$N+1$$ points which all become separated at some instant over $$T$$ seconds.

Call this number $$N(f, \varepsilon, T)$$. So $$N(\varepsilon)$$ is the maximum number of points of $$X$$ our decrepit eyes can distinguish over $$T$$ seconds, i.e. $$T$$ iterations of the function $$f$$.

Now if we allow ourselves more time, then we have a better chance to see points separating. As long as there is one instant of time at which two points separate, we can distinguish them. So as $$T$$ increases, we can distinguish more points. In other words, $$N(f, \varepsilon, T)$$ is an increasing function of $$T$$.

And by our previous argument about $$\varepsilon$$, $$N(f, \varepsilon, T)$$ is a decreasing function of $$\varepsilon$$.

So we’ve deduced that the number of points we can distinguish over time, $$N(f, \varepsilon, T)$$, is a decreasing function of $$\varepsilon$$, and an increasing function of $$T$$.

We can think of the number $$N(f, \varepsilon, T)$$ as an amount of information: the number of points we can tell apart is surely some interesting data!

But rather than think about a single instant in time, we want to think of the rate of information we obtain, as time passes. How much more information do we get each time we iterate $$f$$?

As we iterate $$f$$, and we look at our space $$X$$ over a longer time interval, we know that we can distinguish more points: $$N(f, \varepsilon, T)$$ is an increasing function of $$T$$. But how fast is it increasing?

To pick one possibility out of thin air, it might be the case, that every time we iterate $$f$$, i.e. when we increase $$T$$ by $$1$$, that we can distinguish twice as many points. In that case, $$N(f, \varepsilon, T)$$ doubles every time we increment $$T$$ by 1, and we will have something like $$N(f, \varepsilon, T) = 2^T$$. In this case, $$N$$ is increasing exponentially, and the (exponential) growth rate is given by the base 2.

(Note that doubling the number of points you can distinguish is just like having 1 extra bit of information: with 3 bits you can describe $$2^3 = 8$$ different things, but with 4 bits you can describe $$2^4 = 16$$ things — twice as many!)

Similarly, to pick another possibility out of thin air, if it were the case that $$N(f, \varepsilon, T)$$ tripled every time we incremented $$T$$ by $$1$$, then we would have something like $$N(f, \varepsilon, T) = 3^T$$, and the growth rate would be 3.

But in general, $$N(f, \varepsilon, T)$$ will not increase in such a simple way. However, there is a standard way to describe the growth rate: look at the logarithm of $$N(f, \varepsilon, T)$$, and divide by $$T$$. For instance, if $$N(f, \varepsilon, T) \sim 2^T$$, then we have $$\frac{1}{T} \log N(f, \varepsilon, T) \sim 2$$. And then see what happens as $$T$$ becomes larger and larger. As $$T$$ becomes very large, you’ll get an asymptotic rate of information gain from each iteration of $$f$$.

(In describing a logarithm, we should technically specify what the base of the logarithm is. It could be anything; I don’t care. Pick your favourite base. Since we’re talking about information, I’d pick base 2.)

This leads us to think that we should consider the limit
$\lim_{T \rightarrow \infty} \frac{1}{T} \log N (f, \varepsilon, N).$
This is a great idea, except that if $$N (f, \varepsilon, N)$$ grows in an irregular fashion, this limit might not exist! But that’s OK, there’s a standard analysis trick to get around these kinds of situations. Rather than taking a limit, we’ll take a lim inf, which always exists.
$\liminf_{T \rightarrow \infty} \frac{1}{T} \log N (f, \varepsilon, N).$

(The astute reader might ask, why lim inf and not lim sup? We could actually use either: they both give the same result. In our analogy, we might want to know the rate of information we’re guaranteed to get out of $$f$$, so we’ll take the lower bound.)

And this is almost the definition of topological entropy! By taking a limit (or rather, a lim inf), we have eliminated the dependence on $$T$$. But this limit still depends on $$\varepsilon$$, the resolution of our eyesight.

Although our eyesight is shoddy, mathematics is not! So in fact, to obtain the ideal rate of information gain, we will take a limit as our eyesight becomes perfect! That is, we take a limit as $$\varepsilon$$ approaches zero.

And this is the definition of the topological entropy of $$f$$:
$h_{top}(f) = \lim_{\varepsilon \rightarrow 0} \liminf_{T \rightarrow \infty} \frac{1}{T} \log N(f, \varepsilon, n).$
So the topological entropy is, as we said in the beginning, the asymptotic rate of information we gain in our ability to distinguish points in $$X$$ as we iterate $$f$$, in the limit of perfect eyesight!

As it turns out, even though we heavily relied on distances in $$X$$ throughout this definition, $$h_{top}(f)$$ is completely independent of our notion of distance! If we replace our metric, or distance function $$d(x,y)$$ with a different one, we will obtain the same result for $$h_{top}$$. So the topological entropy really is topological — it has nothing to do with any notion of distance at all.

This is just one of several ways to define topological entropy. There are many others, just as wonderful and surprising and which scratch the tip of an iceberg.

References:

Topological entropy: information in the limit of perfect eyesight
Tagged on: