Artificial neural nets are good enough

A case of being the same but different

A topologist is someone who can’t tell the difference between a donut and coffee mug. So goes the old math joke. Topology only cares about geometric properties that aren’t affected by continuous changes in shape or size. So while donuts and coffee mugs seem different in most respects, they share the property of having precisely one hole, and can thus be deformed into each other without pinching or tearing — what topologists call a “homeomorphism.”

A morphism is a type of equivalence. Math is full of them. Two sets are said to be equivalent “up to isomorphism,” for example, if there’s a one-to-one mapping between the objects and relations in each set.

For breakfast on the go

It’s an unpopular opinion, but what if biological neurons and artificial neural networks are equivalent in an analogous sense? We’re constantly reminded about the vast differences between the two, but, as in topology, what if you can abstract those differences away?

Consider a simple convolutional neural net like the autoencoder. An autoencoder has two main parts: an encoder that maps an input into a lower-dimensional latent space (the “code”), and a decoder that then reconstructs the input from the information in the latent space as best it can. Compression, decompression. By training a model to minimize the error between the original and reconstructed input, autoencoders can learn to extract the relevant features of a class of inputs in a self-supervised fashion. For extracting textual data, for example, an autoencoder might learn to ignore the color or font style of a given piece of text in favor of the regularities in the edges and curves of particular letters. The latent space of those features can be thought of as the “invariants” of the data — what makes a letter in different writing styles count as the same letter — just as in topology one throws out a ton of geometric data in favor of a few key invariants, such as the number of holes.

Thank you Wikipedia

In some sense all of math is just equivalence mappings between different spaces. Yet math lives is the un-computable world of Platonic forms. The real world is messy, full of discrete data points; not some smooth manifold. Neural networks can thus be thought of as a way to approximate functional maps constructively. Take transformer models. Transformers have been shown to be “universal approximators of continuous sequence-to-sequence functions.” That is, with sufficient depth, transformers can approximate the map between any arbitrary input sequence of tokens and output sequence of tokens. Given finite size and compute, a transformer model will never extract an exact mapping between two sequences. But as we’re just now learning, a large enough model gets you close enough for government work.

In turn, we now have language models that can easily map an essay written in English into an “equivalent” essay written in German. The model does this by learning what remains invariant between the two — namely, the shared semantic representations, abstracted away from the idiosyncrasies of each language’s grammar and vocabulary. While the two essays look different to the naked eye, as with the coffee cup and donut, there’s a sense in which they’re actually equivalent, at least up to a kind of “semantic morphism.”

An analogous equivalence can be drawn between biological neurons and artificial ones. Unlike artificial neural networks, the neurons in our brain don’t send continuous values but instead either fire or don’t. Neurons also have multiple dendrites that receive input from multiple sources, come in many cell types, and exhibit rich and varied structure. But are those differences really that important, or are they incidental data — the design choices of a blind watchmaker — that may be relevant for replicating human-like cognition but not intelligence per se?

The answer to that question matters enormously. If you think biological neural networks are doing something special — that they contain some secret sauce that computers lack — you’ll tend to have strong priors against AGI being around the corner. In contrast, if you think artificial neural networks capture certain universal properties of biological neurons, if only in a lower dimensional form, then you’ll tend to have shorter AI timelines, as the differences between the brain and a deep reinforcement neural network becomes more a matter of degree than kind.

I take the finding that cortical neurons “are well approximated by a deep neural network with 5–8 layers” as strong evidence in favor of this latter, “universalist” perspective. On the one hand, it’s a testament to the amazing complexity of the brain that a multi-layer artificial neural network is needed to model a single cortical neuron. On the other hand: Holy shit! Artificial neural networks can efficiently approximate cortical neurons!

At least that’s the reaction I’m having.