The Mathematics ML Runs On · Class 10

What Happens If You Just Keep Multiplying?

Almost everything a model does, underneath, is one small act repeated: multiply, then multiply again. Do it forty-six times to a probability and it vanishes. Do it without end and a single number falls out, one that no 32-bit chip can hold.

There is an old story about a king and a sage. The sage does the king a favour and is offered any reward he likes. He asks for something that sounds humble. One grain of rice on the first square of a chessboard, two on the second, four on the third, and so on, doubling, all the way to the sixty-fourth square. The king agrees, relieved to be let off so lightly.

He should not have been relieved. The doubling starts gently, a few grains, a handful, a bowl. Halfway across the board it is a field of rice. By the sixty-fourth square the sage is owed more than eighteen quintillion grains, more rice than the world has grown in all of history.

Nothing in that story is really about rice. It is about one move, repeated. Double, and double again, and the total does not climb, it erupts. That move, multiplying by a fixed amount over and over, is the whole of this class.

one grain, doubled across a chessboard

Follow the move in one direction and numbers explode past anything a machine can hold. Follow it in the other and they collapse to nothing. Take it to its smooth limit and a single constant falls out, written \(e\), and a single curve, \(e^{x}\), that turns out to be hiding inside nearly every probability a model ever computes. At every step, the move fights the hardware it runs on.

One move, multiply and multiply again, builds the most common function in machine learning, and breaks it on contact with a real chip.

This class follows that one move through five rooms: the two directions it can travel, the number it settles on, the function it becomes, the place it hides, and the wall it hits.

Two directions

Start with the difference between adding and multiplying, because a model lives on the second.

Add a fixed amount each step and you get a tame, straight climb: 100, 110, 120, 130. The increase never changes, because it never depends on what you already have. This is the growth of a coin tin, and it is the growth nobody needs to fear.

Multiply by a fixed amount each step and everything changes. Now each step acts on the whole pile, so the bigger the pile, the bigger the jump. 100 becomes 110, then 121, then 133, and the increases themselves grow. This is the chessboard. Give it enough steps and it overtakes any straight line, then leaves it so far behind that the comparison stops meaning anything.

adding a fixed amount, versus multiplying by a fixed factor

A model is built almost entirely on the multiplying kind, and not always upward. The probability it assigns to a whole sentence is the product of the probabilities of its parts, hundreds of small numbers multiplied together. Each is below 1, so this is the same move run the other way: multiply by a fraction, again and again, and the result does not drift toward zero, it dives.

Here the machine is already in trouble. Multiply 0.1 by itself in a 32-bit float and after just forty-six steps the answer is not tiny, it is exactly zero. The number slipped below the smallest value the chip can store, so it rounded to nothing. The move a model leans on hardest is the one that wipes out its own numbers fastest.

two_directions.js// the same move, both ways, on a 32-bit float
let up = 1;
for (let i = 0; i < 63; i++) up *= 2;          // double 63 times: the chessboard

let down = Math.fround(1);
for (let i = 0; i < 46; i++)
  down = Math.fround(down * Math.fround(0.1));   // multiply by 0.1, 46 times

console.log("up  :", up);
console.log("down:", down);

one move, opposite fates

output appears here

The escape from that collapse, taking logarithms to turn the runaway product back into a sum, is the next class. For now, just hold the picture: one move, two directions. Upward it erupts, downward it vanishes. Everything left here is what happens when you stop taking discrete steps and let the move run smooth.

The number at the end of the move

Steps are jagged. Real growth is usually smooth, happening every instant, and a machine has no instruction for every instant. The only thing it can do is the move: take a small step, multiply, repeat. So it fakes smoothness by making the steps tiny and many.

Take the cleanest case, growing by 100%. Do it in one step and you multiply by 2. Do it in two steps of 50% and you multiply by 1.5 twice, landing on 2.25, more than 2, because the second step grows what the first already added. Do it in \(n\) steps and you multiply by \(\left(1+\tfrac1n\right)\), \(n\) times over, which is \(\left(1+\tfrac1n\right)^{n}\).

Now make the steps infinite. The move is upward, and the chessboard says this should erupt. It does not. It climbs, slower and slower, and parks on one number: 2.71828. Multiply by a hair above 1 infinitely often and you do not get infinity, you get 2.71828. That number is called \(e\).

\( e \;=\; \lim_{n\to\infty}\left(1+\dfrac{1}{n}\right)^{n} \;=\; 2.718281828\ldots \)

Why it parks instead of erupting is the chessboard in reverse: each finer step multiplies by less, so each one buys less than the last. The first split added 0.25, the next 0.12, then 0.07, the gains dying fast enough to sum to a ceiling. The ceiling is \(e\), about 36% above plain doubling and not a fraction more.

more steps, but the value flattens at e

Then try to actually compute \(e\) this way on the ESP32-C3, and the move turns on you. In full 64-bit precision the loop closes in on 2.71828, as it should. In the chip’s 32-bit float it does something stranger: it wanders, 2.59, then 3.29, never settling, and then at one exact value of \(n\) it drops to 1 and stays there.

compute_e_float32.js// (1 + 1/n)^n in double vs single precision, as n climbs
function e64(n) { return Math.pow(1 + 1/n, n); }
function e32(n) { return Math.pow(Math.fround(1 + 1/n), n); }
for (let n of [1e3, 1e6, 1e7, 2**24, 1e8]) {
  console.log(n, e64(n).toFixed(4), e32(n).toFixed(4));
}

double holds; float32 wanders, then dies at 2³⁴

output appears here

That exact value is \(n = 2^{24} = 16{,}777{,}216\), and it is no accident. A 32-bit float keeps 24 bits of precision, the mantissa width from Class 2, so the smallest amount it can add to 1 is \(2^{-24}\). Once the step \(\tfrac1n\) drops below that, \(1+\tfrac1n\) rounds to exactly 1, and 1 multiplied by itself any number of times is 1. The move that defines \(e\) erases itself, at a value you can read straight off the width of the float. Brute force does not just fail here, it fails worse the harder you push. No real library computes \(e\) this way.

So the move, run smooth, hands us a number. Now we let the amount of growth vary, and the number becomes a function.

The move as a dial

\(e\) is one number, the result of one unit of smooth growth. Turn the amount of growth into a dial, call it \(x\), and you get \(e^{x}\), the move applied as much or as little as you ask.

Read a few settings off the dial. No growth, \(e^{0} = 1\). One unit, \(e^{1} = 2.718\). Two units run the move twice, \(e^{2} = 7.39\). Turn the dial negative and the move runs downward into shrinking: \(e^{-1} = 0.37\), \(e^{-2} = 0.14\), sinking toward zero but never touching it. That last part is the one to keep: \(e^{x}\) is positive for every input there is. You cannot multiply your way to zero or below.

\(e^{x}\): never zero, through (0, 1), steep on the right

And it carries one property that is the reason it survives the rest of this course. Growing for a time \(a\) and then for a further time \(b\) is the same as growing for \(a+b\) at once. But the first way is \(e^{a}\) then \(e^{b}\), multiplied; the second is \(e^{a+b}\). The two must agree:

\( e^{a+b} \;=\; e^{a} \times e^{b} \)

Adding on the dial becomes multiplying in the value. The move turns sums into products. Run it backwards and it turns products into sums, which is exactly the escape from the forty-six-step collapse: take logarithms, and a runaway product of probabilities becomes a safe sum. That backwards move is the whole of the next class. \(e^{x}\) is the door; the logarithm is the same door swinging the other way.

Positive everywhere, and it trades sums for products. Those two facts are the entire reason the move is waiting inside every probability a model builds.

Where the move hides

A model spends its life producing raw scores, any number at all, and needing to turn them into probabilities, which must be positive and must behave. There is one natural tool for that job, and it is the function we just built.

A single score becomes a probability through the sigmoid, \(1/(1+e^{-x})\). Feed it a large positive score and \(e^{-x}\) vanishes, so the value approaches 1. A large negative score and \(e^{-x}\) erupts, so the value approaches 0. A score of 0 gives exactly one half. The whole infinite number line is folded smoothly into the range a probability is allowed to live in, and it is the always-positive move that does the folding.

A whole list of scores becomes a distribution through the softmax: raise \(e\) to each score, then divide by the total, \(p_i = e^{z_i} / \sum_j e^{z_j}\). Every \(e^{z}\) is positive, so every result is positive, and dividing by the sum forces them to add to 1. A list of arbitrary numbers becomes a set of probabilities, and again it is the move doing the work.

And the bell curve, the shape that fits almost every measurement ever taken, is \(e^{-x^{2}/2}\): the move running downward, faster the further you go from the centre, symmetric because of the square, positive everywhere because that is simply what the move is.

the same \(e^{x}\), hiding in three different jobs

the same move, three coats

Three different jobs, one function in all of them, for one reason. Each needs a quantity that is positive everywhere and bends smoothly, and multiplying-without-end is the only natural source of both at once. The sigmoid, the softmax, and the bell curve were never arbitrary choices. They are the same move wearing three coats.

Everywhere a model meets a probability, the move is already there. Which would be a tidy ending, except the move does not run for free.

Where the move breaks

Go back to the chessboard one last time. The move, run upward, erupts, and \(e^{x}\) is that eruption made into a function. On a 32-bit float, which tops out near \(3.4 \times 10^{38}\), it erupts straight through the ceiling almost at once. \(e^{88}\) still fits. \(e^{89}\) does not, and the chip returns infinity.

A score of 89 is nothing unusual inside a model, so the softmax, coded the obvious way, breaks: feed it large scores and every \(e^{z}\) overflows to infinity, and infinity over infinity is gibberish. The fix is small and rests entirely on the identity from the last section. Subtract the largest score from all of them first. This multiplies top and bottom by the same amount, so the probabilities are untouched, but now the biggest exponent is \(e^{0} = 1\) and nothing can overflow. Every serious softmax on Earth does this.

softmax_overflow.js// e^x overflows float32 just past x = 88
[88, 89, 90].forEach(x => console.log("e^" + x + " =", Math.fround(Math.exp(x))));

let z = [90, 89, 88];
let naive = z.map(s => Math.fround(Math.exp(s)));      // all Infinity
let m = Math.max(...z);                                // subtract the largest
let safe = z.map(s => Math.exp(s - m));
let tot = safe.reduce((a, b) => a + b, 0);
console.log("naive:", naive.join(", "));
console.log("safe :", z.map((s, i) => (safe[i] / tot).toFixed(3)).join(", "));

naive overflows; the max-subtraction fix does not

output appears here

And even when it does not overflow, the move is expensive. A processor has an instruction for adding and one for multiplying, but none for \(e^{x}\). It has to build the value from a long polynomial, many multiplies and adds for a single result, and on the ESP32-C3, which has no floating-point unit at all, even those multiplies are faked in software. The most common function in machine learning is also one of the most expensive things the chip ever does. Class 13 puts a cycle count on it.

One move. Multiply, then multiply again. Run it upward and it is the chessboard’s eruption; run it downward and it sinks a probability to zero in forty-six steps. Let it run smooth and it settles on \(e\). Put the amount of growth on a dial and it becomes \(e^{x}\), the one function that is positive everywhere and trades sums for products. That is why it hides inside every sigmoid, every softmax, every bell curve a model draws, and why those models overflow and burn cycles on the smallest chips. The exponential is not one topic among many. It is a single act of arithmetic, repeated until something remarkable, and something dangerous, falls out.

The next class takes the move backwards. The logarithm undoes the exponential, turns its runaway products back into sums, and is, quietly, the single most useful line of arithmetic in the whole field.

a short, self-marking quiz