The Sum, the Product, and the Number That Vanishes

Two symbols, sigma and pi, do nothing more than fold a list of numbers into one, by adding or by multiplying. The adding runs almost the whole field. The multiplying hides a flaw that kills a training run after a few dozen steps, and the way out is the single most important piece of arithmetic in machine learning, and the reason logarithms exist at all.

From a list to one number

Class 6 ended with a function: one definite output for each input. So why is that not the end of the story?

Because we almost never ask a function for a single output. We sweep it across a whole collection of inputs and are left holding a list. A loss function returns an error for every example in the dataset: a thousand examples, a thousand errors. A language model returns a probability for every word it predicts: a ten-word sentence, ten probabilities. A vector is itself a list, one number per dimension. And a list is not an answer. What we want is one number drawn out of it: the total error, the score of the sentence, the length of the vector.

The operation sitting underneath almost everything is this: take a list of numbers and collapse it into one. What are the ways to do that?

There are two, and you have used both since long before you could name them. You can add the list, or you can multiply it. Add five prices and you have a bill. Multiply the chances of three coin flips and you have the chance they all land heads, \(0.5 \times 0.5 \times 0.5 = 0.125\). These two ways of folding a list are so common that each earned its own symbol: a capital sigma, \(\Sigma\), for “add this list,” and a capital pi, \(\Pi\), for “multiply this list.” That is the whole connection to Class 6: a function produces a list of values, and sigma and pi are how a list becomes a single number again.

Sigma: add the list

To add a list of numbers in code, you write a loop with a running total:

The sigma is that loop, written in one stroke:

Every part of the symbol matches a part of the loop, and nothing else is hidden in it.

Expanding it just means writing the terms out: \(\Sigma\, a_i\) is \(a_1 + a_2 + a_3 + a_4 + a_5\), which for our list is \(2 + 5 + 3 + 7 + 1 = 18\). Build any sum or product below and the running value updates term by term, exactly as the loop would.

Three things you already rely on are nothing more than a list folded by adding. The first is the loss. Sweep a loss function across the dataset and you get one error per example; fold them by adding and divide by the count to get a typical value.

Written out it is \(\tfrac{1}{n}\sum \text{error}_i\): the \(\Sigma\) adds the errors, \(\text{error}_i\) is the error on the \(i\)-th example, and the \(\tfrac{1}{n}\) turns the total into a typical value. That number is the loss, and the entire activity of training is moving it downward.

The second is the dot product. You have a list of inputs, and beside it a list of weights the model has learned, one per input, each saying how much that input should count. To fold the two lists into a single score you pair them by position, multiply each pair, and add the products.

In symbols this is \(\sum w_i x_i\). Look at when the score comes out large: precisely when big inputs line up with big weights, when the two lists point the same way; it is small when they pull in opposite directions. A dot product measures how much two lists agree. That single idea is why it is everywhere. A neuron is one dot product, scoring an input against the weights it has learned to look for; a layer is many neurons side by side; a deep network is layers of them. So “agreement, summed” is the innermost act of the entire machine, run billions of times, and the same operation later measures how similar two vectors are, which is how models compare meanings.

The third is the expected value. Say a scratchcard pays nothing 70% of the time, pays $5 20% of the time, and pays $50 the other 10%. What is one card worth? Averaging the three prizes, \((0 + 5 + 50)/3 \approx \$18\), is too high, because $0 happens far more often than $50 and a plain average treats them as equally common. Imagine playing 100 times: about 70 cards pay nothing, 20 pay $5, 10 pay $50.

A card is worth $6, found by weighting each prize by how often it happens and adding: \(0.7\times 0 + 0.2\times 5 + 0.1\times 50 = 6\). Each outcome multiplied by its probability, all summed. That is the expected value, \(\sum p_i x_i\), which is just the dot product of the outcomes with their probabilities. Weighting values by how likely they are, rather than treating them as equally common, is how you average when some things happen more than others, and it runs all through probability and learning.

Average, dot product, expected value: each is a list folded by adding. Read a \(\Sigma\) as a for-loop that adds, and most of the symbols in a paper turn back into code you already understand. That is sigma. Now the other fold.

Pi: multiply the list

Pi is the same machine as sigma with one part swapped. The running value starts at \(1\) instead of \(0\), and each step multiplies instead of adds:

You read \(\Pi\, a_i\) exactly like the sigma, with one word changed: the big pi means “multiply everything to my right.” The mechanics are that simple. The real question is why we would multiply a list rather than add it.

Probabilities are the clearest case where adding is the wrong tool. Take a fair coin and ask for the chance of heads three times in a row. Each flip is a \(0.5\) chance. Add them and you get \(1.5\), which is impossible, because nothing has a 150% chance. Adding has broken. The correct way to combine the chances of a sequence is to multiply them.

But why does multiplying give the chance of a sequence? Take a concrete pair of events. It rains on 30% of days, and on rainy days you forget your umbrella half the time. Out of 100 days, how many are both rainy and umbrella-less? First, 30 of the 100 are rainy. Then, of those 30, half, so 15, are the ones you also forgot the umbrella. That is 15 in 100, a chance of \(0.15\). We took 30% of the days, then 50% of those: a fraction of a fraction, which is exactly multiplying, \(0.3 \times 0.5 = 0.15\).

That is the rule. When events are independent, meaning one happening does not change the odds of another, the chance that they all happen is the product of their separate chances, one factor for each “and also.” So a product of probabilities \(\Pi\, p_i\) has a clear meaning: it is the chance that all of them happen at once. And since every factor is below 1, the product shrinks with each new one, the seed of the next section. First, though, this one meaning grows into the most important number in training.

A model is a guess about how the world behaves, and the simplest imaginable is a single number: the chance of rain. A model claiming rain has a 30% chance is already complete, because it gives the probability of either thing that can happen on a day, rain with probability \(0.3\), dry with probability \(0.7\). With a model in hand you can ask how probable any stretch of weather is. For three days that went rain, dry, rain, the model assigns \(0.3\), \(0.7\), \(0.3\), and the chance of that exact run is their product.

That is the model running in its natural direction, from a known model to the chance of an outcome. Real life runs the other way. Nobody tells you a new city’s true rain chance; the days already happened, you watched them, and the model is the unknown you are trying to find.

Given the days we actually saw, which model should we believe?

We cannot look up the true model, but we can take any candidate and ask it the forward question about the data we already have: if this candidate were the truth, how probable was the week we saw? Say the week had two rainy days and three dry ones. A candidate that believes rain is rare, a 10% chance, scores it by multiplying the chance it gave to each day:

Almost nothing. This candidate is shocked that it rained twice, so under it the observed week looks like a fluke, which makes it a poor explanation. A single score means little; it earns its keep in comparison. So try a second candidate that believes rain is more common, a 40% chance, and score the identical five days.

The 40% candidate gives the week \(0.0346\), nearly five times the 10% candidate’s \(0.0073\). It is barely surprised by what happened, so it explains the week far better. That score, the probability a candidate assigns to the data you actually saw, is the likelihood, and the rule for using it is the one we just watched: the higher a model’s likelihood, the better it fits the data.

Two candidates were enough to see the idea. To actually learn the rain chance, let the candidate take every value between 0 and 1 and score each by the likelihood it gives the observed week:

• \(p = 0.1\): \(0.1 \times 0.1 \times 0.9 \times 0.9 \times 0.9 = 0.0073\). Too low, shocked by the two rainy days.
• \(p = 0.4\): \(0.4 \times 0.4 \times 0.6 \times 0.6 \times 0.6 = 0.0346\). Comfortable with both the rain and the dry.
• \(p = 0.8\): \(0.8 \times 0.8 \times 0.2 \times 0.2 \times 0.2 = 0.0051\). Too high, shocked by the three dry days.

Plot the likelihood for every \(p\) and the scores rise to a single clear peak:

The peak sits exactly at \(p = 0.4\), which is just two rainy days out of five, the answer you would have guessed without any of this. That agreement is the reassurance that the method is sane. The real prize is the recipe it followed: choose the model that makes the observed data most probable. That recipe does not care how complicated the model is. It works the same when the model is a single rain chance and when it is the billions of numbers inside a neural network. Choosing a model this way is called maximum likelihood, and most of training is a version of it. The feeling to carry out: the likelihood measures how unsurprised a model is by reality, and training is the search for the model that finds the world least surprising.

A language model is this exact idea with words in place of weather. Its model is not one rain chance but billions of weights, and at each position it gives a probability to the word that actually came next. The likelihood of a whole sentence is the product of those per-word probabilities, \(\Pi\, p_i\), and training adjusts the weights to make the real text as probable as it can, the product as large as it can. A three-word sentence the model rated \(0.2\), then \(0.1\), then \(0.5\) has likelihood \(0.2 \times 0.1 \times 0.5 = 0.01\). So this product is the exact quantity training works to push upward, which turns the next section into a genuine problem.

The product that vanishes

The likelihood is a product of probabilities that training works to make as large as it can, and a real dataset pours hundreds or thousands of probabilities into it. Before any maximizing can happen, a computer has to compute that product, and that is where a quiet disaster waits.

A product of hundreds of probabilities, each a number between 0 and 1. What does a real computer actually hold at the end?

Start with the friendliest product there is, coin flips. The chance of heads \(n\) times in a row is \(0.5\) multiplied by itself \(n\) times, and every extra flip halves the result. Ten flips is about one in a thousand; twenty about one in a million. It drops with astonishing speed. But one thing stays true the whole way down: it is never zero. Halving a positive number leaves a positive number, always. On paper the product shrinks forever and never reaches zero.

Now do it on the chip from Unit 1, which stores numbers as a float. Keep halving and count the flips:

Not a small number. A flat, exact \(0\), and not after thousands of steps, after about 150. A real likelihood is worse, because word and data probabilities sit well below one-half and there are hundreds of them. With each probability around \(0.1\), the product underflows to zero after only 46 of them. The paper math says positive forever; the chip says zero within a paragraph.

The reason it lands on exactly zero is the hinge the whole fix turns on. A float does not keep a long string of digits. It keeps a few significant digits and, separately, an exponent that says where the decimal point sits, a value times a power of two. Those are two different budgets, one for the digits and one for the exponent.

In our shrinking product the digits were never in trouble; the exponent was. A float’s exponent reaches down only to about \(10^{-38}\), and a little further, to roughly \(10^{-45}\), through some special small values. But our product needed an exponent near \(10^{-46}\), then \(10^{-150}\), then beyond. It did not run out of digits; it ran out of exponent. Once a step lands past the smallest exponent the float can write, there is no number left for it to be, so it snaps to \(0\). This is the bottom cliff from Class 4, underflow, and a product of small numbers runs off it within a few dozen steps.

A tiny number rounded to zero sounds harmless. It is not. Training compares candidate models by their likelihoods and climbs toward the higher ones, and the differences between those scores are the entire signal it follows. Underflow erases that signal. A decent model near \(10^{-50}\) and a slightly better one near \(10^{-48}\) both fall past the floor and store as the same flat \(0\); a good model and a hopeless one, \(10^{-50}\) against \(10^{-300}\), also both read as \(0\). The numbers that told these models apart are gone, and once every likelihood is pinned at zero there is no “higher” left to climb toward. Training goes blind.

There is an obvious thing to try, and it leads exactly where the next section goes: the number is too small to store, so do not store the number, store its logarithm. The log of a tiny positive value is a manageable negative one, with no underflow in sight. The instinct is right, but done naively it arrives one step too late. If you let the probabilities multiply all the way down to a stored \(0\) and only then take the log, you are computing \(\log(0) = -\infty\), the infinity from Class 4, which is worse than the zero it replaced: it pours into the loss and turns the gradients to NaN, killing the run outright rather than merely starving it.

This is not a rare corner you can dodge with care. A likelihood multiplies one probability for every example in the data, and any real dataset has far more than the few dozen factors it takes to underflow, so the naive product is all but guaranteed to collapse to zero. The single most important quantity in training cannot be computed the way its definition reads. Pick a probability and a count and watch the wall arrive.

The fix: turn the product into a sum

The demand from the last section is sharp: take the log before the product is ever formed, so the doomed multiplication never happens. At first that sounds impossible. The likelihood is a product, and the log seems to come afterward.

How can you take the log of a product without first building the product?

The entire fix rests on one property of logarithms. A logarithm answers a plain question: what power is this number? In powers of ten, \(\log 1000 = 3\), because \(1000 = 10^{3}\). The log of a number is its exponent, loosely the count of its zeros. Now watch what multiplication does to exponents: \(10^{2} \times 10^{3} = 10^{5}\), and the exponent \(5\) is simply \(2 + 3\). Multiplying the numbers added their exponents. And since a log just is the exponent, the log of a product equals the sum of the logs.

So \(\log(a \times b) = \log a + \log b\). It is not a rule to memorize; it is only “exponents add when you multiply,” stated for any two numbers instead of round powers of ten. And there is something almost uncanny about it: the logarithm is essentially the only continuous function that turns multiplication into addition. That is exactly why logs were invented four centuries ago, to turn the punishing multiplications of astronomy and navigation into additions a person could do by hand. Our underflow is the same old problem in modern dress, and it has the same answer.

Apply that property across the whole likelihood, every probability at once, and the product collapses into a sum.

A product of \(n\) probabilities, a pi, becomes a sum of \(n\) logs, a sigma. This is the moment the first two sections were quietly building toward: the fix literally turns the one symbol into the other. And it meets the demand exactly. You do not multiply the probabilities and then take a log. You take the log of each probability first, getting a modest number from each, and add those up. The product is never formed, so it can never underflow.

Look at the size of the numbers you now carry. \(\log(0.1)\) is about \(-2.3\); \(\log(0.5)\) about \(-0.69\). Add a few hundred of them and you land around \(-200\) to \(-700\), a number a float holds without a flicker. There is no tiny value anywhere to lose, because we only ever add ordinary ones. Here is the same three-probability likelihood from before, done both ways:

There is a deeper way to see why this buys so much room. A float spends only about eight of its bits on the exponent, which is why its reach stops near \(10^{-38}\) and a product runs out of exponent so fast. Taking the log moves that exponent out of the cramped exponent field and into the float’s full value, the whole significant-digits budget, a vastly larger space. We trade a little precision we do not need for an enormous range we very much do. The log-likelihood is the product’s exponent, finally given room to breathe.

One thing should make you uneasy, though. We set out to make the likelihood as large as possible, and now we are making the sum of its logs as large as possible instead, which is not the same number: a likelihood of \(0.01\) has log-likelihood \(\log(0.01) = -4.6\), a negative number. We swapped the target. Is the swap harmless? Two models settle it before trusting it on a billion. Model A gives the data a likelihood of \(0.01\); model B gives \(0.0001\). A is clearly the better fit, a hundred times more likely. By their logs, \(\log(0.01) = -4.6\) and \(\log(0.0001) = -9.2\): the better model’s log is the larger because it sits closer to zero. A is still ahead. The swap did not reorder them, and the reason is general: the log only ever climbs.

Feed the log a bigger input and you always get a bigger output; its graph rises from left to right and never turns back down. So whenever one likelihood is larger than another, its log is larger too, with no exceptions. Taking logs stretches and shifts the numbers but never swaps their order, so the model at the top of the likelihood ranking is still at the top of the log-likelihood ranking. Maximizing the sum of logs lands on the exact same model as maximizing the likelihood would have, by a route that never goes near the underflow cliff.

That quantity \(\sum \log(p_i)\) is the log-likelihood, and training maximizes it. Flip its sign and average it and you have the cross-entropy, also called the negative log-likelihood loss, \(-\tfrac{1}{n}\sum \log(p_i)\), the loss behind essentially every classifier and language model. That formula is not arbitrary. It is a sum of logs precisely so the product it stands for cannot underflow to zero and take the run with it. The two quiet symbols from the opening, sigma and pi, joined by a logarithm, are the difference between a model that learns and one that dies on its first batch.

What to carry out of Class 7

A function hands us a list of values, and we constantly need to fold that list into one number. Sigma does it by adding, pi by multiplying, and each symbol only records where to start, where to stop, and the term to combine, the loops you already write. Adding runs the field: a loss is an average, a neuron is a dot product measuring how much an input agrees with its weights, an expectation is a weighted sum. Multiplying gives the likelihood, the probability a model assigns to the data actually seen, a measure of how unsurprised it is by reality, which training pushes upward. But a product of many probabilities underflows to a hard zero on real hardware, because a float runs out of room in its exponent, the bottom cliff from Class 4, and that zero poisons training through \(\log(0) = -\infty\).

The next class stays with this language and asks a question hiding under every probability we just multiplied: where does a number like “30% chance of rain” come from in the first place? That is counting and the sample space, and it is the floor the whole tower of probability stands on.

See if it stuck

Eight questions, all answerable from this post. Tap an answer for an immediate verdict and the reason.