The question I started with
There's a chip called the ESP32-S3. Espressif makes it, it costs about three dollars, and it's inside smart plugs, sensors, toy robots, and half the IoT gadgets you've ever opened up. The marketing calls it an AI-capable chip, and points at a hardware feature called PIE (Processor Instruction Extensions) that can do eight 16-bit multiplications in a single clock cycle.
Which sounds great. But I kept wondering what it means in practice. Does that hardware actually make real code faster? Where does it stop helping? And how far can a three-dollar chip carry the words "AI-capable" before they snap?
There's only one honest way to find out: benchmark the operation AI leans on hardest, matrix multiplication. Build it four ways, from the schoolbook version up to the dedicated hardware, run each one on the actual chip, and let the numbers talk.
So that's what I did.
What is matrix multiplication and why does AI care about it
A matrix is just a grid of numbers. Multiply two of them, A and B, and you get a third, C. Every number in C comes from taking a row of A and a column of B, multiplying them element by element, and adding it all up. That row-times-column sum is called a dot product.
The simplest possible version of the code looks like this:
for (int i = 0; i < n; i++) // for each row of A
for (int j = 0; j < n; j++) // for each column of B
for (int k = 0; k < n; k++) // dot product
C[i][j] += A[i][k] * B[k][j];Three nested loops. For an n×n matrix, that's n³ multiplications. Double n and the work goes up eight times. This is why big matrix multiplications hurt.
And why does AI care? Because a neural network, underneath everything, is a stack of matrix multiplications. Every layer takes an input vector, multiplies it by a weight matrix, and hands the result to the next layer. Training is finding the right values for those matrices. Inference is pushing your input through them, one multiply after another.
So if a chip multiplies matrices fast, it runs neural networks fast. If it struggles here, it struggles at everything with AI in the name. This is the benchmark that matters.
The hardware and why memory is everything
The ESP32-S3 has two Xtensa LX7 cores running at up to 240 MHz. On the chip itself sits 512 KB of SRAM, fast and close. Hanging off a bus is 8 MB of external PSRAM, much roomier and meaningfully slower.
The single most important thing about this chip, and honestly about most chips, is the memory hierarchy. The CPU can tick 240 million times a second, but the data it chews on has to come from somewhere, and where it lives decides how fast anything actually runs.
What is a cache and why does it exist?
Picture the CPU as a chef. The registers, where the actual computation happens, are the chef's hands. The L1 cache is the countertop: small, right there, instant to grab from. Internal SRAM is a shelf across the room, a short walk. PSRAM is the storage room down the hallway. Plenty of space, but every trip costs real time.
A cache keeps recently used data on the countertop so you stop walking to the storage room. When the CPU wants something, it checks the cache first. If it's there, a cache hit, it's instant. If not, a cache miss, and it fetches from slower memory at a cost of many cycles. Writing fast code on small hardware is mostly the art of having the data on the countertop before the chef reaches for it.
On the ESP32-S3 the data cache is 32 KB. I confirmed this in the sdkconfig file; it's configurable between 16 and 32 KB and defaults to 32 in ESP-IDF v5.x. The cache line is 32 bytes, so every miss drags in 32 consecutive bytes from slower memory.
Here's the hierarchy laid out with the one speed difference this whole benchmark hangs on:
That 40-cycle PSRAM penalty drives everything you're about to see. The CPU can multiply in 1 cycle. If the data lives in PSRAM, it sits idle for 40 waiting for it. For memory-bound work, this chip runs at roughly 2.5% of its theoretical compute speed. Forty cycles of waiting for one cycle of work.
What is PIE SIMD and what does it actually do?
SIMD stands for Single Instruction, Multiple Data. A normal multiply instruction takes two numbers and gives you one result. A SIMD instruction multiplies many pairs at once inside wider registers, in the same single clock cycle.
PIE gives the S3 128-bit vector registers. A 16-bit integer is 16 bits, so eight of them fit in one register. The instruction EE.VMULAS.S16.QACC grabs two packed registers, multiplies all 8 pairs at once, and accumulates. That's what "8-wide SIMD" means: one instruction doing the work of eight.
On paper that's an 8× speedup. Whether you get it depends entirely on whether you can feed the beast. A SIMD unit waiting on PSRAM is exactly as fast as a scalar unit waiting on PSRAM. Hardware is only as fast as the pipeline behind it.
The four implementations, explained from scratch
Each implementation isolates one variable, so you can see what the compiler contributes, what the memory access pattern contributes, and what the dedicated hardware contributes. All matrices live in PSRAM, because that's where they would have to live for any realistically sized model. Internal SRAM is far too small.
The textbook three loops, nothing else. I told the compiler to do nothing with -O0: no unrolling, no register reuse, no reordering. This is what you get if you copy the algorithm out of a textbook and hit build.
Its real crime is how it reads matrix B. The inner loop reads B[k][j], and as k climbs you jump across rows of B, touching memory locations 2×n bytes apart at every step. For large matrices nearly every one of those reads is a cache miss, and every miss is a 40-cycle trip to PSRAM. The CPU mostly stands around waiting.
for (int i = 0; i < n; i++) {
for (int j = 0; j < n; j++) {
int32_t acc = 0;
for (int k = 0; k < n; k++) {
acc += (int32_t)A[i*n+k] * (int32_t)B[k*n+j]; // B[k][j]: jumping rows = cache miss
}
C[i*n+j] = (int16_t)acc;
}
}The identical code, compiled with -O3. GCC now throws everything it has at it: unrolled loops, scheduled instructions, values pinned in registers, redundant work eliminated. The algorithm hasn't changed a character. Only the compiler's output has.
That makes it a clean experiment. The gap between naive and opt is purely the compiler's doing. Whatever gap remains after that is work the compiler cannot do for you.
And this is the lesson it teaches: the compiler is brilliant at instruction-level cleverness and helpless against a bad memory access pattern. If your data is in PSRAM and you're striding across it, no compiler flag will save you.
Here's where it gets fun. Instead of walking the full matrices in one pass, you chop them into small sub-matrices called tiles and work through one tile-triple at a time. Pick a tile size that fits in L1, and suddenly all the data for the hot inner loop is already sitting in cache. No PSRAM trips where it hurts.
The intuition behind cache-blocking
Think about what naive does. For every output element you touch an entire row of A and an entire column of B. At 256×256, matrix B alone is 128 KB. It doesn't come close to fitting in a 32 KB cache, so you keep re-fetching the same data from PSRAM, over and over, for every single output element.
Tiling reorders the work. You compute a small block of C at a time, and while you do, you only need one tile of A and one tile of B. If those three tiles fit in cache together, every element gets loaded from PSRAM once, reused many times from cache, and that's the whole trick. PSRAM traffic collapses.
#define TILE 32 // 3 tiles x 32^2 x 2 bytes = 6 KB total, fits well in 32 KB cache
for (int ii = 0; ii < n; ii += TILE)
for (int jj = 0; jj < n; jj += TILE)
for (int kk = 0; kk < n; kk += TILE)
// inner 3 loops work on one 32x32 block each, all in cache
for (int i = ii; i < ii+TILE; i++)
for (int j = jj; j < jj+TILE; j++) {
int32_t acc = C[i*n+j];
for (int k = kk; k < kk+TILE; k++)
acc += (int32_t)A[i*n+k] * (int32_t)B[k*n+j];
C[i*n+j] = (int16_t)acc;
}T=32 isn't a guess. Three 32×32 tiles of i16 use 3 × 32² × 2 bytes = 6 KB, comfortably inside the 32 KB cache with room left for the stack and the code.
I also tried T=64, which looked great on paper: 24 KB, fits in 32. It was 2.3× slower at n=256. The catch is that the cache isn't exclusively yours. FreeRTOS, the benchmark harness, the stack and the program instructions all fight for the same 32 KB, so what's actually free for your matrices is much less. T=32 survives the competition because it's conservative. T=64 was on the edge and lost. Which is exactly why you measure instead of calculate.
This one uses Espressif's esp-dsp library, whose hand-optimized dsps_dotprod_s16 runs on the EE.VMULAS.S16.QACC PIE instruction. The 8-wide SIMD, eight 16-bit multiplies per clock.
Before multiplying, B gets transposed into a temporary Bt. That matters: the dot product wants both vectors contiguous in memory, one row of A against one row of Bt. Transposing first makes every inner-loop access sequential, which is precisely what keeps a SIMD unit fed.
Only i16 gets this path. PIE has no equivalent instruction for 32-bit integers or floats, so those fall back to the same scalar code as opt_O3. That's why the i32 and f32 numbers for the two are identical.
.option directive. That is RISC-V syntax and does not exist in the Xtensa toolchain. GCC's inline assembly constraint system does not recognize Q registers (the PIE vector registers) as named clobbers, because they are TIE extension registers that exist outside GCC's standard model of the machine. After several failed attempts, the decision was made to use Espressif's own esp-dsp library, which wraps all of this correctly and is what Espressif themselves recommend for production use. The file is called matmul_espdsp.c rather than matmul_asm.c to be honest about what it actually does.
Results
Everything measured at 240 MHz, every data point the median of 10 runs. Timing uses CCOUNT, a hardware counter that ticks every clock cycle, cross-checked against esp_timer, which agreed at 239.8 cycles per microsecond.
i16 benchmark, raw cycles
| n | naive_O0 | opt_O3 | tiled_O3 | espdsp |
|---|---|---|---|---|
| 8Ć8 | 25,288 | 5,994 | 4,441 | 6,648 |
| 16Ć16 | 190,248 | 41,458 | 26,745 | 22,256 |
| 32Ć32 | 1,481,994 | 312,066 | 187,321 | 98,248 |
| 64Ć64 | 11,694,619 | 2,427,644 | 1,498,096 | 516,551 |
| 128Ć128 | 103,655,528 | 29,446,056 | 13,270,140 | 4,541,825 |
| 256Ć256 | 2,438,528,277 | 2,058,759,211 | 106,912,458 | 148,902,291 |
Speedup over naive_O0 (i16)
| n | opt_O3 | tiled_O3 | espdsp |
|---|---|---|---|
| 8Ć8 | 4.2Ć | 5.7Ć | 3.8Ć |
| 16Ć16 | 4.6Ć | 7.1Ć | 8.5Ć |
| 32Ć32 | 4.7Ć | 7.9Ć | 15.1Ć |
| 64Ć64 | 4.8Ć | 7.8Ć | 22.6Ć |
| 128Ć128 | 3.5Ć | 7.8Ć | 22.8Ć |
| 256Ć256 | 1.2Ć | 22.8Ć | 16.4Ć |
The opt_O3 speedup collapsing from 4.8× at n=64 to 1.2× at n=256 is my favourite number in the entire benchmark. The compiler didn't get worse. The data left the cache.
Speedup breakdown at 128Ć128 i16, the sweet spot
256Ć256 across all data types
| impl | i16 | i32 | f32 |
|---|---|---|---|
| naive_O0 | 2,438M | 2,715M | 2,439M |
| opt_O3 | 2,058M | 2,072M | 2,060M |
| tiled_O3 | 106M | 442M | 430M |
| espdsp | 148M | 2,072M | 2,072M |
Notice i32 and i16 running neck and neck for naive and opt. At 256×256 everyone is bottlenecked on PSRAM, and data type stops mattering when you spend your life waiting for memory. For tiled, i16 is 4× faster than i32 and f32, because smaller elements mean more of them per tile and better cache use per byte. For espdsp, i32 and f32 have no SIMD path at all, so they run exactly as fast as opt_O3.
What the numbers actually say
The compiler is powerful, until the data leaves the cache
From n=8 to n=64, -O3 delivers a steady 4.7 to 4.8×. That's real work: unrolling, scheduling, register allocation, all of it earning its keep.
Then at n=128 it drops to 3.5×. At n=256, 1.2×. The compiler didn't change. What changed is that three 256×256 matrices occupy 192 KB, blowing past the 32 KB cache and even the 512 KB SRAM. Every read of B in the inner loop goes to PSRAM at 40 cycles a trip. The CPU is stalled roughly 97% of the time.
The lesson: compilers optimize instructions. When your bottleneck is memory latency, instruction cleverness has nothing left to grab. The compiler cannot rewrite your algorithm to touch memory in a better order. That part is your job.
Memory layout mattered more than hardware acceleration
This one genuinely surprised me. Tiling delivers 22.8× over naive and 19× over -O3 at 256×256. The dedicated SIMD hardware also tops out at 22.8× over naive at its best. Same headline number, completely different roads: tiling gets there by fixing the memory bottleneck, SIMD gets there only where the memory bottleneck already doesn't exist.
Put bluntly: if you could apply exactly one optimization to a large matmul on this chip, the right answer is cache-blocking, not SIMD. No special hardware, no library. Just reorganizing the computation to respect the memory hierarchy.
This is the memory wall, a very general phenomenon wearing a three-dollar costume. The gap between compute speed and memory bandwidth has been widening for decades, and on constrained embedded hardware, memory access patterns dominate performance more than anything else does.
There is a cliff at 256Ć256 that looks nothing like normal scaling
Matmul is O(n³): double the size, eight times the work. That's exactly what shows up at every size transition up to 128, with naive scaling at 7.5 to 8.9×, right on theory. Then from 128 to 256 it scales at 23.5×. Nearly three times worse than the math says it should.
| Transition | Actual scaling (naive i16) | Theoretical (O(nĀ³)) |
|---|---|---|
| 8 ā 16 | 7.5Ć | 8Ć |
| 16 ā 32 | 7.8Ć | 8Ć |
| 32 ā 64 | 7.9Ć | 8Ć |
| 64 ā 128 | 8.9Ć | 8Ć |
| 128 ā 256 | 23.5Ć | 8Ć |
Because three 128×128 matrices just barely fit inside the 512 KB SRAM, and the 256×256 ones don't. They spill wholesale into PSRAM. That's the memory cliff, and it's visible in every implementation except tiled, which was built specifically to step around it. The algorithm didn't get worse at the cliff. The hardware underneath it changed.
PIE SIMD gives 2.9x over tiling, but only when data is in cache
At n=64 and n=128, where the working set fits and the SIMD unit stays fed, espdsp runs 2.9× faster than tiled. That's a real hardware win: 4.5M cycles against 13.3M at 128×128, done in 18.9 ms at 240 MHz. Fast enough for keyword spotting, gesture classification, and small sensor models in real time.
The theoretical max from 8-wide SIMD is 8×. The measured win is 2.9×, about 37% efficiency. The missing factor goes to real costs: transposing B first (overhead that grows with n²), calling the dot product once per output element (n² calls), the function call overhead on each, and the loop bookkeeping around the SIMD instructions. A serious kernel would batch many outputs per call. But even at 37%, the hardware is clearly pulling.
SIMD loses to tiling at 256Ć256, and this is worth understanding
At n=256, espdsp (148M cycles) is noticeably slower than tiled (106M). Which feels wrong at first. Shouldn't hardware always help? Sit with it for a minute and it makes complete sense.
At 256 the matrices live in PSRAM. espdsp calls the dot product 65,536 times, once per output element, paying prologue, epilogue and argument passing on every call, plus the transpose's access pattern. And the SIMD unit stalls on PSRAM exactly like the scalar code does. It can multiply 8 numbers at once. It cannot make a 40-cycle memory fetch take less than 40 cycles.
Tiling wins because it went after the root cause and starved the PSRAM traffic. SIMD sped up the computing and left the waiting untouched.
A SIMD engine is only as fast as the data pipeline feeding it.
What this means for the AI claim
So, is the ESP32-S3 AI-capable? Based on these measurements: yes, honestly so, with limits that are worth stating just as honestly.
Up to 128×128 with i16 data, the chip does 22.8× over naive and finishes a full multiplication in under 20 ms. That is genuinely enough for:
- Keyword spotting (small audio classification models)
- Gesture detection (IMU-based, lightweight)
- Anomaly detection on sensor streams
- Small image classifiers on low-resolution inputs
Past 128×128 it hits the memory wall. PSRAM cannot feed the CPU no matter how the code is arranged. At 256×256 tiled still holds its 22.8× over naive, but the absolute numbers get ugly, and real models with bigger layers will make them uglier.
The S3 is not competing with inference accelerators or a Raspberry Pi, and it was never trying to. It was built for small, always-on inference at milliwatt power and a three-dollar price. At that job, the label on the box is earned.
And the deeper lesson has nothing to do with this particular chip. On any hardware where compute outruns memory, the bottleneck is almost never the CPU. Reorganizing memory access beat dedicated SIMD hardware in this benchmark. Thinking about where data lives and how it moves is the most valuable performance skill on constrained hardware, full stop.
The things that went wrong
Two things ate more time than the actual benchmarking. I'm writing them down because they're exactly the kind of thing tutorials never mention.
GCC 14.2.0 internal compiler error
Early on, the build kept dying with an internal compiler error deep in GCC's IRA (Integrated Register Allocator) pass, triggered by esp_lcd_panel_rgb.c. It only fired with one specific combination: -mdisable-hardware-atomics (needed for octal PSRAM), plus -Og, plus having esp_lcd in the build. Three innocent things, one crash.
The fix was to throw esp_lcd out of the build in CMakeLists.txt, since the project never uses it. It's a known bug in that version of the Xtensa toolchain. The compiler itself was crashing, not my code. The lesson: before you spend hours debugging your own files, check whether the error is even coming from your side of the fence.
Xtensa PIE inline assembly turned out to be a dead end
The original plan was to write the SIMD path in inline assembly, calling PIE instructions directly. That hit two walls in about ten minutes. The Xtensa assembler doesn't know the .option directive, which turns out to be RISC-V syntax. And GCC's inline assembly needs you to declare which registers you touch, but the PIE vector registers (Q0 to Q7) are TIE extension registers that live entirely outside GCC's machine description. There is no constraint string for them. As far as GCC is concerned, they do not exist.
After trying standalone assembly files, intrinsics and different toolchain versions, the honest conclusion was that Espressif's own esp-dsp library had been the right answer from the start. It's production-tested, maintained, and wraps the PIE instructions correctly. The implementation file is named matmul_espdsp.c on purpose: it uses the library, not hand-written assembly, and I'd rather the filename say so.