class 16 · unit II · system-on-chip and deep learning accelerators
When the CPU is not enough.
Last class you measured how many multiply-adds one core can do. Today: the chips that do thousands at once, and how to read their headline numbers without being fooled.
Theory runs to slide 13; the market-map exercise is one block at the end (slides 14 to 16). Print or share the six device cards from slide 14 before class, after checking each figure against the vendor's current page.
last class's exit ticket
Why can int8 be faster, not just smaller?
fewer bytes to move
One byte per weight instead of four. Moving data costs more energy than multiplying it.
more per instruction
Sixteen int8 values fit in a 128-bit register where only four floats would.
Ask two or three teams for their measured int8 rate and write them on the board. You will need that number in the next slide.
Typical answers mention one effect; the full answer has both. Collect the class's measured rates; use a middle value in the hook calculator.
try it live · the doorbell's person detector
A person detector at 10 frames a second: can the S3 keep up?
your S3 rate · M MAC/s
model
frames per second
MobileNet figures are the published multiply-add counts for the 224×224 full-width versions (MobileNetV1 paper: 569 M; MobileNetV2 paper: 300 M). The tiny detector is an order-of-magnitude stand-in for a 96×96 person-detection model. Start with your class's measured rate; the gap with the MobileNets is the motivation for the whole class.
try it live · click a job
A system-on-chip: a dozen specialists on one piece of silicon.
The point: each job goes to the block built for it, because the same job on the big CPU costs far more energy. Every block shares one road to DRAM, the Class 15 bottleneck again. The layout is a generic phone chip, not any specific product.
Your board is a system-on-chip too, just a small one.
ESP32-S3
Raspberry Pi 5 · BCM2712
phone chip
CPU
2 × Xtensa LX7, 240 MHz
4 × Cortex-A76, 2.4 GHz
big and little cores, up to ~8
graphics
none
VideoCore VII GPU
GPU
ML help
vector instructions
none on chip; add-on Hailo module
NPU, often tens of TOPS
memory
512 KB SRAM + 8 MB PSRAM
4–16 GB LPDDR4X
8–16 GB LPDDR5
power
well under a watt
a few watts to ~12 W
a few watts, in bursts
Raspberry Pi's AI Kit and AI HAT+ add a Hailo accelerator over PCIe, which is exactly the "add an NPU beside the SoC" pattern. Power figures are rough ranges; check the board documentation if a student asks for exact values.
Four tricks, all aimed at Class 15's two lessons.
MACs in a grid
hundreds or thousands of multiply-add units working in lockstep
data reuse
load each number once, use it many times inside the grid
low precision
int8 or lower: smaller multipliers, fewer bytes
fixed function
no fetch, decode or loop bookkeeping for common layers
Compare slide 4 of Class 15: six instructions for one multiply-add. An accelerator spends almost all its ticks on the maths.
The next two slides show the first two tricks working; the fourth has a price, on slide 9.
try it live · a 4 × 4 systolic array
Weights sit still; inputs ripple through, one hop per tick.
Each cell holds one weight. Each input enters on the left and is passed to the right every tick, so one value loaded from memory is used by four cells. Partial sums flow downward. The diagonal wave is the "systolic" heartbeat. Counts assume 4 input vectors through the array; the CPU column assumes two loads per multiply-add with no reuse, as in Class 15.
try it live · drag the grid size
The bigger the grid, the less each multiply-add pays for memory.
grid size N × N
CPU · 2 SRAM reads per MAC
N × N grid
Model: an N × N array multiplying N × N matrices loads 2N² numbers and does N³ multiply-adds. Energy uses Class 15's Horowitz figures (5 pJ per SRAM read, 0.23 pJ for an int8 multiply plus add). Real chips also pay for writing results and for DRAM, so treat this as the shape of the curve, not exact numbers.
try it live
One layer the NPU cannot run, and the whole model slows down.
Illustrative timings. The unsupported layer runs on the CPU, and its data has to be copied out of the NPU's memory and back in again. Ask before revealing: which costs more, the slow layer itself or the two copies? This is why vendors publish lists of supported operators, and why Unit IV will choose layers with the hardware in mind.
TOPS: trillions of operations per second. It is just multiplication.
MAC units × 2 operations × clock = peak operations per second
256
MAC units in an Ethos-U55 micro-NPU (largest size)
× 2 × 400 MHz
each MAC counts as a multiply and an add
≈ 0.2 TOPS
about 205 billion operations a second
Your S3 at 20 M MAC/s is 0.00004 TOPS. Note the "× 2": one multiply-add is quoted as two operations, so halve TOPS to get MACs.
256 × 2 × 400 × 10⁶ = 204.8 × 10⁹. Alif quotes "up to 204 GOPS" for its 256-MAC Ethos-U55 at 400 MHz. 20 × 10⁶ × 2 = 4 × 10⁷ ops/s = 0.00004 TOPS: a factor of about 5,000.
try it live · what does a headline TOPS actually buy?
Peel the headline number back to frames per second.
headline TOPS
quoted with sparsity?
utilisation on your model
Sparsity: some vendors (NVIDIA's 67 TOPS is one) quote a figure that assumes half the weights are zero and skipped; most models are not trained that way, so the dense figure (33) is the honest one. The Coral and Hailo figures are dense, so leave "no" for those. Utilisation varies hugely by model and compiler; 30% is a plausible illustration, not a measurement. Model: MobileNetV2 at 0.6 billion operations per frame.
Before you believe a TOPS number, ask four questions.
1
at what precision?
int8? int4? sparse? The same chip can claim very different numbers.
2
what utilisation?
Peak assumes every unit is busy every tick. Real models rarely get close.
3
at what power?
On a battery, TOPS per watt matters more than TOPS.
4
does it run my layers?
An unsupported layer falls back to the CPU (slide 9).
TOPS ÷ watts = TOPS/W · e.g. 4 TOPS ÷ 2 W = 2 TOPS/W
Better still than TOPS: a measured benchmark on a model like yours, in frames per second and joules per inference. MLPerf Tiny and MLPerf Inference Edge publish such results.
our running example
The person detector is too heavy for the S3. Three ways out.
A
shrink the model
Smaller input, fewer channels, int8. Costs some accuracy. Unit IV.
B
run it less often
Only when the PIR fires (Class 14). Costs nothing if people are rare; costs latency.
C
add an NPU
A board or module with an accelerator. Costs money, power and a new toolchain.
Real products often combine A and B. Next: where would C come from? You decide, from the published numbers.
End of theory. Hand out the six device cards, one set per team.
exercise · teams of four · cards and a log–log grid
Place six accelerators on performance against power.
11:00
ESP32-S3
CPU only
perfyour Class 15 rate × 2 ops
power~0.3 W active (estimate: measure it)
source: your lab
Cortex-M55 + Ethos-U55
micro-NPU, e.g. Alif Ensemble
perfup to ~0.2 TOPS
powertens to hundreds of mW (check datasheet)
source: Alif; CNX Software, Dec 2022
Google Coral Edge TPU
USB or M.2 module
perf4 TOPS int8
power~2 W (2 TOPS/W)
source: Coral datasheet
Hailo-8
M.2 module
perf26 TOPS
power~2.5 W typical (vendor)
source: Hailo product page
Jetson Orin Nano 8 GB
Super mode
perf67 TOPS sparse · 33 dense
power15 W or 25 W modes
source: NVIDIA, JetPack 6.2 blog
Phone NPU
e.g. Apple A17 Pro
perf35 TOPS (Apple)
powernot published
source: Apple, Sept 2023
Both axes go up in factors of ten. Draw a diagonal for 1 TOPS/W. Which cards are missing a number you need?
Figures change between product generations: check each against the vendor's current page before printing, and keep the source line. The phone NPU has no published power figure, and that gap is part of the exercise. The ESP32-S3 power is a rough active-board estimate.
reveal after teams have placed their cards · click to add each device
Diagonals are equal efficiency. Up and to the left is better. Hollow markers are estimates.
The S3 point uses the rate typed into slide 3. Expect five orders of magnitude between the S3 and the Jetson. Ask: which device sits on the best diagonal? Which would you choose for a battery doorbell, and which for a shop with four cameras on mains power?
exercise · worksheet
Five answers per team.
1 · efficiency
which device has the best TOPS/W?
2 · battery doorbell
which would you pick, and why?
3 · four cameras, mains power
which now, and why?
4 · missing numbers
what was not on the cards that you needed?
5 · the S3
does it belong on this chart at all? what can it still do?
Question 5 is the bridge back to the S3: it is the cheapest, lowest-power device on the chart, and it still runs keyword spotting and gesture models comfortably. "Cheapest thing first" from Unit I.
exit ticket · on paper, before you leave
Accelerator A has twice the TOPS of accelerator B but uses four times the power. For a battery-powered camera, which would you investigate first, and what else would you want to know?
Next class: FPGAs, ASICs and edge servers. What if you design the chip yourself?
B has twice the TOPS/W, so investigate it first; then ask the four questions (precision, utilisation, supported layers, real benchmark) and whether B is fast enough at all for the frame rate.