The cloud has unlimited memory, the biggest models and nobody worrying about batteries. So why squeeze AI onto a chip with 512 KB?
Start with the recall before this slide if you prefer: read two exit tickets from Class 1 aloud, one good and one almost-right, and fix the almost-right one together.
last class
Last class was about where. Today is about why.
edge AI
Running AI inference on, or near, the device that collects the data.
train vs infer
Train in the cloud, once. Infer at the edge, all the time.
Two minutes only. If someone still says "edge AI means no internet", correct it here: most products are hybrids.
The BrickML could have streamed every vibration to the cloud. Why didn't it?
Which would you build? Give me reasons. As many as you can.
Ask, then stop talking. Write every answer on the board in a single column, unsorted. You sort them into five buckets on the next slide; most rooms find four of the five without help.
five reasons, one word
B
bandwidth
L
latency
E
economics
R
reliability
P
privacy
This framework comes from AI at the Edge by Situnayake and Plunkett, our main textbook. Every edge AI product you will ever see is justified by at least one of these letters.
Reveal one letter at a time and point at the board: which of their answers goes under this letter? By P, the board should be fully sorted. Anything left over is worth discussing.
B · bandwidth
Sensors produce more data than networks can carry cheaply.
One 1080p camera streams roughly 4 to 8 megabits a second; take 5. The scale is logarithmic: every grid line to the right is 1,000 times more data.
Do each sum on the board before its bar appears: 5 Mb = 625 KB; × 3,600 ≈ 2.25 GB; × 24 ≈ 54 GB; × 1,000 ≈ 54 TB. The green bar is the punchline: a person detected at the gate is one short line of text.
L · latency
Some decisions cannot wait for a round trip.
A request to the cloud and back typically takes tens to a few hundred milliseconds, and it changes with the network.
At 100 km/h a car covers 27.8 metres every second. While it waits for an answer, it keeps moving.
Lane keeping, a drone dodging a wall, a machine guard stopping a blade: the answer is useless if it arrives late.
Derive 27.8 m/s with them: 100,000 m ÷ 3,600 s. Then multiply by 0.1 s and 0.3 s. The bars are to scale with each other.
E · economics
The cloud bills you every time. A chip is paid for once.
A sensor that checks a motor once a second makes this many decisions a year:
1 × 60 × 60 × 24 × 365 = 31,536,000
Now suppose each cloud call cost just ₹0.01. That is ₹3.15 lakh per sensor, every year.
₹0.01 is a made-up round number to make the sum easy. Real prices vary; the shape of the chart does not.
The cloud line rises by the same amount every year, forever. The chip line is so low it sits on the axis. That gap is the whole argument. Then multiply by a factory of 1,000 motors.
R · reliability
The network is not always there. The machine keeps running anyway.
Farms, mines, ships, tunnels, basements, flood zones. And devices that must never go blind: an insulin pump, a fall detector, a smoke alarm.
Ask how many of them lost mobile signal in the last week. Then: would you want your grandmother's fall detector to look like the middle row?
P · privacy
Data that never leaves the device cannot leak from a server.
The law is moving the same way: India's Digital Personal Data Protection Act, 2023 puts duties on whoever collects personal data. Collecting less of it is the simplest way to comply.
Ask which of the two they would put in their own child's room. Keep the legal point brief and general; the engineering point is "minimise what leaves the device".
A product does not need all five. One strong letter is enough.
product
B
L
E
R
P
Smart speaker wake word
Car lane keeping
Traffic camera counting cars
Fitness band step counter
Camera in a hospital ward
BrickML on a motor
?
?
?
?
?
Filled = the letter that justifies putting the AI on the device. No row needs all five.
Before each row appears, ask which letters to fill. The BrickML row is deliberate: pairs argue it for two minutes. Good answers: B (continuous vibration from many motors), R (factory floors with poor coverage), E (thousands of machines).
Edge AI against cloud AI, side by side.
dimension
edge AI
cloud AI
where it runs
on or near the sensor
a remote data centre
latency
milliseconds, predictable
tens to hundreds of ms, varies
memory
KB to a few GB
effectively unlimited
model size
KB to a few hundred MB
up to hundreds of GB
power
milliwatts to watts, often battery
mains, at data-centre scale
network
optional
always required
privacy
raw data stays local
raw data leaves the device
cost
hardware once, ~free per inference
pay per call, storage, bandwidth
updating the model
hard: thousands of devices
easy: redeploy one server
Before each reveal, ask the class to fill the row. They will get most of it. Spend the most time on the last row: it is the one that favours the cloud.
The edge is not free. You pay in three ways.
Small
limited compute and memory
The model must be shrunk to fit, and shrinking can cost accuracy. Unit IV is about doing that well.
Scattered
updates are hard
A better model means updating thousands of devices in the field, some of them offline.
Varied
every chip is different
A model tuned for one board may need rework for the next. The cloud runs on a few standard machines.
This slide keeps the course honest. Students should leave seeing edge AI as a trade-off, not a slogan.
When the right answer is the cloud.
the model is huge
A large language model answering open questions will not fit on a microcontroller.
it needs everyone's data at once
Fraud detection compares one card's transaction with millions of others.
the model changes daily
Recommendations retrained every night are easy to redeploy on one server.
none of the letters is strong
Low volume, no rush, nothing private, good network: there is nothing to gain.
Have one student say aloud: "edge AI is a design choice, not a default". It sounds silly; they remember it.
our running example
A smart doorbell. What goes where?
Most real products look like this: a hybrid, split by BLERP.
Ask the class to place each task before revealing the cloud side. The doorbell returns in every class of this unit; today it gets its first architecture decision.
Hands up: edge or cloud?
task
answer
deciding letter
Wake word on a smart speaker
edge
P, E: always on, always private
Translating a full paragraph with a large model
cloud
model too large; no rush
Crop disease from a leaf photo, in a field with no signal
edge
R: there is no network
Monthly sales forecast for a company
cloud
no letter applies
Heart-rhythm alert on a wearable
edge
L, R, P
Real hands, every row, before each reveal. If the room splits on a row, take one argument from each side.
Score it: should this AI live on the device?
Pick a product from the poll. The class votes 0 to 3 on each letter; 3 means "this alone would force it onto the device".
BHow much raw data, and can the network carry it?
LHow fast must the answer arrive to be useful?
EHow many decisions a year, across how many devices?
RWhat happens when the network is down?
PWould anyone mind the raw data sitting on a server?
?And from the other side: will the model actually fit on the device?
cloudCloud: nothing to gain at the edgeedge
Run it live on two products from the poll: the smart-speaker wake word and the monthly sales forecast. Then once with "fit = no" to show where Units II and IV come in.
exit ticket · on paper, before you leave
Pick one app on your phone that uses AI. Should its AI move onto the phone, or stay in the cloud? Name the letter that decides it.
Next class: applications. Every edge AI product is a sensor, a decision, and a reason to decide locally.
Collect the slips. The best two arguments, one for each side, open Class 3.