Each pixel measures how much light fell on it. Today: how those measurements are made, what they cost in bytes, and what else can "see" a scene without a normal camera.
Theory runs to slide 12; the lab is one block at the end (slides 13 to 15). The kit has no camera, so the lab uses the laptop webcam in Colab: test it on the classroom machine before class.
last class's exit ticket
A pump with faults up to 2 kHz.
wide-band
vibration sensor or piezo
> 4 kHz
sample rate, twice 2 kHz
260 Hz
the MPU6050's widest filter
Today the same question for light: what does the sensor actually capture, and how much of it can the board afford?
Ask a team for the sample rate they measured in last lab's Part B; it is usually far below 4 kHz.
Which of these does the doorbell's model actually need?
The green square is genuinely that small at this scale. Ask why we would ever pay for the 12 MP sensor if the model sees 96 × 96. (Clips for humans; zoom; quality.) 36 MB = 4000 × 3000 × 3 bytes.
Each pixel is a photodiode collecting charge.
More light, more charge. After the exposure time, each pixel's charge is read out, amplified and digitised: Class 10's chain, repeated millions of times.
Resolution is just how many pixels. Each pixel is still a single light meter.
The readout happens row by row on most cheap sensors. Hold that thought for two slides.
A 2 MP sensor is 1.92 million of these light meters, each with its own reading per frame.
A colour filter over each pixel. Each pixel sees only one colour.
A photodiode cannot tell colours apart. So each one gets a red, green or blue filter, in a repeating 2 × 2 pattern.
The missing two colours at every pixel are guessed from the neighbours (demosaicing). The full-colour image is partly interpolated.
Twice as many green pixels, because our eyes are most sensitive to green. And a greyscale model throws the colour away again: one byte per pixel instead of three.
The point for ML: a colour image has three times the bytes but not three times the information for many tasks. Person detection works well in greyscale.
try it live · a vertical pole moving left to right
Rows read one after another bend fast motion.
shutter
motion
Each band is one row of pixels, read slightly later than the one above. In a rolling shutter the pole has moved on by the time the lower rows are read, so it slants. Global shutters expose every row at once but cost more. Speeds here are illustrative.
Exposure time: the trade between noise and blur.
short exposure, dim light
Freezes motion, but too little light: dark and noisy.
long exposure, person moving
Enough light, but the person smears across the frame.
enough light
Short exposure and plenty of light: the case your training data probably came from.
Simulated images. The doorbell works at night, so its model will see the first two far more often than the third. That is training–serving skew from Class 6. Many night cameras add infrared LEDs and drop colour.
try it live
Resolution × bytes per pixel × frame rate.
size
format
fps
one frame vs SRAM · 512 KB
one frame vs PSRAM · 8 MB
Bytes per pixel for JPEG is a rough average (about 10:1 against 24-bit colour); real JPEG size varies a lot with the scene and quality, which the lab measures. Good sequence: QVGA grey 10 fps (fine), VGA RGB565 15 fps (the exit ticket), UXGA RGB565 (does not fit SRAM, barely PSRAM).
JPEG is small to send, but a model cannot read JPEG.
The OV2640 class of sensor can output JPEG directly, which is why camera boards are popular for streaming. For inference, the decode step is pure overhead.
try it live
How few pixels before you cannot tell there is a person?
Most microcontroller person detectors work at around 96 × 96. The question is not "how sharp", it is "how small can it be and still be right".
Smaller input means a smaller first layer, fewer activations, less SRAM (Class 7). Resolution is a design choice, not a given.
A simulated doorway scene. Ask the room at each size: person or no person? Most can still tell at 24 × 24; at 12 × 12 opinions split. A model is similar, but it also needs enough pixels to tell a person from a coat on a hook.
Four ways to see the same doorway.
greyscale camera
Brightness only. The default for TinyML vision.
thermal array · 32 × 24
Heat, not light. Works in the dark; no face to recognise.
depth · time-of-flight or stereo
Distance per pixel. Gestures, obstacles, people counting.
event camera
Only pixels that changed, as events. Very fast, very low data when nothing moves.
All four images are simulated from the same scene. 32 × 24 is a common resolution for low-cost thermal arrays; check specific parts. Event cameras are still mostly a research and industrial tool.
Sometimes the best camera is the one that cannot see a face.
a hospital ward at night
Count people, detect someone leaving a bed, without ever recording who they are. A thermal array or radar (Class 14) does the job; a camera creates a privacy problem.
the doorbell
Small greyscale frames, woken by a PIR sensor, feed the person detector. A colour JPEG clip is made only when someone rings, and only then leaves the device.
Choosing the sensor is a privacy decision, made long before any model is trained. That is P from BLERP, decided in hardware.
End of theory. Move the class to laptops for the lab block.
lab · teams of four · one laptop per team
No camera in the kit, so the laptop is the camera.
25:00
1
Capture one frame
In Colab, open the Code snippets panel, insert "Camera Capture" and run it. It saves photo.jpg. Browser blocks the webcam? Upload a phone photo instead.
2
Measure it
Run the next slide's cell: bytes as raw RGB, RGB565, greyscale and JPEG at two qualities; time a resize to 96 × 96.
3
Look at what the model sees
View your 96 × 96 greyscale image enlarged. Then try 48 and 24. Fill the worksheet.
The "Camera Capture" snippet is in Colab's Code snippets sidebar (the </> icon); it asks for webcam permission. Test it on the classroom network before class. Timings in step 2 are on the laptop, not the S3; that is fine, the byte counts are the lesson.
# Colab · after "Camera Capture" has saved photo.jpgfrom PIL import Image
import numpy as np, io, time
img = Image.open('photo.jpg').convert('RGB')
w, h = img.size
print('size ', w, 'x', h)
print('raw RGB888 ', np.asarray(img).nbytes, 'bytes')
print('as RGB565 ', w * h * 2, 'bytes')
grey = img.convert('L')
print('greyscale ', np.asarray(grey).nbytes, 'bytes')
for q in (90, 30):
buf = io.BytesIO(); img.save(buf, 'JPEG', quality=q)
print(f'JPEG q={q} ', buf.tell(), 'bytes')
t = time.perf_counter()
for _ in range(100):
small = grey.resize((96, 96))
print('resize ', (time.perf_counter() - t) * 10, 'ms each')
display(small.resize((384, 384), Image.NEAREST))
lab · steps 2 and 3
Count the bytes. Then look.
Write every printed number into the worksheet.
Change (96, 96) to 48 and 24 and run the last two lines again. When does your face stop being a face?
Bonus: point the webcam at a busy bookshelf and a blank wall. Why do the JPEG sizes differ so much?
The resize time is the 100-iteration total in seconds × 1,000 ÷ 100, i.e. × 10 ms. The bonus answer: JPEG compresses smooth areas well and detailed ones poorly, which is why the calculator's JPEG figure is only a rough average.
lab · fill these six boxes as a team
Your frame, your numbers.
frame size
width × height
raw RGB888
bytes; fits in S3 SRAM?
greyscale
bytes; how many times smaller?
JPEG q90 vs q30
bytes; ratio to raw RGB
96 × 96 greyscale
bytes; and is it still you?
verdict
which format would you send, and which would you feed the model?
Two teams read out their raw and JPEG numbers. Different webcams give different sizes; the ratios are what to compare.
exit ticket · on paper, before you leave
A team wants to count cars at 15 frames a second at VGA RGB565 on the ESP32-S3. Calculate the data rate. What would you change, and why?
Next class: motion, position, force and touch. The MPU6050 again, and the S3's own touch pins.
Answer: 640 × 480 × 2 × 15 = 9,216,000 bytes ≈ 8.8 MB per second, and each 600 KB frame does not fit in SRAM. Better: greyscale, QVGA or smaller, fewer frames per second (cars do not need 15), or count only when motion is detected.
