Teaching the Pokédex to See
Without the Internet

The Physical AI: "Who's That Pokémon?"

Q-dex has two ways to recognise a Pokémon. One asks the internet. The other thinks for itself.

The Reverse Image Version uploads a photo and queries Google Lens — powerful, covering all 1025 Pokémon, but dependent on a network connection. The PhyAI Challenge mode is the opposite: a neural network we trained ourselves, running directly on the device's processor, that recognises four Pokémon — Bulbasaur, Charizard, Pikachu and Squirtle — from the live camera feed with no internet whatsoever.

This is the literal meaning of Physical AI: intelligence embedded in the physical device, computing locally, in real time, from its own sensors. When you point the camera and the name appears with no network round-trip, that answer was computed on the Arduino itself.

Why On-Device, Not Cloud

The reverse-image mode already works. So why train our own model at all?

Because a challenge about Physical AI is asking a specific question: can the device itself be intelligent? Offloading every decision to a cloud API sidesteps that question. Running a model locally answers it. Four concrete reasons drove the decision:

The Dataset

Four visually distinct Pokémon, ~40 images each, gathered and organised into class folders.

The four classes were chosen deliberately: Bulbasaur (green/blue, quadruped), Charizard (orange, winged), Pikachu (yellow, bipedal) and Squirtle (blue, shelled) are visually separable by colour and silhouette, which is exactly what makes a small-data classifier viable. The dataset was structured so folder names become the class labels:

pokemon_data.zip
  bulbasaur/   (~40 images)
  charizard/   (~40 images)
  pikachu/     (~40 images)
  squirtle/    (~40 images)

Architecture — MobileNetV2 Transfer Learning

We don't teach the model to see from scratch. We borrow a model that already can.

MobileNetV2 is a convolutional network designed specifically for mobile and embedded devices — small, fast, efficient. Pre-trained on ImageNet (over a million photos across 1000 categories), it already knows how to detect edges, textures, shapes and colours. We freeze all of that learned knowledge and train only a small classifier head on top that maps its understanding onto our four Pokémon. This is why the whole thing works on ~160 images instead of the millions you'd otherwise need.

The model, layer by layer

Input (160×160×3)
   │
   ├─ Data augmentation      (train-time only: flip, rotate, zoom, brightness, contrast)
   ├─ MobileNetV2 preprocess (scale pixels to [-1, 1])
   ├─ MobileNetV2 base       (ImageNet weights — frozen in phase 1)
   ├─ GlobalAveragePooling2D (collapse feature maps to a vector)
   ├─ Dropout(0.3)           (regularisation — fights overfitting)
   └─ Dense(4, softmax)      (the only fully-trained head → 4 class probabilities)

Training — Two Phases

Trained on a free Colab T4 GPU, in two deliberate stages.

Before any training, data augmentation is applied so the model never sees the exact same image twice. With so few photos, this is what prevents memorisation:

RandomFlip('horizontal')
RandomRotation(0.15)
RandomZoom(0.15)
RandomBrightness(0.2)
RandomContrast(0.2)

Results

The number that matters: 95.8% validation accuracy on the exported, quantized model.

Accuracy was measured not on the original float model, but on the final int8 model that actually ships on the device — 23 of 24 held-out validation images classified correctly. This is the honest figure, because it's the model that runs on the Arduino, quantization losses included.

Confusion matrix (validation set)

Where the model agrees and disagrees with the truth. The green diagonal is correct predictions; near-perfect separation across all four classes.

Quantization — Making It Fit

A GPU-trained model is too heavy for an embedded CPU. Quantization is what shrinks it.

The trained model uses 32-bit floating-point weights. To run efficiently on the UNO Q's CPU, it's converted to 8-bit integers (int8) — roughly a 4× reduction in size and a large speed-up, at the cost of a little accuracy. Crucially, the converter needs real images to calibrate the integer ranges correctly:

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset  # real training images
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type  = tf.uint8
converter.inference_output_type = tf.uint8

Because quantization can cost accuracy, the exported model is re-validated after conversion — the 95.8% figure above is that post-quantization number, not the float model's.

Deployment on the UNO Q

From a Colab download to a live camera classifier on the device.

Two files travel together: pokedex_phyai.tflite (the model) and labels.txt (the class order — turns a class number back into a name). On the device, inference runs through LiteRT (ai-edge-litert), not full TensorFlow, which would overflow the board's limited storage.

The Other Half: Voice AI

Seeing is only half of it. Q-dex also listens and talks — and, like the vision model, the listening and speaking both happen on the device itself.

Point the camera and the Pokédex recognises a Pokémon. But press the talk button and you can also have a conversation with Professor Oak. That voice assistant is a chain of three AI stages, and two of the three run entirely on the Arduino UNO Q — only the language reasoning in the middle uses the network.

whisper.cpp — offline speech-to-text

What it is: whisper.cpp is a fast, lightweight C++ port of OpenAI's Whisper speech-recognition model. It converts recorded audio into text — entirely on the device, with no internet.

Why we use it: to answer a spoken question, the device first has to turn your voice into text. Running that locally (rather than a cloud speech API) means no round trip, no per-use cost, and the assistant keeps working offline. We use the tiny.en English model — the best accuracy-for-speed trade-off on this hardware: small enough to run quickly on the UNO Q, accurate enough for short questions. Audio is captured with arecord at 16 kHz mono via the plughw:1,0 ALSA device (the raw hw device returns 44.1 kHz and breaks recognition).

eSpeak — offline text-to-speech

What it is: eSpeak is a compact, open-source speech synthesiser that turns text into spoken audio, running locally on the board.

Why we use it: once Oak's reply comes back as text, eSpeak reads it aloud through the PAM8403 amplifier and USB speaker — no cloud voice service, works offline. A physical mute button (GPIO 10) cuts the audio instantly, mid-sentence, so the conversation feels natural and interruptible. While Oak talks, the WS2812 lens LED blinks teal, driven by the ESP32.

Claude — the brain behind Oak

What it is: Claude is an AI language model (an LLM). It is the component that actually writes what Professor Oak says.

Why it's here — and why it's the one cloud step: generating natural, in-character, context-aware answers is beyond what fits on the board, so this single stage runs in the cloud. Claude is trainer-aware — it's given the active trainer's profile, so Oak can reference your real XP, badges and progress rather than generic replies. The honest framing: in Q-dex's voice loop, hearing (whisper.cpp) and speaking (eSpeak) are on-device; only the thinking uses the network.

Summary

The full training notebook — every cell, runnable end-to-end on a free Colab GPU — is included in the repository as pokedex_phyai_train.ipynb.