Skip to content

Vww eval akida #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dmclellandbc wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Vww eval akida #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
91bf1d5
Clean skip benchmark if no device
dmclellandbc Jul 10, 2026
6f188bd
feat(VWW): Aligned to plant_village template, updated models for new …
dmclellandbc Jul 10, 2026
ada67a7
fix(env): pyftdi added to pyproject.toml
dmclellandbc Jul 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion akida1/model_zoo/plant_village/plant_village_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import argparse
import json
import pathlib
import sys
import time

import numpy as np
Expand Down Expand Up @@ -48,7 +49,9 @@
# Device
# -------------------------------------------------------------------------
device = get_akida_device(target_version=ak_model.ip_version)
assert device is not None, 'No compatible Akida hardware device found.'
if device is None:
sys.exit('No compatible Akida hardware device found. Skipping benchmarking')

# TODO: Add a check to get clock frequency specific to device
CLOCK_FREQUENCY = 400e6 # 400 MHz for AKD1500

Expand Down
3 changes: 3 additions & 0 deletions akida1/model_zoo/plant_village/plant_village_eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import json
import pathlib
import numpy as np
import tensorflow as tf
from tqdm import tqdm

import akida
Expand All @@ -19,6 +20,8 @@
from plant_village_data import get_data
from brainchip_utils.hardware_utils import get_akida_device

tf.config.experimental.enable_op_determinism()

# ---------------------------------------------------------------------------
# Evaluation on Akida
# ---------------------------------------------------------------------------
Expand Down
9 changes: 5 additions & 4 deletions akida1/model_zoo/plant_village/plant_village_train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,18 @@ def train_plant_village(model, train_ds, val_ds, epochs, learning_rate, regulari
# ---------------------------------------------------------------------------
# Model
# ---------------------------------------------------------------------------
model.compile(optimizer=Adam(learning_rate=learning_rate),
loss=SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])

if regularization is not None:
print('Adding Activity Regularization to ReLU layers')
regularizer = regularizers.L1L2(regularization, regularization)
for layer in model.layers:
if isinstance(layer, ReLU):
layer.activity_regularizer = regularizer

model.compile(optimizer=Adam(learning_rate=learning_rate),
loss=SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])


# ---------------------------------------------------------------------------
# Training
# ---------------------------------------------------------------------------
Expand Down
197 changes: 51 additions & 146 deletions akida1/model_zoo/vww/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,7 @@

# Visual Wake Words (VWW)


## Dataset

![Sample VWW images (MS-COCO 2014, 96×96)](docs/vww_sample_mosaic.png)

Visual Wake Words is a binary image classification benchmark, specifically
designed to target edge deployment on resource-constrained devices. It is
derived from the MS-COCO 2014 dataset. Each image is labelled **person**
or **non-person** based on whether a person occupies at least 2% of the frame.
Images are resized to **96 × 96 RGB**. The dataset contains approximately
115k training images and 8k validation images.

Reference: Chowdhery et al., *Visual Wake Words Dataset* (2019),
[arXiv:1906.05721](https://arxiv.org/abs/1906.05721).

## Model
## Model Card

<table>
<thead>
Expand All @@ -31,9 +16,9 @@ Reference: Chowdhery et al., *Visual Wake Words Dataset* (2019),
</thead>
<tbody>
<tr>
<td align="center">87.01%</td>
<td align="center">84.65%</td>
<td align="center">84.68%</td>
<td align="center">89.33%</td>
<td align="center">88.54%</td>
<td align="center">88.42%</td>
<td align="center">68.29%</td>
<td align="center">226,906</td>
</tr>
Expand Down Expand Up @@ -104,39 +89,24 @@ sparsity is particularly high take very little processing time.

<img src="docs/ref_benchmark_results_layers.png" alt="Per-layer latency breakdown" width="700">

## Pipeline
## Requirements

Training follows a three-stage quantization pipeline, followed
by conversion to Akida format:
For environment requirements and setup, see the [Requirements](../../../README.md#requirements)
section of the top-level README.

| Stage | Description |
|---|---|
| Full-precision | Float32 training from scratch, 50 epochs |
| Post-training quantization | `cnn2snn quantize` reduces to 4-bit weights and activations (8-bit input) |
| Quantization-aware tuning | 2 epochs fine-tuning of the quantized model to recover accuracy |
| Conversion to Akida | Automated conversion to Akida model format |
## Dataset

## Requirements
This example generated and tested under
```
tensorflow[and-cuda]==2.19.1
tf_keras==2.19.0
akida==2.19.1
quantizeml==1.2.3
cnn2snn==2.19.1
akida_models==1.14.0

ipykernel
pooch
```
![Sample VWW images (MS-COCO 2014, 96×96)](docs/vww_sample_mosaic.png)

The hardware benchmarking (`vww_benchmark.py` and step 9 in the notebook) and results plotting steps
additionally require the `brainchip_utils` package from this repository. Install it into
your environment from the repo root:
Visual Wake Words is a binary image classification benchmark, specifically
designed to target edge deployment on resource-constrained devices. It is
derived from the MS-COCO 2014 dataset. Each image is labelled **person**
or **non-person** based on whether a person occupies at least 2% of the frame.
Images are resized to **96 × 96 RGB**. The dataset contains approximately
115k training images and 8k validation images.

```bash
pip install -v -e .
```
Reference: Chowdhery et al., *Visual Wake Words Dataset* (2019),
[arXiv:1906.05721](https://arxiv.org/abs/1906.05721).

## Dataset setup

Expand All @@ -159,17 +129,44 @@ ln -s /path/to/your/data/vw_coco2014_96 ./data/vw_coco2014_96

This way the scripts work out of the box without any extra arguments.

## Pipeline

Training follows a three-stage quantization pipeline, followed
by conversion to Akida format:

| Stage | Description |
|---|---|
| Full-precision | Float32 training from scratch, 20 epochs |
| Post-training quantization | `cnn2snn quantize` reduces to 4-bit weights and activations (8-bit input) |
| Quantization-aware tuning | 5 epochs fine-tuning of the quantized model to recover accuracy |
| Conversion to Akida | Automated conversion to Akida model format |

## Reference Models

Pretrained models are made available here, within the `pretrained_models/`
folder. However, those are handled using the `git-lfs` package (git large
file storage). For those to be downloaded with the repo, you will need to
set up `git-lfs`. For further instructions, see the
[Trained models](../../../README.md#trained-models) section of the top-level README.

## Usage

### Notebook

[vww_notebook.ipynb](vww_notebook.ipynb) walks through the complete training
pipeline end-to-end. It is written to expose and explain the Akida-specific
Two notebooks are provided that walk through a) preparation of a trained Akida-compatible model and
b) evaluation and benchmarking of that model on Akida.

[vww_notebook_training.ipynb](vww_notebook_training.ipynb) walks through the
complete training pipeline end-to-end. It is written to expose and explain the Akida-specific
aspects of the workflow: how the model is constructed for Akida compatibility,
what the quantization constraints mean in practice, and what the conversion
step does. Start here if you want to understand *why* the pipeline is structured
the way it is.

[vww_notebook_benchmark.ipynb](vww_notebook_benchmark.ipynb) walks through
evaluation of model accuracy on Akida and, if a hardware device is available, covers benchmarking
of model latency and power.

### Script

For straightforward reproduction of the training and evaluation results, run
Expand All @@ -182,99 +179,7 @@ bash vww_train.sh [DATADIR]
The optional `DATADIR` argument overrides the default dataset location
(`./data/vw_coco2014_96`).

The script executes the following steps in order. This will take from
1-2 hours to run if a modern GPU is available, almost all of which is
the 50 epochs of initial training. If you want to save time and focus on
the Akida-evaluation steps only, you can skip directly to step 5b,
which downloads a pretrained, quantized model directly from the BrainChip
servers:

**1. Build the model**
```bash
python vww_model.py -s models/akidanet_vww_untrained.h5
```
Instantiates AkidaNet (alpha = 0.25, 96 × 96 input) and saves the untrained
weights.

**2. Float training**
```bash
python vww_train.py -l models/akidanet_vww_untrained.h5 \
-s models/akidanet_vww.h5 \
-e 50 -lr 1e-3
```
Trains from scratch for 50 epochs in full precision (Float32).

**3. Float evaluation**
```bash
python vww_eval.py -l models/akidanet_vww.h5
```
Reports validation accuracy of the float model.

**4. Post-training quantization**
```bash
cnn2snn quantize -m models/akidanet_vww.h5 -i 8 -w 4 -a 4
```
Quantizes the model to 8-bit inputs, 4-bit weights, and 4-bit activations,
producing `akidanet_vww_iq8_wq4_aq4.h5`.

**5. Quantization-aware tuning (QAT)**
```bash
python vww_train.py -l models/akidanet_vww_iq8_wq4_aq4.h5 \
-s models/akidanet_vww_qat.h5 \
-lr 1e-4 -e 2
```
Fine-tunes the quantized model for 2 epochs at a lower learning rate to
recover accuracy lost during quantization.

**5b. ALTERNATIVE SHORTCUT: Download pretrained quantized model**
```bash
wget -N https://data.brainchip.com/models/AkidaV1/akidanet/akidanet_vww_iq8_wq4_aq4.h5
mv akidanet_vww_iq8_wq4_aq4.h5 ./models/akidanet_vww_qat.h5
```
To save time by skipping the training steps and focussing on the Akida-specific
evaluation steps, you can download the pretrained, quantized model directly from
BrainChip's servers.

**6. Quantized evaluation**
```bash
python vww_eval.py -l models/akidanet_vww_qat.h5
```
Reports validation accuracy of the quantized model.

**7. Conversion to Akida format**
```bash
cnn2snn convert -m models/akidanet_vww_qat.h5
```
Converts the quantized Keras model to the Akida `.fbz` format ready for
on-chip deployment.

**8. Akida evaluation**
```bash
python vww_eval.py -l models/akidanet_vww_qat.fbz
```
Reports validation accuracy running inference through the Akida model,
confirming parity with the quantized Keras result. This evaluation step
is done using an Akida 1 hardware device if one is connected. Otherwise
it will fall back to using the Akida software backend, that is, you can
check the final accuracy of an Akida model without the need for access
to a hardware device.

**9. Akida benchmark**
```bash
python vww_benchmark.py -l models/akidanet_vww_qat.fbz
```
This will only run if an Akida 1 hardware device is connected. It benchmarks
the inference latency across a large number of samples (but at batch-size=1)
and if power measurement is available on the machine, records total and
dynamic power consumption, and energy per inference. These benchmarks are run
in both `Minimal` and `AllNps` mapping modes.

Following this, a second set of benchmarks are run (in `Minimal` mapping mode)
which break down the latency per layer The activation sparsity per layer is
also recorded.

Benchmark results are recorded in two plots, `benchmark_results_full.png` and
`benchmark_results_layers.png`.
That will take about 1 hour to run if a modern GPU is available.

## Contributing and Maintenance

Expand All @@ -288,10 +193,10 @@ and Akida model versions, plus the hardware benchmark, including the
`--save-metrics` argument, and then regenerate the README from the template
using `update_readme.py`:
```bash
python vww_eval.py -l models/akidanet_vww.h5 --save-metrics
python vww_eval.py -l models/akidanet_vww_qat.h5 --save-metrics
python vww_eval.py -l models/akidanet_vww_qat.fbz --save-metrics
python vww_benchmark.py -l models/akidanet_vww_qat.fbz --save-metrics
python vww_eval.py -l pretrained_models/akidanet_vww.h5 --save-metrics
python vww_eval.py -l pretrained_models/akidanet_vww_qat.h5 --save-metrics
python vww_eval.py -l pretrained_models/akidanet_vww_qat.fbz --save-metrics
python vww_benchmark.py -l pretrained_models/akidanet_vww_qat.fbz --save-metrics
python update_readme.py
```
Then commit the changed files (template, metrics and updated README).
Expand Down
Loading