Skip to content

Performance Testing & Benchmarking

Test your Loss Prevention and Automated Self-Checkout pipeline performance on various hardware configurations. This guide covers everything from quick performance checks to comprehensive system capacity testing.

Quick Start (5 minutes)

Goal: Run a basic performance test to verify your system works correctly

1. Initialize Performance Tools

make update-submodules

2. Run Quick Benchmark

make benchmark-quickstart

What this does:

  • Benchmarks the six workloads in the default camera configuration using the default mixed CPU/GPU/NPU workload mapping
  • Runs headless (no display needed)
  • Uses registry images by default (REGISTRY=false builds the benchmark image locally)
  • Generates consolidated metrics in benchmark/metrics.csv

Expected results: You'll see FPS metrics and resource utilization for each workload.

Understanding Performance Testing Types

Basic Performance Testing

Default Benchmark Command:

make benchmark

Configuration:

  • Single pipeline instance (PIPELINE_COUNT=1)
  • Mixed CPU/GPU/NPU processing (WORKLOAD_DIST=workload_to_pipeline.json)
  • Standard camera setup (CAMERA_STREAM=camera_to_workload.json)
  • No visual output (RENDER_MODE=0)

Environment Variables Reference

Category Variable Description Common Values
Display RENDER_MODE Show/hide visual output 0 (headless), 1 (visual)
Performance PIPELINE_COUNT Number of parallel pipeline instances 1, 2, 4 (higher = more stress)
Hardware WORKLOAD_DIST Target processing hardware workload_to_pipeline_cpu.json, workload_to_pipeline_gpu.json, workload_to_pipeline_gpu-npu.json, workload_to_pipeline_hetero.json
Camera Setup CAMERA_STREAM Camera configuration camera_to_workload.json
Build REGISTRY Use pre-built vs local images true (faster), false (custom builds)

Workload Configuration Options

Camera Stream Configurations

Full Workload Testing (camera_to_workload.json):

Camera Workload
cam1 items_in_basket
cam2 hidden_items
cam3 fake_scan_detection
cam4 multi_product_identification
cam5 product_switching
cam6 sweet_heartening

Hardware Distribution Options

Configuration File Best For
CPU Only workload_to_pipeline_cpu.json Testing, development environments
GPU Only workload_to_pipeline_gpu.json Production, high performance
Mixed GPU/NPU workload_to_pipeline_gpu-npu.json Assign selected workloads to GPU or NPU
Heterogeneous workload_to_pipeline_hetero.json Maximum performance across all hardware
Default Mixed workload_to_pipeline.json CPU/GPU/NPU distribution used by default

Advanced Performance Testing (15-30 minutes)

GPU Performance Testing

make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json

Multi-Pipeline Stress Testing

# Test with 2 parallel pipelines
make PIPELINE_COUNT=2 benchmark

# High stress test with 4 pipelines
make PIPELINE_COUNT=4 benchmark

Custom Hardware Configuration

# Test heterogeneous workload distribution
make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false

Automated Self-Checkout Performance

# Object detection workload
CAMERA_STREAM=camera_to_workload_asc_object_detection.json WORKLOAD_DIST=workload_to_pipeline_asc_object_detection_gpu.json make benchmark

# Age verification workload
CAMERA_STREAM=camera_to_workload_asc_age_verification.json WORKLOAD_DIST=workload_to_pipeline_asc_age_verification_gpu.json make benchmark

Viewing Results

Generate Consolidated Metrics

make consolidate-metrics

Output: benchmark/metrics.csv containing:

  • FPS (frames per second) for each pipeline
  • CPU/GPU/NPU utilization percentages
  • Memory usage statistics
  • Power consumption data
  • Latency measurements

View Results

cat benchmark/metrics.csv

Stream Density Testing (30+ minutes)

Goal: Find the maximum number of parallel pipelines your system can handle while maintaining target performance.

Basic Stream Density Test

make benchmark-stream-density

Default behavior:

  • Requires every stream to reach at least 95% of its effective target FPS; by default targets come from each camera's targetFps, then fps, with a fallback of 15.
  • Uses OOM protection to prevent system crashes
  • Reports maximum sustainable pipeline count

Custom Target FPS

Set the target FPS for each camera using the targetFps field in the selected camera_to_workload_*.json file. Choose the value based on workload criticality and the source FPS.

If targetFps is not defined for a camera (or is not a positive value), the benchmark uses that camera's fps field instead. If neither field provides a positive value, it falls back to the default target FPS of 15.

To override the camera configuration, set the TARGET_FPS environment variable. When supplied, TARGET_FPS applies the same target to every stream and takes precedence over both targetFps and fps.

In summary, the effective target FPS for each camera is resolved in this order:

  1. TARGET_FPS environment variable, when explicitly supplied (overrides the camera configuration for every stream).
  2. The camera's targetFps in camera_to_workload_*.json.
  3. The camera's fps, used when targetFps is missing or not a positive value.
  4. The default target FPS, 15, when neither camera field provides a positive value.

The benchmark passes when each stream reaches at least 95% of its effective target. The resolved target FPS and its source are recorded in stream_density.log.

CLI examples:

# Test for different performance thresholds
make TARGET_FPS=13.5 benchmark-stream-density

# With explicit camera and workload configurations
make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density

Measurement interval and warmup period

MEASUREMENT_WINDOW_SECONDS controls how long FPS samples are collected for each density step. Its default is 100 seconds, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing measurement interval, the final count is confirmed by run acceptance criterion (two consecutive passes), and a lane is only given up after two failing measurement interval in a row. Each measurement interval is evaluated independently; measurement interval are not combined into one longer measurement interval.

INIT_DURATION sets the warmup period, defaulting to 10 seconds, before measurement starts. A longer warmup period lets the pipelines stabilize more, and a longer measurement interval can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved warmup period is also printed in the stream-density result summary.

make benchmark-stream-density \
  INIT_DURATION=120 \
  MEASUREMENT_WINDOW_SECONDS=120

Stream Density Environment Variables

Variable Description Values
TARGET_FPS Override the target FPS for all streams; unset uses camera configuration Unset by default, 13.5, 20.0
MEASUREMENT_WINDOW_SECONDS Duration FPS samples are collected per density step (minimum 100 seconds) 100 (default), 120
INIT_DURATION warmup period before measurement starts 10 (default), 120, 60
OOM_PROTECTION Prevent out-of-memory crashes 1 (enabled), 0 (disabled)
PASS_TOLERANCE_RATIO Fraction of each target FPS required to pass 0.95 (default), up to 1.0

Warning: Setting OOM_PROTECTION=0 may crash your system requiring a hard reboot.

Note: stream-density caches camera-target mappings using (config_path, file_mtime) and auto-refreshes when the camera config changes; unit tests for this behavior are in the performance-tools repository at benchmark-scripts/stream_density_test.py.

Expected Output

Stream density completed: use case density <lanes> lanes, stream density <streams> streams. For the detailed report, see benchmark/stream_density.log.

Visualization & Analysis

Generate Performance Graphs

make plot-metrics

Output: benchmark/plot_metrics.png showing:

  • CPU Usage Over Time
  • NPU Utilization Over Time
  • GPU Usage for each GPU device found

Useful Maintenance Commands

make validate-all-configs    # Validate configuration files
make clean-images           # Remove dangling Docker images
make clean-containers       # Remove stopped containers
make clean-all             # Remove all unused Docker resources

Custom Configuration (Advanced)

Creating Custom Workloads

The application is highly configurable via JSON files in the configs/ directory:

Camera Configuration (camera_to_workload.json)

{
  "lane_config": {
    "cameras": [
      {
        "camera_id": "cam1",
        "fileSrc": "sample-media/video1.mp4",
        "targetFps": 12.5,
        "workloads": ["items_in_basket", "multi_product_identification"],
        "region_of_interest": {"x": 100, "y": 100, "x2": 800, "y2": 600}
      }
    ]
  }
}

Pipeline Configuration (workload_to_pipeline.json)

{
  "workload_pipeline_map": {
    "items_in_basket": [
      {"type": "gvadetect", "model": "yolo11n", "precision": "INT8", "device": "CPU"},
      {"type": "gvaclassify", "model": "efficientnet-v2-b0", "precision": "INT8", "device": "CPU"}
    ]
  }
}

To Add Custom Workloads:

  1. Edit configs/camera_to_workload.json to add your camera and assign workloads
  2. Edit configs/workload_to_pipeline.json to define the pipeline for your workload
  3. Place your video files in performance-tools/sample-media/ and update the fileSrc path
  4. Run make validate-all-configs to verify your configuration
  5. Re-run the pipeline: make benchmark

Detailed Hardware Distribution (Reference)

Heterogeneous Configuration Breakdown

The workload_to_pipeline_hetero.json distributes workloads across multiple processing units:

Workload Object Detection Classification Inference
items_in_basket CPU CPU -
hidden_items GPU GPU -
fake_scan_detection GPU GPU -
multi_product_identification NPU NPU -
product_switching CPU CPU -
sweet_heartening NPU - NPU

Mixed Configuration Details

The workload_to_pipeline.json balances workloads across available hardware:

  • CPU: items_in_basket, multi_product_identification, sweet_heartening
  • GPU: product_switching, hidden_items
  • NPU: fake_scan_detection

Project Structure (Reference)

  • configs/ — Configuration files (camera/workload mapping)
  • docker/ — Dockerfiles for containers
  • download-scripts/ — Model and video download scripts
  • src/ — Pipeline runner scripts
  • benchmark/ — Benchmark logs, consolidated metrics, and plots (default RESULTS_DIR)
  • Makefile — Build automation commands

Appendix: System Configuration

Proxy Configuration (If Required)

If your organization requires proxy settings for internet access:

Shell Session Proxy

export http_proxy=http://<proxy-host>:<port>
export https_proxy=http://<proxy-host>:<port>
export HTTP_PROXY=http://<proxy-host>:<port>
export HTTPS_PROXY=http://<proxy-host>:<port>
export NO_PROXY=localhost,127.0.0.1,::1

System-Wide Proxy (/etc/environment)

sudo nano /etc/environment
# Add the following lines:
http_proxy=http://<proxy-host>:<port>
https_proxy=http://<proxy-host>:<port>
HTTP_PROXY=http://<proxy-host>:<port>
HTTPS_PROXY=http://<proxy-host>:<port>
NO_PROXY=localhost,127.0.0.1,::1

Docker Proxy Configuration

Create /etc/systemd/system/docker.service.d/http-proxy.conf:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo nano /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]
Environment="http_proxy=http://<proxy-host>:<port>"
Environment="https_proxy=http://<proxy-host>:<port>"
Environment="no_proxy=localhost,127.0.0.1,::1"

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart docker