Computer vision · 2026

Multi-view pedestrian detection with partial labels

Keep multi-camera people detectors accurate when most ground-plane labels are missing, then prove it on simulated crowds and a live Unity demo.

MVDet MODA 22.7 → 81.5 with 60 % of labels dropped (Wildtrack)

Camera 4 of the simulated plaza scene, a crowd on cobblestones in front of a café terrace
Full write-up: problem, architecture, results, training, lessons. One-minute summary. Switch to Deep dive for the full write-up.
Problem
Multi-view detectors (MVDet, SHOT, MVDeTr, EarlyBird, TrackTacular) need every person labelled on the ground plane in every frame. Drop 60 % of those labels and MVDet falls from 86.9 to 22.7 MODA on Wildtrack, MVDeTr to 0.
Approach
An Ablation Loss (AABL) for BEV heatmaps that stops punishing confident predictions on unlabelled background, plus detector pseudo-labels. Two Unity-simulated crowd datasets with perfect calibration to test transfer, and a server + browser + Unity demo that runs every checkpoint live.
Result
At 60 % dropped labels: MVDet 22.7 → 81.5, SHOT 4.9 → 66.7, MVDeTr 0.0 → 76.1 MODA on Wildtrack, across all five architectures. Simulated sets reproduce real crowd behaviour that MultiviewX lacks (38 % standing, 54 % in groups on the plaza, against 0 % and 8 %). Live demo at 2.3 Hz on a 6 GB laptop GPU.

What I did

The code is private while the work is under review for SAC 2027 and will be released afterwards. The numbers below are copied from the experiment spreadsheet, the training logs and the demo UI.

  • Loss function, with the team. Co-designed the Ablation Loss (AABL) for ground-plane heatmaps, ran the threshold ablation grid on MVDet and SHOT, and wired the loss and detector pseudo-labels into all five codebases so every architecture could be compared under the same dropped-label protocol.
  • Two simulated datasets. Built the Unity 2018 + Python pipeline that turns 1,256 PersonX characters into MultiviewX-format datasets with analytic calibration (median reprojection error under 0.001 px), then designed a social crowd model so the data has the standing, grouping and speed statistics of a real square, which MultiviewX does not.
  • Demo system. One Python server and browser UI that runs the thesis checkpoints on datasets, on a live Unity crowd streamed over TCP, or on any set of videos, with a Kalman tracker, per-frame MODA scoring, a custom CUDA deformable-attention build and a Docker image.

Problem

Multi-view pedestrian detection fuses several calibrated cameras into one bird’s-eye-view (BEV) occupancy map, so people hidden in one view are still found by another. The catch is supervision: a 7-camera, 400-frame set such as Wildtrack means 2,800 images and over 40,000 boxes, and every ground-plane position must be labelled or the model learns that a visible person is “background”.

We simulate that failure by dropping 20 %, 45 % and 60 % of the training annotations and measuring MODA (multiple-object detection accuracy) on the untouched test split. The baselines collapse, and the transformer-based MVDeTr collapses first because its default threshold never fires.

MODA, labels dropped → 0 % 20 % 45 % 60 %
MVDet, Wildtrack 86.9 81.0 59.2 22.7
SHOT, Wildtrack 92.1 84.7 26.9 4.9
MVDeTr (thr 0.6), Wildtrack 91.9 17.1 0.0 0.0
EarlyBird, Wildtrack 90.5 85.7 57.9 31.8
MVDet, MultiviewX 82.9 68.7 30.8 11.1
SHOT, MultiviewX 89.4 80.7 51.0 20.0
MVDeTr (thr 0.6), MultiviewX 93.4 60.1 2.8 0.0
EarlyBird, MultiviewX 94.9 89.9 68.5 57.6

Precision stays near 99 % in every one of these runs. The models are not hallucinating people; they are being taught to ignore the real ones that nobody labelled.

Pipeline

Pipeline — select a block to read what it does

The loss

A missing label looks exactly like background to a standard heatmap loss, so the gradient pulls a correct detection towards zero. AABL splits the heatmap into three regions after building the soft ground truth:

  1. Positive pixels, where the soft target is at least pos_thr: standard MSE to the target.
  2. Easy negative pixels, background where the prediction is low: standard MSE to the target.
  3. Confusing pixels, background where the detached prediction is already at least confuse_thr: either pull them towards 1 (mirror mode, treating them as potential positives) or keep the MSE but scale it by beta.

The three sums are divided by the total pixel count, so the loss stays on the same scale as the original and the schedules did not need retuning. The assignment uses a detached prediction, so the model cannot shrink the loss by moving pixels between regions. An earlier focal-style variant had exactly that bug: with a mirrored positive term, predicting “person everywhere” drove the loss to zero, and the optimiser found it within an epoch. The fix, which we kept, is that recalibration only ever scales a penalty down, never replaces it with a term that vanishes at p = 1.

We swept pos_thr ∈ 0.5 × confuse_thr ∈ 0.6 on MVDet and SHOT, then transferred the best settings (β = 0.1, confuse 0.3 for MSE-based models; β = 0.05, confuse 0.4 for MVDeTr) to the other architectures unchanged.

Results

MODA with the same dropped-label splits, all five architectures, before and after adding AABL. Precision stays above 93 % in every AABL row, so the gains come from recall.

MODA at 20 / 45 / 60 % dropped Wildtrack, no AABL Wildtrack, AABL MultiviewX, no AABL MultiviewX, AABL
MVDet 81.0 / 59.2 / 22.7 88.6 / 84.7 / 81.5 68.7 / 30.8 / 11.1 79.1 / 68.7 / 55.5
SHOT 84.7 / 26.9 / 4.9 89.6 / 86.6 / 66.7 80.7 / 51.0 / 20.0 88.3 / 83.5 / 76.0
MVDeTr (thr 0.6) 17.1 / 0.0 / 0.0 89.4 / 88.0 / 76.1 60.1 / 2.8 / 0.0 92.6 / 91.9 / 88.8
EarlyBird 85.7 / 57.9 / 31.8 88.6 / 86.0 / 76.3 89.9 / 68.5 / 57.6 94.4 / 91.4 / 87.2
TrackTacular (SegNet) 85.9 / 43.3 / 6.8 91.2 / 89.9 / 83.8 85.8 / 59.6 / 50.1 95.2 / 94.1 / 91.8

Adding the pseudo-label term on top of AABL helps most where AABL alone still lags, at 60 % dropped labels:

MODA at 60 % dropped Wildtrack: none → AABL → AABL + pseudo MultiviewX: none → AABL → AABL + pseudo
MVDet 22.7 → 81.5 → 83.0 11.1 → 55.5 → 56.0
SHOT 4.9 → 66.7 → 68.6 20.0 → 76.0 → 75.5
MVDeTr (thr 0.6) 0.0 → 76.1 → 88.1 0.0 → 88.8 → 91.0
EarlyBird 31.8 → 76.3 → 85.0 57.6 → 87.2 → 90.2
TrackTacular (MVDet) 8.1 → 82.1 → 84.1 23.8 → 89.1 → 90.9

The fully-unsupervised setting (no labels, pseudo-labels only) is still open: MVDeTr reaches 62.8 MODA on Wildtrack at 20 % but drops to 6.0 at 60 %, so the label-free direction is the next experiment rather than a result.

Training

MVDeTr + AABL + Detic pseudo-labels, MultiviewX, 45 % labels dropped Real run exported from training_log/log.txt (Kaggle, 10 Oct 2026, 10 epochs, lr 5e-4, batch 1). Validation loss is the test-split loss logged after each epoch; MODA is in percent.

Loss per epoch

Training and validation loss per epoch, 10 epochs 0.000.751.502.253.00 1610

MODA (%) per epoch

MODA (%) per epoch, 10 epochs 0.00255075100 1610 93
Show as table
EpochTrain lossVal lossMODA (%)
12.4970.4173
21.1670.24783.3
30.9810.23382.7
40.8810.16888.1
50.7980.14589.5
60.7150.11390.8
70.6660.09391.8
80.6090.08892.5
90.5590.08492.4
100.5310.08892.5

The run converges in eight epochs and ends at 92.5 MODA, 98.6 precision and 93.8 recall. The pseudo-label term stays small (0.05 of the total loss) by design: its job is to keep confident detector responses from being treated as background, not to replace the labels.

Two simulated datasets

MultiviewX is itself synthetic, but its crowd walks at one speed, almost never stops and almost never groups. Real squares look nothing like that. I built a pipeline that renders MultiviewX-format data from Unity so we could generate test scenes with real-like behaviour, perfect labels and zero annotation cost.

Six synchronised camera views of the simulated plaza: cobblestone square, cafe terrace, market stalls and a crowd of PersonX characters
The plaza set, frame 200: the six cameras of one frame. Same rig, FOV and 25 × 16 m area as MultiviewX, so every MultiviewX checkpoint runs on it without changes.
Plaza (main set) City square
Frames / cameras 400 at 2 fps / 6 × 1920 × 1080 400 at 2 fps / 6 × 1920 × 1080
People per frame (mean) 41.3 38.8
Annotated identities 214 192
Cameras covering a location 4.38 (MultiviewX: 4.41) 4.38
Calibration analytic from the Unity cameras, median reprojection 0.0001–0.0008 px same
Scene POLYGON city pack rebuilt as a square: café terrace, clock column, hot-dog cart, benches Japanese Otaku City asset, placed by an automatic height-map search
Behaviour walk alone or in groups, stop and chat, avoid each other, route around obstacles, queue at two stalls, sit (35 % sit 20–60 s when a seat is free); unlabelled background pedestrians walk alone or in groups, stop and chat, avoid each other; 100 unlabelled background pedestrians
Camera 1 of the plaza scene: cobblestone square with market stalls, a hot-dog cart and benches
Plaza, camera 1. The cart and benches are obstacles the annotated crowd has to route around.
Camera 4 of the plaza scene: café terrace with umbrellas and a planter partly blocking the view
Plaza, camera 4. The terrace sits right in front of the camera to create the occlusion MultiviewX’s C4 has.

The behaviour statistics come from the same script run on the annotations of all three sets (tools/behaviour/behaviour_stats.py, re-run on 26 Oct 2026). The plaza lands on Wildtrack’s numbers; MultiviewX does not.

Crowd statistic (from labels, 2 fps) Wildtrack (real) MultiviewX Plaza (mine)
Stationary (< 0.3 m/s) 37 % 0 % 38 %
Walking speed, p10–p90 0.58–1.64 m/s 1.15–1.21 m/s 0.57–1.67 m/s
In a group 46 % 8 % 54 %
Standing in a group 21 % 0 % 26 %
Walking alone 38 % 92 % 35 %
Nearest neighbour, median 1.01 m 1.44 m 0.96 m
Body overlaps per 100 person-frames 0.46 2.33 0.28
Frame 200 of the MultiviewX rig in the original dataset and in the simulated plaza, Camera 1 (MultiviewX) Frame 200 of the MultiviewX rig in the original dataset and in the simulated plaza, Camera 1 (Plaza (mine))
MultiviewX / Plaza (mine), same frame and camera. Drag the handle or use the arrow keys; pick a camera above.

How the pipeline works, in order:

  1. Characters. 1,266 PersonX prefabs are converted to Humanoid rigs by script (1,256 valid), given one shared walk/idle blend tree, and speed-matched so feet do not slide.
  2. Cameras. The six MultiviewX cameras are rebuilt from the original calibration (2.2 m high, 15° down, 94° horizontal FOV) and verified against Unity’s own projection to under 0.2 px. The rig never moves; the environment is moved and rotated around it, so calibration and the BEV grid are identical across every dataset.
  3. Scene placement. A height map of the city asset is scanned over 72 rotations to find flat, unobstructed spots where no camera ends up indoors; candidates are rendered and picked by eye.
  4. Crowd. A seeded steering model with personal speeds, group formation, chatting stops, avoidance within 1 m and NavMesh routing around obstacles. The same code runs in Play mode, in the Editor preview and in batch generation, so what you see is what gets rendered.
  5. Render and convert. Unity renders 6 × 1920 × 1080 per frame (about 13 minutes and 5 GB per set). A Python script writes the calibration XML, rectangles.pom (3.84 million rectangles), annotations_positions, and a BEV preview for checking.
Automatic check image: numbered people on the bird's-eye-view grid above the six camera views with projected boxes
Automatic label check: each annotated person on the BEV grid and its projected box in all six cameras.
Four frames 0.1 seconds apart from camera 5 showing a walking character mid-stride
Walk cycle at 0.1 s steps: animation speed follows the simulated velocity, so poses vary across frames.
Top row: cameras 1, 4 and 5 of the simulated plaza. Bottom row: the same three cameras in the original MultiviewX dataset
Same frame, same camera layout: the plaza (top) and the original MultiviewX (bottom; hou-yz/MultiviewX, CC BY-SA 4.0). The terrace in front of C4 and the clock column are placed where MultiviewX has its umbrellas and column.

What the sets revealed. Running the MultiviewX-trained MVDeTr + AABL checkpoints on the 40 test frames of the city-square set without any retraining costs about 17 MODA, almost all of it recall:

MVDeTr + AABL trained on MultiviewX with MultiviewX test MODA City-square test MODA Precision Recall
20 % dropped 93.0 75.9 96.6 78.7
45 % dropped 92.5 73.6 94.8 77.8
60 % dropped 91.2 74.1 96.6 76.8

Precision holds despite 100 unlabelled background pedestrians, and all three checkpoints degrade by the same amount, so the partial-label training is not what breaks. The gap is scene and crowd behaviour (standing groups, denser crowds, a new city), which is exactly the gap the sets were built to expose, and the number our next experiments have to beat.

Demo system

The demo exists so that a reviewer can verify any claim on one screen: which checkpoint, its logged test MODA, this frame’s TP / FP / FN, the latency and which calibration was used. Detections come from the checkpoints only; 2-D boxes are projections of a 1.8 m person through the calibration, never model outputs.

Architecture. A Python server (PyTorch, fp16) with a checkpoint registry that scans the repo for MultiviewDetector.pth files, including inside zips, and reads dataset, drop ratio, thresholds and the logged test MODA from each run’s log.txt. Backends for MVDet and MVDeTr, with a custom-built CUDA deformable-attention op that runs 2.2–2.4× faster than the PyTorch fallback at identical output (MODA 90.23 on 40 Wildtrack test frames, heatmap difference 4e-5). A constant-velocity Kalman tracker with Hungarian matching (1 m gate) gives IDs and velocities. The browser UI is plain HTML, CSS and JavaScript.

Demo UI, Dataset tab: six camera tiles, a ground-plane map with detections and GT rings, and tiles reading GT 40, TP/FP/FN 34/0/6, frame MODA 85.0 percent
Dataset tab on the city-square set, frame 360, MVDeTr trained on MultiviewX with 20 % dropped labels (logged test MODA 93.0). This frame: 40 people, 34 found, 0 false positives, 6 missed, frame MODA 85.0 %. The first run of a dataset includes loading its calibration, hence 1.9 s; the same model runs at about 270 ms per frame in fp16 once warm.

Three input modes.

  • Dataset. Browse Wildtrack, MultiviewX or the simulated sets, run the real checkpoint on the GPU and score the frame against ground truth: GT rings, matched, false-positive and missed colours, per-frame MODA.
  • Live · Unity. The Unity project renders six cameras into one 3 × 2 atlas with AsyncGPUReadback (no JPEG encoding, no GPU stall) and streams it over raw TCP with the GT feet and the rig poses. The server builds the MultiviewX calibration from the rig (1.12 px error against Unity’s own projection) and runs the checkpoint; the tracker’s velocities let both the browser and Unity draw every person at position + velocity × elapsed, so dots move with people between detections. Pause freezes Unity (timeScale = 0) and the browser on the same frame, from either window.
  • Scene · Video. Any N videos (per-video sync offsets, sample rate, time span) or an image folder, with a calibration folder, calibration-free estimation (48 frames → YOLO + MapAnything → ground-plane fit → foot refinement, about 5 minutes), or no calibration at all. A BEV editor drags, rotates and mirrors the checkpoint’s fixed grid over the camera footprints.
Demo UI, Live tab paused at frame 367: tracker nodes on the map, tiles reading detector rate, model time 268 ms, tracks 35, MODA over 10 frames 81.3 percent, precision 99.4, recall 81.8
Live tab fed by the replay client (city-square set from frame 360, paused at 367). Model time 268 ms in fp16; running MODA over 10 frames 81.3 %, precision 99.4 %, recall 81.8 %. The 0.57 Hz rate here is the Python replay client’s pacing, not the model; the Unity client reaches 2.3 Hz.

Interaction. Click a person in any camera, on a node or on the map to highlight them in every camera; Ctrl-click to select up to six. Click an empty map point to project a 1.8 m probe box into every camera and see whether someone really stands there. Follow one track with a trail. Two layouts: a grid, or “Surround”, which draws each camera at its calibrated position around the BEV with its field-of-view edges.

One selected pedestrian highlighted with a yellow box in all six cameras and an orange node on the map; the detail panel lists the box coordinates per camera
Select a person: one click in camera 0 highlights the same ground point in all six views, with the box coordinates per camera listed below.
A probe point on the map projected as yellow boxes into four cameras, with 'Not in this view' on the other two
Probe a point: a 1.8 m box at (3.75 m, 3.18 m) projected into every camera that sees it, with the distance to the nearest detection and GT person.
Surround layout: the ground-plane map in the centre with camera icons C1 to C6 at their calibrated positions and the six views arranged around it
Surround layout: cameras drawn at their calibrated positions with field-of-view edges; each view is placed in the slot nearest its icon.
Dataset tab on the plaza set: six cobblestone-square views, map with detections, tiles reading GT 44, TP/FP/FN 35/0/9, frame MODA 79.5 percent
Plaza, frame 360, same checkpoint: 44 people, 35 found, 0 false positives, 9 missed (frame MODA 79.5 %). Obstacles and seated people cost recall, which is what the set is for.
Scene setup dialog with four steps: Frames, Cameras (calibration folder, calibration-free, or none), Checkpoint, and BEV area placement
Scene · Video setup: frames, cameras (folder, calibration-free estimation, or none), a checkpoint with a matching camera count, and the BEV area editor.

Run it.

# clone, then install torch (CUDA), torchvision, kornia, numpy, Pillow, scipy, opencv-python
.\demo_multiview\build_deform_attn.ps1     # optional: CUDA 12.8 + VS 2022 Build Tools
.\demo_multiview\run_demo.ps1              # http://127.0.0.1:8765, Live tab at /#live
# or without Unity: replay a recorded sequence into the Live tab
python demo_multiview\live_replay.py F:\PersonX\MultiviewX_Otaku_opt1_social_v6 --start 360 --loop
# or: docker compose up -d    (image with code + 20 checkpoints, datasets mounted, runs on a 6 GB GPU)

Measured on an RTX 4050 Laptop (6 GB), GPU shared with Unity, 60 s runs:

Configuration Detection rate Unity fps Track error p90 MODA
default (2.5 captures/s) 2.3 Hz 12–14 16–17 cm 82–83 %
smoother Unity (3 ms GPU gap, 10 captures/s) 1.5 Hz 18 16 cm 82.5 %
fastest detection (10 captures/s) 3.5 Hz 7 21 cm 79.5 %

Median track error is 6–7 cm in all three because positions are extrapolated with velocity. Video playback decodes seven Wildtrack streams at exactly 10 samples/s while the detector runs at 3.25 Hz on the newest sample; the first version managed 0.3 samples/s because eight requests were seeking the same H.264 file.

Challenges and lessons

  • A loss that could be gamed. The first AABL variant let the optimiser zero the loss by predicting foreground everywhere. We now only ever scale penalties down and always threshold on a detached prediction. Lesson: check what the global minimum of a new loss is before training on it.
  • A silent no-op. SHOT’s model called plain MSE inside forward, so --loss brl changed nothing in its early runs. Found by comparing logs that should have differed; those runs were discarded and the README now states it.
  • Coordinate conventions. MultiviewX’s calibration is mirrored (all projected points have negative depth). Rebuilding the rig in Unity meant flipping one axis and proving it with a depth check and a 0.2 px reprojection test, then keeping that test in the pipeline.
  • Numbers that look fine while images are wrong. Drawing foot dots and 3-D boxes on the rendered frames early caught every large bug (wrong screen-point API in batch mode, foot sliding, a 1:10 scale city) that the summary statistics missed.
  • GPU budget. Unity and PyTorch share 6 GB. Reading cameras back asynchronously, sending one atlas instead of six JPEGs, and copying with memcpy instead of Unity 2018’s per-element NativeArray.CopyTo (350 ms for 22 MB) is what made 2.3 Hz possible.

Credits

Capstone group GFA26AI02 at FPT University: Lai Le Dinh Duc, Ho Dang Manh, Nguyen Nhut An and me, supervised by Dr. Dao Duy Son. Built on the public MVDet, SHOT, MVDeTr, EarlyBird and TrackTacular codebases, the Wildtrack and MultiviewX datasets (MultiviewX images above: hou-yz/MultiviewX, CC BY-SA 4.0), PersonX characters, and the Japanese Otaku City and POLYGON city assets.