Disclosure: I’m Philip, with SingNova. These measurements and implementation details come from a colleague on our engineering team. This text was drafted with AI assistance. I’m sharing their profiling notes and will relay technical questions to them, then share their confirmed answers here.
In a browser-to-device object-detection demo, preprocessing takes 51.0–59.5 ms, while the worker’s inference stage takes 59.5–62.7 ms. Preprocessing, inference and postprocessing currently run serially. The useful question for us is how to break down the preprocessing measurement before deciding whether to reduce that work or overlap it with inference.
The demo also illustrates a measurement distinction: the output stream is configured at 25 FPS, but one run completed only about 7.5 new inferences per second. Those are different quantities, and neither establishes capture-to-display latency.
Setup and video path
- Host CPU architecture: riscv64; model execution: a dedicated DFU accelerator, not the host CPU.
- OS: openEuler 24.03 LTS-SP2, Linux 6.6.118+.
- Runtime: 1.0.3, with a model-specific plugin and explicit cache synchronization.
- Source video: 640×360 at 25 FPS.
- Decode: FFmpeg software decoding. Hardware video decode is being developed separately and is not part of these measurements.
- Model: YOLOv12s; input: 1×3×640×640, with BGR input for this setup.
- Preprocessing: colorspace conversion, scaling and black letterbox padding.
- Postprocessing: tensor decoding, confidence filtering, NMS and box rendering.
Browser → WebRTC/WHIP → ZLMediaKit → FFmpeg software decode
→ preprocessing → DFU inference → postprocessing/overlay
→ FFmpeg H.264 encode → RTSP → WebRTC/WHEP → browser
What the timers measure
We have two levels of instrumentation.
Inside the runtime, kernel_ms measures device execution. total_ms covers the inference call, including input/output copies and cache synchronization. The reported YOLOv12s averages are:
| Runtime measurement |
Reported mean |
Device execution: kernel_ms |
46.45 ms |
Complete runtime call: total_ms |
~53 ms |
One individual invocation is shown below. It is a sample, not the averages above:
[DFU_PROFILE] infer_id=977 status=ok
input_copy_ms=1.677 input_cache_clean_ms=0.263
kernel_ms=50.570 result_pre_read_ci_ms=0.095
output_copy_ms=1.043 result_post_read_ci_ms=0.114
total_ms=53.834
input_bytes=2457600 result_bytes=1411200
Separately, the worker reports pipeline-stage timings:
| Pipeline measurement |
Reported range |
preprocess_ms |
51.0–59.5 ms |
infer_ms |
59.5–62.7 ms |
post_ms |
11.3–12.9 ms |
infer_ms is the worker-level measurement, not an alias for kernel_ms. The runtime averages and pipeline ranges should be read separately. The worker-level inference timings are higher than the reported runtime-call mean, but these are different measurement scopes and summaries. We have not yet quantified or attributed the difference using paired per-invocation traces.
The preprocessing timer is aggregated. It does not yet separate colorspace conversion, scaling/padding, tensor preparation, copies or any waiting included in that scope. We cannot yet identify which operation dominates it.
Throughput, dropped frames and unmeasured latency
One run completed 977 inferences in 130 seconds, approximately 7.5 new inferences per second.
The source is capped at 25 FPS. When processing cannot keep up, the current implementation drops older input frames; the engineering notes report approximately 69% input frames dropped in the YOLOv12s test.
Output video remains configured at 25 FPS and can repeat the last detection result between inferences. A smooth output stream therefore does not imply a new detection result for every frame.
Model-switch initialization is recorded separately: the first YOLOv12s pipeline invocation was 1459.9 ms, versus approximately 60–61 ms in steady state. Those invocation figures are not whole-pipeline or capture-to-display latency.
Capture-to-display latency has not been measured. We do not currently have frame IDs connecting browser capture with the returned result. The figures above describe processing time and throughput, not end-to-end latency or frame age at display.
Questions for people profiling similar vision pipelines
- Where would you place additional trace points? How would you separate FFmpeg/filter work, scaling/padding, tensor preparation and copies, especially when a stage timer might include waiting for a frame or buffer?
- What evidence would make you prioritize reducing preprocessing work versus overlapping preprocessing and inference? The current three-stage worker is serial, and input buffers are explicitly synchronized before device execution. How would you measure the opportunity while checking buffer ownership and safe reuse?
- How do you preserve frame identity through a latest-frame/drop-old pipeline? I’m interested in approaches that let you measure frame age at display without assuming synchronized browser and device clocks.
Specific profiling methods or examples from comparable pipelines would be useful. These are measurements from one implementation on one engineering unit, not a comparison across devices or a general claim about RISC-V performance.
