Anonymized engagement
An executive team asked what it would cost to build a video dataset in house rather than license one. We surveyed the sources, designed the label taxonomy, and specified the annotation and ingestion pipeline that turns raw footage into data a model can learn from.
The brief
Commercial licensing quotes for a single source ran between $150,000 and $225,000. The question was whether an equivalent corpus could be assembled from open-access repositories and permissively licensed material, annotated to a professional standard, for materially less — and what the architecture would have to look like to keep that credible at scale.
The answer was yes, with conditions. What follows is the architecture, with the client and the subject matter removed. The shape is domain-independent: it is the same whether the footage is road scenes, industrial inspection or anything else where video becomes model data.
01 · Source survey
Open-access repositories first, permissively licensed material second, custom capture only where coverage was genuinely missing. Each candidate source was assessed against the same six questions, so sources could be compared rather than argued about.
WHAT IS IN IT
Clip counts per category, not just totals. A corpus with a long tail of thin categories is a different proposition from an evenly covered one.
IS IT USABLE
Resolution and duration decide whether a clip survives preprocessing. Structure decides whether it can be addressed at all.
CAN WE USE IT
The cheapest corpus is worthless if its licence does not cover the intended use. Licence terms were recorded per source alongside the access method.
02 · Taxonomy
A label set that looks obvious at four clips falls apart at five hundred. We specified the taxonomy as a deliverable in its own right: a flat, explicit class list that grows on a schedule rather than by accretion, expanding to thirty to forty classes by the pilot tier.
Every label carries a start and an end. That single decision is what makes a corpus addressable at the segment level instead of the file level — you can ask for the four seconds that matter rather than the two-minute clip that contains them, which changes both storage economics and what a sampling strategy can do.
03 · Ingestion
Open-source loader libraries made the difference between a workable pipeline and a storage problem. A filtered subset is requested declaratively — split, class, maximum duration, maximum sample count — and only the matching material is fetched.
FETCH
A handful of lines pulls a filtered slice of a public corpus. Nothing is downloaded in bulk, and the same expression reproduces the slice later.
NORMALISE
Everything lands in the same shape regardless of source: 1080p MP4 alongside labels as JSON or CSV, so downstream code never branches on provenance.
COST
The ingestion layer is free and open source end to end. The spend sits in annotation and review, which is where it buys something.
04 · Annotation and review
Annotation is the expensive part, so it is staged. Each tier is sized to answer one question before any larger commitment is made, with hours estimated up front and quality controls added as volume grows.
TIER 0 · 4 CLIPS
Proves the annotation workflow end to end and produces the first labelled output in the agreed format.
TIER 1 · 20 CLIPS
Scales the workflow and introduces the first real quality controls, including sampling and format validation.
TIER 2 · 50 CLIPS
Enough material to exercise the expanded taxonomy and find the classes that are ambiguous in practice rather than on paper.
TIER 3 · 500 CLIPS
Production-scale pilot on a quality-control workflow that holds at volume, with accuracy measured rather than asserted.
05 · When the footage does not exist
Some of the cases that matter most are the ones a camera rarely catches. On separate work we build the environment in software and expose it through the standard reinforcement-learning interface — a discrete action space, dictionary observations of typed numeric arrays, a step and reset contract — so an agent developed against the simulator talks to it exactly as it would to any other environment, and the simulator can be swapped for recorded data without touching the agent.
That interface is the join between the two halves of this page. Recorded footage gives you the distribution the world actually produces; a simulated environment gives you the rare case on demand. Most serious programmes need both.