Foundational research and dataset lineage

Ethan AI is built on E-BED, the Ethan AI Behavioural Events Dataset, a consented, multimodal (video + audio) record of the safety behaviours that matter in daily autism care. E-BED builds on more than a decade of public research. This page traces that lineage: what came before, what we added, how we measure, and where the evidence stops.

The lineage: from clips to events

Computer vision research in autism has largely treated behaviour as a classification task, given a short, curated clip, decide whether a stereotypical behaviour is present. That framing produced the field’s foundational public datasets. The problem families and therapy centres actually face after diagnosis is different: continuous detection of safety behaviours in ordinary rooms, delivered as an alert within seconds. E-BED was built to cross that gap.

SSBD (Self Stimulatory Behaviours in the Wild, 2013) is the field’s founding public dataset, roughly 75 uncontrolled “in the wild” videos gathered from public platforms, covering arm flapping, head banging, and spinning, with a bag of visual words baseline near 47%. ESBD (2021) extended it with a hand action category. ARRBD (2025) expanded coverage to 780 curated clips across 10 behavioural categories, reaching very high accuracy on that curated set. Each of these is a real contribution, and each shares the same shape: short, pre segmented clips, classified after the fact.

We derived task specific training material from the public datasets and folded it into E-BED. Doing so is what let us watch recall climb as the data matured: on the two most safety critical behaviours, pooled recall rose from 0.40 (public clips alone) to 0.60 (adding our consented pilot data) to 0.90 (adding augmentation and temporal composition).

E-BED: a dataset of events, not clips

E-BED is a consented behavioural events dataset of roughly 14,000 annotated events across 61 subjects and about 640 hours of co recorded video and audio, with 6,240 neutral segments held separately. It is assembled from four sources:

Augmented material is used for training only, evaluation folds contain no augmented data, and every augmentation is grouped with its source subject so it can never leak across a train/test split. E-BED is an actively growing dataset; the exact event tally is approximate.

The annotation layer

Each event carries more than a category label. Every event is labelled with its onset and offset boundaries, its category, its modality (video, audio, or both), an intensity (1 to 3), and an annotator confidence (1 to 3). Every event is also preceded by a labelled 20 second pre event window, with neutral segments held separately from the event count.

This pre event layer is not present in other autism behaviour datasets we are aware of.

Behavioural intervention rests on antecedent behaviour consequence analysis. A human log begins the moment a behaviour is noticed, so the antecedent is reconstructed from memory after the fact. A system that is already watching records the antecedent instead. E-BED’s pre event annotation supports the study of early escalation, not only the classification of extreme events.

Annotation used CVAT for temporal boundaries plus an internal review tool; four certified special educators and one consulting clinical reviewer annotated against a shared codebook after a six hour calibration, with quarterly re calibration. A 12% subset was double coded, yielding Cohen’s κ = 0.79; full cohort inter observer agreement is still being computed.

What the pre event layer gives ABC and ABA practice

In applied behaviour analysis, the evidentiary base of a functional behaviour assessment is ABC recording: what happened immediately before a behaviour (the antecedent), the behaviour itself, and what followed (the consequence). The log is the clinical work in its raw form. The antecedent is the part that changes a behaviour plan.

Manual ABC recording is weakest at the antecedent. The log begins the moment a behaviour is noticed, so the trigger is reconstructed from memory after the fact. Recall misses more than half of what happened, observers disagree, and the analysis arrives too late to act on. E-BED’s 20 second pre event window records the antecedent instead of reconstructing it.

The pre event window supports what manual ABC cannot reliably deliver:

• The antecedent is recorded, not recalled. Escalation can be studied on real pre event data rather than after the fact narration.
• Patterns become visible across settings. A behaviour that clusters after a particular transition, at a particular hour, or only in one room can be seen rather than suspected.
• A functional behaviour assessment gains a continuous substrate. An FBA can draw on weeks of consistent, time stamped antecedent data instead of a handful of sampled observations.
• The clinician’s scarce time shifts from capturing data to interpreting it.

What it does not do: an automated log records observable events, not internal states, and it does not perform the analysis. The hypothesis about function, the intervention design, and the clinical judgement stay with trained humans.

Related reading

The antecedent problem: why ABC data collection fails at the moment it matters most

→

How we measure, two separate evaluations

We report two evaluations separately:

• Offline recognition (E1). Subject independent leave one group out 5 fold cross validation across the 61 E-BED subjects, individuals in training and testing never overlap. Test folds contain only original, unaugmented material from held out subjects.
• In the wild deployment (E2). The deployed system evaluated across an 8 week study of 34 children, none of whom appear anywhere in the training data. Field performance therefore measures generalisation to unseen children, not personalised performance.

Curated clip benchmarks report higher headline accuracy on an easier task. A continuous stream in a real room is harder than a pre segmented clip, and the two numbers are not comparable. We report both, on unseen children in each case.

Per behaviour results (offline, subject independent)

Two ablations, run under the same protocol and test sets, isolate the effect of each design choice. Removing generative augmentation drops weighted F1 from 0.86 to 0.74. Removing the temporal composition stage (leaving atomic per frame heads) drops it from 0.86 to 0.79, a significant difference (paired t across folds, p = 0.004).

What it looks like in the field

The system was deployed for eight weeks across 14 sites, therapy centres and homes in Visakhapatnam and Hyderabad, on cameras the sites already owned. Measured against ground truth logged in real time by certified special educators:

Full study write up: clinical validation results presented at NIMHANS Autism Synergies 2026.

From bag of visual words to trajectory modelling

The founding SSBD work used a bag of visual words framework: break a video into frames, extract visual features, cluster them into a “visual vocabulary,” and represent each video as a histogram of how often each pattern appears. It is effective at detecting that a repetitive pattern occurred, but it discards temporal order. It captures how often something happens, not how it evolves.

Ethan AI models both spatial structure (what the body is doing in each frame) and temporal dynamics (how the behaviour changes across seconds), and we look for signals like a behaviour intensifying over 20 seconds, sustained upward trends in motion energy, joint velocity, or repetition rate, relative to an individual’s own baseline. Treating behaviour as a trajectory rather than an unordered count turns action recognition into behavioural progression analysis, and supports early detection of escalation rather than post hoc classification of isolated events.

Limits

Where the system is weak or unproven:

• Drooling and safe zone exit are our weakest behaviours, and fine, brief, or occluded behaviours, hand flapping, hair pulling, harmful ingestion, are not yet reliable.
• Infrared night mode degrades performance materially (weighted F1 falls from 0.86 to 0.68); below very low light, audio carries detection alone.
• Field sensitivity is characterised in observed therapy sessions; home and low activity periods contribute precision and alert data, but their sensitivity is not yet measured.
• The study had centre level clinical review, not a constituted Institutional Ethics Committee, a real constraint on its standing.
• One region, single arm, eight weeks, 34 children. The caregiver burden result is a feasibility signal, not evidence of efficacy, and generalisation beyond two Indian cities is untested.

The privacy floor: no facial recognition, no biometric templates, no identity matching; children are tracked only by a site local profile ID. Cameras exclude bathrooms and changing areas. Data are held in India region storage under role based access, and consent covers de identified derived data only, with deletion honoured on request.

What we plan to share

E-BED itself cannot be released, consent terms do not allow child footage to leave our governance. But two derived resources can be shared with research collaborators without releasing any footage:

• Our temporal annotation layer, onset/offset, neutral, and pre event labels, for the public SSBD, ESBD, and ARRBD videos.
• Non reversible derived signals from consented data: pose sequences, motion energy and repetition rate series, and audio event embeddings.

These let others study behavioural escalation without any child being identifiable. A full methods paper describing E-BED and the field study is in preparation.

References

Ethan AI is a caregiver-support aid, not a diagnostic instrument. The figures on this page are drawn from an internal study; a peer-reviewed methods paper is in preparation. E-BED is not publicly released; derived, non-reversible resources are available to research collaborators under agreement.