Detecting and tracking resident space objects directly in a neuromorphic camera’s event stream — never reconstructing a frame, and therefore never paying a frame’s latency. The whole pipeline is NumPy and SciPy on two CPU cores. There is no GPU in it, and nothing in it was trained.
An event camera on a sidereally-tracked telescope sees a nearly-static star field. The sensor is differential — a pixel only speaks when its log-intensity changes — so after noise rejection, almost every surviving event came from something that moved. The motion segmentation is free, in hardware, before any software runs.
The conventional move is to render those events back into frames and run a convolutional detector over them. That throws the free segmentation away and buys a per-frame compute bill that commodity CPU hardware cannot absorb inside a 40 ms window. It also applies a general object detector to a target that is two to five pixels across — an unresolved point source with no shape and no texture to recognise.
SPARSE stays in the event domain from decode to track. Doing so turns the latency budget from a constraint into a non-issue: the measured end-to-end cost is about one millisecond, which leaves the remaining thirty-nine for whatever an operator wants to do next.
(t, x, y, p) array.
once, at loadk, no neighbour graph.
0.22 msEvery stage is linear in the number of events. Nothing is quadratic in anything, which is why throughput degrades gracefully as the noise rate climbs rather than falling off a cliff.
Inside the association gate, PDA blends candidate measurements by likelihood instead of committing to the closest one. That is what keeps a faint target alive when a star scintillates beside it — nearest-neighbour association hands the track to the star. The filter returns sub-pixel position and angular rate, which is what a space-situational-awareness operator actually wants out of a tracker.
“Real-time capable” is not a claim. These are wall-clock measurements per processed 10 ms window on an AMD EPYC with cores and threads pinned, at 640 × 480 over 245,161 events.
| Metric | 1 core | 2 cores |
|---|---|---|
| Mean end-to-end latency | 1.32 ms | 1.01 ms |
| p95 | 2.11 ms | 1.33 ms |
| p99 | 3.91 ms | 1.46 ms |
| Throughput | 0.93 M ev/s | 1.21 M ev/s |
| Windows over a 40 ms budget | 0 % | 0 % |
| Precision / recall / F1 | 1.000 / 0.638 / 0.779 | 1.000 / 0.638 / 0.779 |
| Localisation error | 0.96 px | 0.96 px |
Accuracy is identical on one core and two, because every random number generator is seeded. A single core still meets a 40 ms budget ten times over.
Every number in the table above is from a synthetic scene, not from a sensor. They are produced by the DVS contrast model in our own scene generator, which gives exact sub-pixel ground truth and a clean magnitude ladder — the two things a fixed recording will not give you. They are honest measurements of real code doing real work, and they are not evidence of performance on a real telescope.
The section immediately below is what happened when we ran the same code on a real one.
We ran the pipeline against the Western Sydney EBSSA set — real neuromorphic recordings from a real telescope, openly published. The recording is an ATIS sensor at 304 × 240, 5.8 million events over 198 seconds, twelve human-labelled objects.
Tuned as it was for synthetic scenes, the pipeline scored F1 0.000 and found 0 of 12 objects. Latency looked superb the whole time. The failure was completely silent — which is exactly the failure a synthetic-only benchmark is built to hide.
The diagnosis is the useful part. On the synthetic scene roughly 5% of events belong to a target. On the real recording it is 0.081% — prior odds of one to twelve hundred. And the noise gate was asking the wrong question. It keeps an event only if one of its eight neighbours fired within 5 ms, but a tracked space object is an unresolved point source emitting about fifteen events a second onto one or two pixels, so its own events arrive 65 ms apart. Inside windows that contained a labelled object, one event in 273 survived the filter. The pipeline was working perfectly on an empty stream.
The second cause was the 10 ms window. For a counting detector against a Poisson background the figure of merit is signal over the square root of background, not signal over background. We read the wrong column first, and the correction moved the optimum from 100 ms to about a second.
| Integration | Signal events | Background | S/√B | Object smear |
|---|---|---|---|---|
| 10 ms — the old default | 0.23 | 0.09 | 0.8 | 0.06 px |
| 100 ms | 3.91 | 0.86 | 4.2 | 0.63 px |
| 250 ms | 9.04 | 2.12 | 6.2 | 1.57 px |
| 1000 ms | 28.14 | 8.18 | 9.8 | 6.28 px |
| 2000 ms | 41.89 | 16.19 | 10.4 | 12.57 px |
That is why detection now runs at more than one scale at once. A scale is an integration depth with its own detector and its own tracker, because an object drifting at 6 px/s and one transiting at 640 px/s do not share an answer. A deep scale integrates without overlap: re-clustering an almost identical buffer every window makes a chance cluster reappear a hundred times, which quietly destroys the independence that m-of-n confirmation depends on.
| On the EBSSA recording | Result |
|---|---|
| Mean end-to-end latency | 0.17 ms |
| p99 latency | 1.23 ms |
| Speed against real time, 2 cores | 57× |
| Windows over a 40 ms budget | 0 % |
| Precision / recall / F1 on tracks | 0.052 / 0.079 / 0.063 |
| Localisation error | 3.04 px |
| Objects found | 1 of 12 |
| What an operator sees in 198 s | 4 tracks: 1 real, 3 false |
That is a weak accuracy result and we are not going to dress it up. The pipeline finds and holds the primary tracked object on real data, to three pixels, while running fifty-seven times faster than real time on two cores — and it raises three false tracks in three and a half minutes. Ten of the twelve labelled objects are transits crossing at 244 to 641 px/s; they deposit fewer than two events in a 10 ms window, so there is nothing for a point-source clusterer to cluster. That is arithmetic rather than tuning, and reaching those objects needs a streak-aware detector we have not built.
Two smaller things the run settled. The same code handled a second sensor at a second resolution — a DAVIS at 180 × 240 — with no change at all, which until then had only been shown on synthetic scenes. And the synthetic profile is bit-for-bit unchanged: calibrating for real data adds a named parameter set, it never quietly retunes numbers already published.
F1 on confirmed tracks across a magnitude ladder, at three sensor resolutions. Higher magnitude means a fainter object.
Read plainly, this is a cliff, not a slope. Between magnitude 8 and magnitude 10 at 640 × 480, F1 goes from 0.99 to 0.11. And the cliff moves brighter as the sensor shrinks: more pixels across the point-spread function means more contrast crossings per unit of flux, so a larger sensor reaches fainter.
The cause is not detector sensitivity. Recall is measured per window, and a faint target emits events intermittently — so in some windows it simply does not accumulate enough events to clear the cluster floor. The remedy is track continuity: predicting through the gaps rather than building a larger model. We publish the cliff because a stated limitation is worth more than a suspiciously clean result.
FEAST — an unsupervised spiking feature layer with adaptive selection thresholds — is implemented and works. On this noise process it costs 2.4× the end-to-end latency and moves F1 from 0.779 to 0.776: inside noise, and in the wrong direction. So it ships disabled and is reported as a measured ablation rather than quietly dropped. It may still earn its place on real sensor noise, which is a different process from the synthetic model.
Every latency figure above comes from one scene at one input rate. That says the pipeline is fast on a small sensor; it says nothing about the ceiling. So we raised the input rate on a 1920 × 1080 sensor — larger than anything in the EBSSA set — until the pipeline broke, and recorded where.
The largest single run: 83,189,520 events, 1.08 GB of raw event data, a thousand consecutive 10 ms windows. It held all three objects at F1 0.978 with a mean of 28.25 ms and a p99 of 36.70 ms per window, sustaining 2.81 million events per second. Seven windows out of a thousand crossed the 40 ms budget.
| Events / window | Filter | Detect | Track | Total | Over budget | F1 |
|---|---|---|---|---|---|---|
| 3,245 | 2.39 ms | 0.58 | 0.61 | 3.58 ms | 0 % | 0.995 |
| 11,126 | 4.80 ms | 0.65 | 0.62 | 6.07 ms | 0 % | 0.995 |
| 21,497 | 8.23 ms | 0.69 | 0.68 | 9.60 ms | 0 % | 0.995 |
| 42,236 | 14.68 ms | 0.74 | 0.71 | 16.13 ms | 0 % | 0.995 |
| 83,712 | 23.08 ms | 0.93 | 0.62 | 24.63 ms | 0 % | 0.995 |
| 166,660 | 47.52 ms | 28.74 | 35.83 | 112.09 ms | 100 % | 0.085 |
| 332,557 | 90.18 ms | 21.90 | 22.91 | 135.00 ms | 100 % | 0.000 |
Per-window latency stays inside 40 ms up to 8.4 million events per second. But sustained real time is a stricter test that the budget never states: a 10 ms window has to be processed in under 10 ms, or the queue grows. That line is crossed at ~2.2 million events per second. Between the two, the pipeline is simultaneously inside its latency budget and falling behind its sensor. Both numbers are true; they answer different questions, and a system quoting only the flattering one is not quoting a capability.
The noise gate costs 2.39 ms at three thousand events per window and 23.08 ms at eighty-four thousand — about 0.28 microseconds per event, linear, exactly what the complexity says. Meanwhile detection and tracking stay flat between 0.6 and 0.9 ms across a 26× change in input rate, because the gate hands them a near-constant workload however loud the input gets. That is the whole architecture in one row of numbers, and it is measured rather than argued.
At 166,000 events per window the gate stops holding. Filtering goes from 23 to 48 ms, detection from 0.93 to 28.74 ms — a thirty-onefold jump — and F1 collapses from 0.995 to 0.085 in the same step. Compute and accuracy break together and for a single reason: enough noise survives to cluster like a target. It is a cliff rather than a slope, which is the same shape as the brightness limit above.
Up to that point accuracy is flat at F1 0.995 across a 26× input-rate change. That is the claim worth making from this: throughput costs latency, not correctness — until the gate gives way, when it costs both.
This is a synthetic stress test, and the honest detail is the tail. At the headline rate a two-second run reports zero windows over budget. The same rate over ten seconds reports seven in a thousand. The overruns only exist in the tail of a longer run, so a short benchmark would have shown a clean sweep and been wrong about the system. We report the ten-second number.
A list of what a system uses shows assembly. A list of what it rejected shows judgement. Every component below was assessed against three filters: does it fit the latency budget on commodity CPU, is its licence permissive, and does it actually improve a result.
| Component | Verdict | Reasoning |
|---|---|---|
| Grid clustering | KEEP | Single O(N) pass, no k, no neighbour graph. DBSCAN’s memory profile is the wrong shape for a streaming window. |
| Kalman + PDA | KEEP | Gives sub-pixel position and angular rate at 0.34 ms per window. |
| YOLOv8 / ultralytics | CUT | Two independent reasons. Its AGPL-3.0 licence is copyleft with a network clause. And a general object detector is the wrong instrument for a two-to-five-pixel point source. |
| Pretrained COCO weights | CUT | Weights carry their own terms, independent of the code that loads them. Nothing here is trained, so there is no checkpoint to encumber. |
| Sliced inference (SAHI) | CUT | Exists to find small objects inside large frames. We never build a frame. |
| Appearance re-ID (DeepSORT) | SWAP | An unresolved point source has no appearance to re-identify. Replaced with probabilistic data association. |
| PostGIS | CUT | Wrong coordinate system. Detections live in pixel coordinates mapping to right ascension and declination on the celestial sphere — not latitude and longitude on the Earth’s surface. |
| Redis / message broker | CUT | A broker between stages that each complete in under a millisecond adds latency, a process boundary and a failure mode, in exchange for nothing. |
| Trained SNN (snnTorch, Norse) | CUT | Backpropagation-through-time on two CPU cores. FEAST gives a genuine spiking mechanism with no training run at all. |
| Time-series store, 3D globe | DEFER | Defensible in production; worth nothing in a prototype. A celestial globe is actively misleading until astrometric plate-solving and TLE correlation exist to feed it. |
SPARSE ships as a multi-stage linux/amd64 image that runs as an
unprivileged user, so nothing it writes into a mounted volume comes back
root-owned. There is no CUDA base, no GPU runtime, no model weights and no
dataset inside it.
It is published and pullable right now, from our own read-only registry. No account, no login, nothing to configure:
docker pull vixitai.com/sparse:1.0
docker run --rm -v $PWD/data:/data vixitai.com/sparse:1.0 \
synth --out /data/scene.npz --shape 480x640 --duration 2.0 \
--mags 6.0 7.5 9.0 --seed 0
docker run --rm -v $PWD/data:/data vixitai.com/sparse:1.0 \
run --input /data/scene.npz --gt /data/scene_gt.npy --out /data/outThose two commands land on F1 0.779, 3 of 3 objects and 0.96 px
— the same canonical scene quoted in section 03, reproducible to four
decimals on a machine that has never seen this project. Add
--profile sparse-sky to run the same image against real sensor
data.
Output is a JSONL track record, a per-stage latency report, and a single self-contained HTML viewer — no server, no CDN, no build step — that plays the events, detection boxes and track trails from inside the output volume with one double-click.
NumPy and SciPy (BSD-3), expelliarmus (MIT) and
h5py (BSD-3), on a Debian-based Python 3.11 image. Every version is
pinned exactly, because a result you cannot reproduce is not a result: the same
scene gives identical accuracy to four decimal places across two major NumPy
releases and across one and two cores.
The whole stack is permissively licensed, with no copyleft anywhere in the tree and no encumbered weights — which is a deliberate position, not an accident of what happened to be convenient.