Projects |

GridStage

Sat, 20 Jun 2026 00:00:00 +0000

2026 — in active development

Why I built this

As a dancer, formations were always the part nobody wanted to handle. How do you place everyone so the stage actually looks right? Will two people run into each other halfway through a transition? Those questions bothered me every time we put a piece together. I went looking for an open-source tool and there was none — and I was not about to pay a subscription for the commercial ones. So I built GridStage.

What it is

A top-down 2D stage where you place performers, add formations along a timeline synced to the music, and preview the walk-in transitions in 2D or 3D. It exports walk-chart PDFs (one page per dancer), videos, and GIFs. Several people can edit the same piece at once — live cursors, presence, and undo that only reverts your own changes — over a CRDT, a data structure where concurrent edits merge without a central referee. The part I care most about is the vision pipeline: pause a rehearsal video on a formation, click the four stage corners once, and the app detects the dancers, maps their foot positions onto the stage floor, and drops them into the editor as a normal editable formation. All of it runs in the browser.

Where it stands

v0.8.6 · 130 commits · ~16k lines of strict-mode TypeScript in a pnpm monorepo · 24 unit-test files plus a Playwright end-to-end suite. Three ways to run it, none requiring an account: a Windows/macOS desktop app, an installable PWA that works offline on a phone, and a Docker self-host stack.

Measured, not claimed:

60 performers × 4 formations in a permanent end-to-end stress test — dragging stays accurate and playback holds ~57 fps (CI floor 20).
Video → formation capture runs entirely client-side (YOLOX-nano through onnxruntime-web, WebGPU with a wasm fallback): 6 of 7 dancers recovered from a real six-dancer rehearsal photo, ~1.8 s per frame on CPU alone. Zero server cost, and no one’s rehearsal footage leaves their laptop.

Against ArrangeUs, the category leader, and the other commercial formation apps I audited before starting, these are the things they do not do at all (checked July 2026; their feature lists do move):

Capability	GridStage	ArrangeUs & other commercial apps
Real-time multi-user editing	CRDT (Yjs) — live cursors, presence, per-user undo	one editor at a time, files passed around
Formations captured from a rehearsal video	on-device detection + homography	nothing on the market does this
Rehearsal-vs-plan review	per-dancer drift, C-position centering, left–right asymmetry	—
Transition collision warnings	Hungarian matching + sweep-line crossing detection	—
3D preview	camera presets + follow-a-performer	limited
Video / GIF export with music	yes	limited
Price and licence	free, MIT, self-hostable	subscription, closed source

The place I am clearly behind is distribution, not capability: ArrangeUs has 457k+ users and years of tutorials, GridStage has an install page.

What’s next

Building a user community. The capability gap is closed; the distribution one is not. Demo videos and an in-app guide are already in progress, and the next step is getting the editor in front of real dance teams and cheer squads and running on their feedback — the work that decides whether GridStage stays a portfolio piece or becomes a tool people actually choreograph with.
Accounts and cloud sync. Today a piece lives on one machine and travels as a .gridstage file. This is the largest remaining gap against commercial tools, and the one architectural decision I still owe the project — self-host or a hosted backend.
Full rehearsal-vs-plan deviation report. The first cut compares one paused frame against the plan. Scanning a whole rehearsal into a per-formation error timeline is the version that closes the plan → rehearse → correct loop, and it is the hardest part for anyone to copy.
Stage lighting and cue sheets. Colored lights on the 2D stage and a 3D wash, cues on the timeline, exported as a printable cue sheet — the other half of what a stage manager actually needs.
Plugins and a community template library, so formation libraries can grow without me writing every one of them.

MakeNTU

Fri, 08 May 2026 00:00:00 +0000

May 8–10, 2026

The brief

AUO makes display panels, and the sponsor prompt came down to a single question: if the windshield itself becomes the display, what is actually worth putting on it? The easy answer is a bigger speedometer. We treated it as an interface problem instead — the glass should show what the driver’s current state calls for, and stay out of the way otherwise.

What we built

A first-person driving simulator that runs entirely in the browser, built on real map data: the Xinyi district road network pulled live from OpenStreetMap Overpass, routes from OSRM, and procedurally generated buildings filling the blocks in between. Four scenarios ship — city, highway, coastal drive, and a parked rest stop.

The windshield is composed in four layers, so the result reads as glass rather than as a game overlay:

Layer	What is in it
4 · Glass	Vignette, film grain, top tint, A-pillars and mirror silhouettes
3 · HUD	Frosted DOM/SVG panels — speed, compass, nav card, ETA, climate, now playing, minimap
2 · AR	POI cards projected into screen space onto building facades, fading when occluded
1 · World	Three.js — roads, buildings, cabin, traffic agents, route arrows, destination pillar

And the subsystems that make it more than a renderer:

Subsystem	How it works
Driver monitoring	MediaPipe FaceLandmarker blendshapes feed one state machine with two outputs. Hard warning: both eyes shut past 1.2 s, cooldown-gated so it does not machine-gun at a driver who is genuinely asleep. Continuous 0–1 fatigue score: rolling PERCLOS (a one-minute EMA of eyelid closure) combined with yawn rate — a jaw held open past 0.55 for 800 ms counts as a yawn, over a two-minute window.
Fatigue-driven HUD	The score is the interesting half. As fatigue climbs the HUD raises contrast on rear traffic, sheds non-essential chrome, and drifts wake-up particles across the glass — the display responds to the driver instead of shouting at them.
Head-pose lane-change intent	Looking more than 18° to one side for 800 ms reads as a shoulder check, and the panel surfaces rear traffic in that adjacent lane. Releases at a looser 8° with a 500 ms cooldown — hysteresis, so a brief glance forward does not flicker the panel on and off.
Door-opening safety gate	In rest mode the car is parked roadside with traffic flowing past. Asking to get out forward-simulates the traffic schedule: if any vehicle will come within 12 m of the door in the next 4 s the gate stays red, otherwise a tap confirms.
Voice + AI reservation call	Speech recognition is always listening. Say 「找餐廳」and five nearby POIs come back from Overpass; pick one by number and the in-car assistant dials it, speaking a reservation script over TTS while the overlay renders the transcript as chat bubbles. Phone numbers carry a separate spoken form so Chinese TTS reads them digit by digit instead of turning 09 into a place value.
Four-screen cabin	Each screen runs the same app with `?role=driver / passenger / left / right`. The driver publishes state snapshots to a small WebSocket relay that fans them out unparsed; each role applies its own camera offset — passenger 0.45 m inboard of the right door, side windows yawed ±90° — and keeps its own mouse-look on top, so a passenger can still turn their head independently.
Physical steering wheel	You drive it with an actual wheel. Steering angle comes off a six-axis IMU (three-axis accelerometer plus gyroscope) mounted on the wheel itself, so the rotation is read directly from how the wheel moves — no shaft encoder or mechanical end-stops to fabricate in three days. An Arduino Nano packages the reading as one JSON line per packet at 115200 baud, roughly 20 packets a second, which the browser reads through Web Serial. Every value is clamped on arrival so a noisy reading cannot push the vehicle physics past its own caps, and the 60 fps render interpolates between packets rather than stepping.

Demoing to the judges — hands on the laser-cut control rig, the simulator running on the screen in front of me.

My role

I was the team lead. The sponsor prompt named a display, not a product, so most of the work up front was deciding what a windshield should actually do — I ran that ideation with the team, and then led the software side through to a working demo. My own commits landed the highway, coastal and rest-stop scenarios, the driver-monitoring pipeline and everything the fatigue score drives, collision handling, the head-pose lane-change intent, and the exit gate.

Roughly 20k lines of TypeScript went in over the three days, with 194 unit tests across 16 files covering the pure logic — the state machines, geometry and traffic simulation are all written as pure functions precisely so they could be tested at hackathon speed instead of debugged through the UI.

Team 37, NoThingUnique — 劉允中, 賴泓安, 鄭竣陽, 胡軒綸, 張勝勛 — at our bench on the AUO track.

Results

First place, AUO corporate award.

RecoveryDAgger

Mon, 01 Sep 2025 00:00:00 +0000

Sep – Dec 2025

The problem

Imitation learning lets an agent pick up complex behaviour from expert demonstrations, but plain behaviour cloning drifts. Small errors push the agent into states the expert never demonstrated, and from there the mistakes compound.

DAgger fixes the drift by querying the expert on the states the learner actually visits. That works, but it asks constantly — and expert supervision is the expensive part of the whole setup.

What we built

RecoveryDAgger keeps DAgger’s interactive loop but adds a cheaper first response to trouble. In a risky state the agent does not call the expert straight away. It first runs a recovery policy that corrects itself locally by ascending the gradient of a learned Success Q-function — a network that estimates the probability the task will still be completed from the current state. The expert is queried only when recovery is unreliable, which is to say only when the state is genuinely novel.

The point is to spend expert supervision on new information, rather than on situations the agent could have dug itself out of.

My role

I led a four-person team and built the expert policy and the training pipeline myself, in PyTorch and Stable-Baselines3. The expert is a rule-based controller that steers toward the goal from the current state — deterministic, 100% success on the task — and it generates the 100-trajectory offline dataset every method in the paper starts from. Behaviour cloning on that dataset lands at roughly 26% success, which is exactly the gap online training has to close.

The design decision I care most about is how recovery actually produces an action. PointMaze has a continuous action space (forces along x and y), so the obvious version of recovery — enumerate candidate actions, take the arg max of the Success Q-function — is not available. We instead start from the action the imitation policy already proposed and run a few steps of gradient ascent on the Success Q-function with respect to the action, projecting back into the valid range after each step. Recovery is a local correction of the policy’s own intent, not a second controller fighting it.

The other decision was to stop trusting a single Q-network. One learned Success Q is only an approximation, and recovery follows its gradient, so its errors get amplified rather than averaged out. We trained an ensemble instead: the ensemble mean defines risk, the ensemble variance defines novelty, and the recovery objective subtracts a penalty λ·Var(Q) so recovery never climbs toward states the ensemble disagrees about. One family of networks ends up serving all three roles — risk detector, novelty detector, and recovery direction.

Results

Evaluated on a four-room PointMaze navigation task against ThriftyDAgger, the standard query-efficient baseline. Every method starts from the same behaviour-cloned policy.

Method	Expert queries	Recovery queries	Success rate
Behaviour cloning	—	—	0.26
ThriftyDAgger (baseline)	20,048	—	0.81
Single-Q recovery (ours)	2,504	9,232	0.86
Ensemble-Q recovery (ours)	2,693	9,296	0.88

Ensemble-Q recovery ends up more accurate than the baseline while querying the expert 7.4× less often. Partway through training the gap is even wider: both recovery variants reach ~0.7 success after about 2k expert queries, where ThriftyDAgger needs roughly 20k to get to the same place.

The interesting part is where the savings come from. Most of what a risk-gated method spends its query budget on is not genuinely unfamiliar — it is recoverable: local deviations near a doorway or a wall that the agent could fix by itself. Separating “novel” from “merely risky” is what buys the 7.4×.

The project placed 3rd among 30 teams, most of them graduate students, and the work has been accepted to IEEE Potentials.