Computer vision shoulder range-of-motion app for a clinical trial

The situation

A research group had a shoulder mobility pilot to run and an app that could not measure anything. Development had stopped: the branch that actually worked was 161 files ahead of main, and the pose-estimation layer was four separate engines, none of which ran. Two of the bundled binaries carried licences the study could not use — one copyleft, one a commercial SDK tied to an individual rather than the institution. Before any new feature could be scoped, somebody had to say plainly what was salvageable.

What the work involved

• Established a contractual baseline tag at handover, so the fee discussion referenced a fixed state of the repository rather than a moving one.
• Removed the four non-functional pose engines, the never-compiled inference plugin, and 66 MB of licence-encumbered binaries — 45 files and roughly 2,700 lines — while leaving the working questionnaire app untouched.
• Established that Apple's monocular 3D body-pose lift collapses at end-range elevation, regressing the raised wrist onto the head and capping readings near 120–140°. No downstream arithmetic can correct a wrong skeleton, so the measurement was rebuilt in the 2D image plane against the patient's own trunk axis — camera-tilt invariant, and valid for exactly the reason clinical goniometry is.
• Built the capture module as a native Swift and SwiftUI module on Apple Vision, with no third-party licences: automatic repetition detection from the elevation curve, and gates for framing, headroom, distance and stance that refuse to record a shot that would not be measurable.
• Reported compensations — trunk lean, extension, rotation, shoulder hike, bent elbow — alongside the angles rather than silently subtracting them, with per-trial confidence downgraded for out-of-plane movement.
• Saved a per-frame keypoint trace for every repetition as versioned JSON, which became the validation dataset the study needed and which the principal investigator accepted in place of storing video.
• Redesigned the data protection posture as local-first: assessments stay on the device and leave only by explicit clinician export, removing hosting cost and data-residency questions from the pilot entirely. The server remains in the repository, hardened and dormant, for the later multi-site phase.
• Removed the legacy authentication stub outright, so no code that stores a password exists in the application.
• Shipped results on screen and in the exported PDF, a JSON export for the postdoc, full trilingual interface strings, and a one-page operator guide written for clinicians rather than developers.

Outcome

Delivered against the agreed scope and confirmed working on the target iPad. Measurement geometry is covered by a shell-runnable assertion suite in the repository plus unit tests in the measurement library, and the coordinate-space conventions are documented so the next developer does not have to re-derive them.

Stack

React Native · Swift · Apple Vision · Computer vision · Privacy by design