Parallax AR helmet prototype displaying procedure guidance overlay

MHCI Capstone · NASA Johnson Space Center 2015

Parallax AR System

AR guidance system for astronaut procedures aboard the International Space Station.

↗ Spring Research Booklet ↗ Summer Prototyping Booklet

The Problem Space

Astronauts aboard the ISS regularly perform high-stakes maintenance, experimentation, and repair operations — often as novice users following dense paper procedures, in microgravity, with bulky gloves and no internet connection. A single misread step can cascade into hours of lost time or equipment failure.

Our team at Carnegie Mellon (Team Parallax) was assigned the NASA capstone: understand how procedure execution breaks down in constrained environments, then design a forward-facing system to improve it.

Research: Going to the Work

Rather than interviewing astronauts directly, we embedded ourselves in analogous high-stakes procedural environments where we could actually observe work in progress.

Two team members navigating a smoke maze during firefighter training at a fire station — Experiential learning at a fire station — navigating a smoke maze with full gear to feel procedural breakdown firsthand.

Sequence model diagram from Habitat for Humanity contextual research — Sequence model from our Habitat for Humanity site visit, capturing how volunteers execute construction procedures with incomplete knowledge.

We conducted contextual inquiry across biotech manufacturing, construction (Habitat for Humanity), scuba diving, and a neurosurgery interview. Each setting exposed a different failure mode: misread steps, tool-finding interruptions, loss of place during distractions.

Flow model diagram mapping interactions between a neurosurgeon and the entities in their operating environment — Flow model from our neurosurgery interview — mapping all the entities a surgeon must track while performing a procedure.

Synthesis: 1,000+ Notes

We synthesized our field research into an affinity diagram — over a thousand individual observations grouped by theme. The clusters pointed directly at the design opportunity: astronauts need a system that tracks their place in a procedure and directs them to required tools, without adding cognitive load.

Large affinity diagram with hundreds of color-coded sticky notes arranged on a wall — Affinity diagram synthesis. Over 1,000 field notes organized into themes that shaped our design direction.

From the affinity work, we generated 150+ candidate design features and mapped them across an impact/feasibility matrix — focusing our prototype scope on what could be built in months and deliver the most mission value.

Impact vs feasibility scatter plot matrix with design features plotted across four quadrants — Impact/feasibility matrix — the upper-right quadrant defined our prototype scope.

Prototyping: Low to High Fidelity

Early prototypes tested the core interaction concepts with everyday participants before any hardware was involved.

Low-fidelity prototype test: participant follows recipe instructions on a projected screen while an arrow points to ingredients — Lo-fi AR test — participants followed recipes using screen-projected instructions and a directional arrow. Validated the core guidance metaphor.

Low-fidelity prototype with audio instructions and wristband display — Wristband form-factor prototype testing audio instructions + small display. Audio proved too interruption-heavy for complex tasks.

Medium-Fidelity Prototype

The mid-fi prototype put a functional heads-up display on a participant’s head for the first real usability tests. We adapted an ISS rodent habitat maintenance procedure and had participants work through it on a cardboard mockup of the habitat.

Medium-fidelity helmet prototype: a bike helmet with a phone holder mounting an iPhone in front of the visor — Helmet form factor — an iPhone mounted at eye level in a custom holder, rendering the UI directly in the user's line of sight.

Medium-fidelity user interface showing a procedure step on a phone screen — I built the mid-fi UI as a web app rendering Google Slides — each slide a procedure step. A separate experimenter dashboard let us jump between steps based on user actions.

High-Fidelity Prototype

For the final prototype, we replaced the phone with teleprompter glass for true AR projection, added an iPad mini for a larger interface, and wired in speech recognition and Bluetooth beacon-based tool proximity detection.

Final high-fidelity helmet with teleprompter glass projecting the AR interface into the user's field of view — Final helmet. Teleprompter glass projects the procedure UI into the user's view. Voice commands advance steps hands-free.

Final user interface showing procedure step with directional arrows pointing toward required tools — Final UI. The backend tracks procedure progress and communicates with a Beacon reader to surface directional arrows toward the next required tool.

IoT hardware components: Intel Edison, two Raspberry Pis, and Bluetooth Beacons laid out on a table — The IoT stack — Intel Edison (Python backend), two Raspberry Pis (frontend + Beacon reader), and Bluetooth Beacons affixed to tools for proximity sensing.

Final Deliverable

Team member wearing the completed Parallax AR helmet prototype, performing a procedure on the rodent habitat mockup — The completed Parallax system in use. The helmet, speech recognition, and Beacon proximity layer work together to guide a user through an ISS maintenance procedure — hands-free.

Role

Led the project and owned most of the software stack — the AngularJS prototype interface, the Edison Python backend, and the Windows Phone speech module. Kept the team on scope through a semester-long NASA engagement.