BasketGame — food items falling from the top, the player catching them with colored baskets

CMU Honors Thesis · FLAIRS 2014 2012

Improving Task Performance in an Affect-mediated Computing System

Can a computing system that detects and responds to a user's emotional state measurably improve their task performance?

↗ FLAIRS 2014 Publication ↗ Full Thesis (KiltHub)

Abstract

In this project, I researched how an affect-mediated system, a computing system that adapts its actions and behavior to the emotional state of its users, can improve their abilities to complete tasks and meet their goals.

In particular, I applied facial expression recognition, one method for estimating a human’s emotional state, to a children’s game, giving it the ability to adjust its difficulty based on its player’s perceived unease. Through experimentation with this game, we sought to understand whether affect-mediation can help users achieve their goal of winning the game, and whether in general affect-mediated systems can aid in user task completion.

The written thesis spans 97 pages.

The Question

Can a computing system that reads a user’s emotional state and adapts its behavior in real time actually help that user perform better? This senior honors thesis investigated that question by building the system and running an IRB-approved study with children.

The domain: affect-mediated computing — systems that change their behavior based on detected emotional state. The mechanism: facial expression recognition. The test bed: a children’s game with five escalating difficulty phases.

Facial Expression Recognition Engine

I built a recognition engine on top of Jason Saragih’s FaceTracker, which could localize and track a face in a video frame. On top of that, I trained a support vector machine (SVM) using the Cohn-Kanade+ emotion-labeled face dataset to classify Paul Ekman’s seven fundamental emotions from the tracked facial geometry.

Face mesh overlay on a participant expressing surprise — points tracked across forehead, eyes, nose, and mouth — The Modified FaceTracker outputting a facial mesh. The SVM classifier reads geometry changes across these points to identify emotional state.

BasketGame

The game — BasketGame — was designed specifically as a controllable test environment. Colored food items fall from the top of the screen; the player catches them with matching baskets. Five phases of increasing difficulty require the player to manage more variety, faster fall speeds, and less predictable positions.

The key mechanic: if the FER engine detected frustration or anger, the game would hold the current phase rather than advancing — keeping the player in a range of challenge they could manage, rather than pushing them to overwhelm.

Left: early prototype with basic shapes. Right: final version with illustrated food items and sound effects. — Left: first prototype testing core mechanics. Right: final build with illustrated assets and sound.

BasketGame screenshot showing food items falling and colored baskets at the bottom — BasketGame in play. Five phases of increasing difficulty — the affect engine decides whether to hold or advance each phase.

Timeline diagram comparing the control and affective engine difficulty curves across a session — The two engine modes. The control engine advances difficulty on a fixed schedule; the affective engine holds phases when negative emotion is detected.

Experiment

Participants were four and five-year-old children from CMU’s Children’s School. Paired experimental-control sessions: one group played with the affective engine enabled, the other with a fixed-difficulty control engine.

Results across three measures:

Bar chart: affect condition shows higher catch/total ratio than control — Catch/total ratio — affect condition participants caught a higher proportion of items overall.

Bar chart: affect condition shows fewer mean struggles than control — Struggle count — affect condition participants experienced fewer consecutive miss sequences.

Scatter plot showing struggles more evenly distributed across the session in the affect condition — Struggle distribution — struggles in the affect condition were spread more evenly across the session, suggesting sustained engagement rather than concentrated difficulty spikes.

The results pointed toward a state psychologist Mihaly Csikszentmihalyi calls flow — the condition of being optimally challenged, neither bored nor overwhelmed. The affect-mediated engine appeared to nudge participants toward it.

Csikszentmihalyi's flow diagram showing flow channel between anxiety and boredom axes — Csikszentmihalyi's flow model. The thesis concludes that the affective engine may be capable of inducing and sustaining flow in its users.

Role

Built everything from the ground up — the recognition engine, the game, the study protocol — and ran it with real participants under IRB approval. Published at FLAIRS 2014.