Notre Dame, Ind. – October 06, 2026 -- Researchers at the University of Notre Dame are building an AI system called Voice4PIMD to translate the facial expressions, gestures, eye gaze, and vocalizations of nonverbal individuals into text or audio alerts caregivers can act on.
Cameras and sensors capture nonverbal cues for caregiver training
Led by computer engineering scientist Jarek Nabrzyski, the team's system uses cameras and sensors to record facial expressions, eye gaze, body movements, heart rate, and vocalizations. Caregivers train the model to interpret these signals for a specific individual, since the same gesture can carry different meanings from person to person.
The project targets people with profound intellectual and multiple disabilities (PIMD), a population the researchers say has received limited attention from research funding agencies. Ed Fennell, who has cared for his granddaughter Haley Hilt for nearly two decades, says she communicates constantly through subtle cues that trained caregivers learn to read—cues he fears others may miss once her current caregivers are no longer present.
Notre Dame expands testing through a South Bend community partner
Nabrzyski's team is beginning additional testing with families through LOGAN Community Resources in South Bend, Indiana. Researchers are also examining whether the same personalized AI approach could apply to people with autism and dementia, expanding the potential caregiver and clinical market beyond the PIMD population.
The technology remains in early-stage research, with no commercial release date disclosed. Its core design goal is continuity: preserving an individual's established communication patterns as caregivers change over a person's lifetime.