HarmonEyes H-SMART eye tracking technology measures and predicts cognitive load and fatigue to prevent safety risks during space missions.
An individual can use the AI-enabled wearable while performing tasks so that H-SMART collects real-time data and predicts future cognitive load and fatigue. Image: HarmonEyes
Astronauts face significant cognitive load challenges due to a combination of numerous psychological and environmental factors. Their environment includes altered gravity, radiation exposure, isolation and confinement, sleep loss, circadian rhythm disturbance, high operational workload, and communication delays. Cognitive workload becomes dangerously elevated when demands exceed attentional capacity, leading to increased error rates and a potentially detrimental narrowing of attention.
Research confirms anecdotal reports of “space fog,” highlighting observed decrements in processing speed, visual working memory, and sustained attention, particularly early in flight. High physical workloads, especially during complex tasks such as extravehicular activity (EVA), further deplete cognitive resources. This makes mission-critical decisions and responses to off-nominal or emergency scenarios highly vulnerable to performance degradation.
The Translational Research Institute for Space Health (TRISH) is a consortium between the Baylor College of Medicine’s Center for Space Medicine, Caltech, and MIT. The team collaborates closely with the NASA Human Research Program to link spaceflight operational needs with human health. TRISH has partnered with the University of Pennsylvania and the Australian Antarctic Division’s Polar Medicine Unit (PMU) to research the physical and mental effects of astronauts while working in extreme conditions. The goal is to better understand and reduce behavioral risks during long-duration spaceflight.
Research at Australia’s three Antarctic stations and at Macquarie Island currently requires Antarctic expeditioners to complete a range of cognitive tests on a laptop once a month during winter. Expeditioners also wear a small biosensor (Bio-Button) on their chest that measures physiological parameters, including heart rate, temperature, breathing, sleep, and movement.
Davis station Expedition Medical Officer, Dr. Anthony Rengel, holds a biosensor (left), and Casey station boilermaker Mike Hartman performs a cognitive test (right). Image: Nick Bye (left) and Andy Warton (right)
The cognitive tests are also being used on the International Space Station and on crewed commercial spaceflights. They measure the speed and accuracy of mental processes, such as thinking, learning, memory, and perception. The problem is that such tests require astronauts to use a laptop for a designated period. More modern wearable solutions can replace such laptops and monitor cognitive load in real time without interrupting the astronauts’ workflow, providing a better dataset for more accurate predictions.
Eyes are the windows of real-time cognitive load
“Essentially, the definition of cognitive load is the level of mental effort. It really gets to your working memory, how many things you’re storing, and your capacity to store those things and adapt,” said Melissa Hunfalvay, CPO and cofounder of HarmonEyes. “When somebody is in the ideal state, [they are in] moderate load, not low load. If you’re in low load for a period of time, what can happen is you can become inattentive, like you’re driving along a long stretch of road, you start looking elsewhere, and you drive off the road, for example. But moderate load means that you’re attentive and you’ve got enough room to make decisions and react appropriately if something else unexpected occurs. Whereas high load is, ‘I can’t think of doing.’ It’s like dual processing. ‘I can’t think of yet another thing. If you ask me another question right now, I’m going to say: Just stop, I’ve got to get this done first.’”
This chart shows an example of how performance depends on cognitive load. Image: HarmonEyes
The traditional way of measuring astronauts’ cognitive load is the NASA Task Load Index (TLX), a survey that asks astronauts to subjectively rate the mental, physical, and temporal demands, as well as their performance, effort, and frustration, during a task. Astronauts can rate their experience on a scale from “very low” to “very high,” with 21 gradations.
“The problem with a survey like that is a couple of major things. Number one, you can’t do it while you’re doing the task, so you have to think back as to how you felt, and it’s therefore post-pre, and you can do it pre and post, but it’s not during a task,” said Hunfalvay. “The second is that, being survey-based, you can write down whatever you want. So, there was a lot of malingering on it. They were looking for something that was real time, and that was scalable and accurate.”
That’s where HarmonEyes’ eye tracking solution comes into play. Tracking eye movement patterns provides real-time data on a person’s cognitive load and can predict when they will experience excessive workloads, helping prevent potential safety risks.
“The interesting thing about eye movements and eye tracking is that the eye is the only sense that is directly connected to your brain,” said Hunfalvay. “The really interesting thing about that is, no matter the task and things, if you’re stressed, we know that your heart rate goes up, right? That’s a common thing. So, if you’re loaded, we know that your pupil responds in a different way than if you’re not loaded.”
Eye movement provides insight into cognitive processing. Image: HarmonEyes
Pupil dilation, eye closure, blinking frequency, eye movement, and movement speed indicate a person’s mental workload. Continuously monitoring such data reveals patterns that can be used to understand a person’s cognitive load during a particular task and determine if the person is experiencing high cognitive load.
“There’s a signature for high load in your eye movements,” said Hunfalvay. “Those signatures persist across different people, but it’s your perception of what’s going on that may change, that influences that signal.”
This screenshot shows an individual’s current cognitive load is moderate, but the time to high load is 14 seconds. Image: HarmonEyes
HarmonEyes uses its eye tracking technology to answer the following questions: What is a person’s current cognitive load, and when will the cognitive load change? Low cognitive load can lead to inattentiveness; moderate cognitive load yields few performance effects; and high cognitive load results in slower responses and reduced decision-making capabilities. The technology also helps predict future load levels, when those loads will occur, and the likelihood of their being reached.
HarmonEyes H-SMART predicts cognitive load and fatigue
About 13 years ago, Hunfalvay and Adam Gross founded RightEye, a bespoke, FDA-cleared medical device. Optometrists, sports teams, performance teams, the military, and other groups use the device, which tracks eye movements, collects extensive data on individuals, and provides outcomes that help assess whether their eye movements are functional. Fast forward a decade, and that’s a whole lot of data.
“We have 13 million or so records right now, unique records in eye tracking that have a lot of different metadata associated with them that can help us further understand a person, their state, how they’re feeling, all those kinds of solutions,” said Hunfalvay.
A couple of years ago, Hunfalvay and Gross realized the massive data mine they had acquired and recognized a paradigm shift in the eye-tracking market.
“Devices have become less expensive, and they’ve proliferated on almost every device you have. You can do eye tracking on your phone, you can do it on your laptop, your webcam, they’re in smart glasses, they’re in HMDs — they’re everywhere,” said Hunfalvay. “So, the combination of our large data set and the hardware and software being available in all of these devices now is really where the concept of HarmonEyes came to be, which is: Can we use our massive data set to create solutions across all of these different devices, where eye tracking is now, and add value? So, things like looking at cognitive load or fatigue or motion sickness, right? All of those can be used and deployed using our SDK across all these different devices.”
However, Hunfalvay noted they faced a huge challenge: interoperability.
“The signal that comes from the phone is different than the signal that comes from the webcam on your laptop, which is different from the HMD. That is a massive area in which HarmonEyes is unique and different in that we are automating the ability to have the eye tracking signal be interoperable across all of these different devices,” she said. “And then we’re also using our massive data set to provide value on top of understanding that signal, so that we can tell people when they’re driving a car if they’re fatigued, or, more importantly, predicting when they’re going to become highly fatigued, so that they can pull over and take some corrective action.”
This screen displays intervention actions based on the current and predicted cognitive load and fatigue. Image: HarmonEyes
For NASA astronaut training, the HarmonEyes Human State Monitoring and Readiness Tool (H-SMART) will be deployed at the Antarctic and sub-Antarctic stations run by the Australian Antarctic Program. By measuring subtle changes in eye movements, H-SMART delivers passive, highly accurate, real-time monitoring of cognitive load and fatigue. In long-duration missions, where even minor performance decrements can be mission-critical, the technology provides astronauts and controllers with a critical early warning system and delivers interventions and corrective actions in real time. By applying advanced AI and machine learning for predictive modeling, H-SMART can adapt to the unique demands of extended space missions, where maintaining peak performance over months or years is mission-critical.
These solutions will be implemented to monitor and predict participants’ actual (rather than perceived) levels of cognitive load and fatigue under extreme environmental conditions.
“There are three times they’re wearing it,” said Hunfalvay. “One time is when they go for their monthly medical exams, and another time is when they’re doing what they call medical refreshes, where they do EMT, emergency types of medical training, because they’re in extreme environments. And then the third time is whenever they would like to wear it, so open.”
For now, the devices will measure astronauts in training on Earth, but the intention is to get them into space and track real-time cognitive load while working at the Space Station — or even on the Moon or Mars.
“That’s why eye tracking and eye movements and what we’re trying to do is so valuable, because it is such a sensitive measure,” said Hunfalvay. “It doesn’t just provide one output. It doesn’t just say whether you’re loaded. We’ve talked about being loaded today, but we’ve got 150 different user states — ways that we can use the same exact information to get a different signature for things like stress or resilience. All sorts of different characteristics. You do one thing, but you can get many different outputs that are valuable. So that’s the direction that we’re going.”
Watch a demonstration of HarmonEyes’ technology while an individual drives a car:
HarmonEyes
harmoneyes.com
Filed Under: AI Engineering Collective, Aerospace + defense, AI • machine learning
