Emotion correlation-based multimodal cognitive load quantification

Modern human–computer interaction (HCI) systems are characterized by high information density and complex tasks. In such systems, information is often presented in large volumes at rapid speeds, requiring users to process and comprehend it quickly. At the same time, tasks are intricate and multifaceted, involving multiple steps and decision-making processes. Monitoring the cognitive load in HCI tasks is of paramount importance because it can significantly enhance system performance and reduce the likelihood of operational errors. When users experience excessive cognitive load, their ability to process information and make decisions may be impaired, leading to decreased efficiency, increased error rates, and even system failures. By effectively monitoring cognitive load, system designers can identify potential bottlenecks and optimize the user experience to ensure smooth and efficient interaction. Cognitive load theory, supported by extensive causal research, provides a framework for understanding and categorizing cognitive load into three distinct types: internal, external, and associative. Internal cognitive load is closely tied to the brain’s information processing mechanisms, particularly memory resources. It reflects the mental effort required to encode, store, and retrieve information from memory. Conversely, external cognitive load stems from the presentation and difficulty of learning materials. It is influenced by how information is organized and displayed to the user, as well as the inherent complexity of the task. Both internal and external cognitive loads are critical factors that affect a user’s cognitive state during HCI tasks. However, directly measuring these two types of cognitive load is often challenging and time-consuming because it requires invasive or complex experimental setups. Given the limitations of direct measurement, this study proposes an innovative approach by leveraging the relationship between negative emotions and multimodal signals as a proxy for direct cognitive load measurement. Emotions, especially negative ones, are inherent psychological products of interactive tasks and are closely linked to cognitive processes. When users experience a high cognitive load, they often exhibit emotional responses such as frustration, anxiety, or stress. These emotional states can be reflected in various physiological and behavioral signals such as facial expressions, voice tone, eye movements, and physiological indicators such as heart rate and skin conductance. By capturing and analyzing these multimodal signals, we can indirectly infer the cognitive load experienced by users. To simulate the occurrence of cognitive load in HCI tasks, three repeatable visual stimuli were designed: digit memory, tracking, and combined memory-tracking tasks. The digit memory task focuses on memory by requiring participants to remember sequences of numbers. The tracking task emphasizes attention by asking participants to follow a moving target on the screen. The combined memory-tracking task integrates both memory and attention by requiring participants to remember numbers while simultaneously tracking a moving target. The difficulty of each task type increases progressively, and the overall stimulus difficulty was modeled as a linear increase. This design allowed for a systematic investigation of how cognitive load evolves as task complexity increases. A total of 53 participants completed the designed paradigm experiments. They were required to complete the NASA-TLX workload assessment and PANAS emotion assessment scales. Several multimodal and multichannel signals during the experiment were recorded for subsequent analysis, including physiological data, eye-tracking data, and behavioral responses. This study proposes a novel cognitive load measurement method to quantify the psychological stress generated by subjects during HCI. By analyzing the differences in emotion-related indices, conducting quantitative calculations, and performing classification verification, the study aims to establish a robust model for cognitive load assessment. The results demonstrated that the proposed progressive paradigm successfully induced a psychological load in the subjects. A significant correlation was found between the NASA-TLX scores and PANAS negative emotion scores, indicating that as the cognitive load increased, participants tended to experience more negative emotions. During the experiment, numerous multimodal and multichannel signals exhibited significant differences, suggesting that emotion-related signals undergo marked changes during cognitive load induction. According to the proposed calculation method, a significant correlation exists between the quantitative load and subjective load-level reports of the subjects. This finding validates the effectiveness of the proposed method in capturing cognitive load variations. Using different signals, the linear classification model achieved an accuracy of over 93%, which demonstrates its potential to accurately predict and assess cognitive load in real time. The ability to dynamically track the cognitive load can provide valuable feedback to system designers, enabling them to make timely adjustments and optimizations to the user interface and task design. Overall, the proposed model holds great promise for enhancing the user experience in HCI by providing a noninvasive and efficient way to monitor cognitive load.