Research

The Cognitive Systems Lab uses an interdisciplinary and integrative approach, which combines highly-realistic, interactive, and immersive mixed-reality environments with high-resolution, multimodal physiological measurements, and state-of-the-art AI approaches. 

We conduct research in the following four areas: 

• Affective & multisensory processing

• Clinical AI applications

• High-risk decision-making

• Multi-level AI alignment

Affective & multisensory processing

How do we process facial movement into emotions and social signals? How are signals from different modalities integrated for efficient processing? Would algorithms and robotic systems "understand" the complex world of affective and multisensory signals in the same way as the human brain? 

Current work in the lab focuses on how to parametrize affective signals used for communication - both via the face and via touch, how reliable affective datasets really are, and whether Koreans are able to categorize other (East Asian) populations. 

Publications:

• M. Nusseck, D. W. Cunningham, C. Wallraven, and H. Bülthoff. The contribution of different facial regions to the recognition of conversational expressions. Journal of Vision, 8(8):1:1-23, 06 2008.

• D. W. Cunningham and C. Wallraven. Temporal information for the recognition of conversational expressions. Journal of Vision, 9(13):1-17, 12 2009

• C. Dahl, C. Wallraven, H. Bülthoff, and N. Logothetis. Humans and macaques employ similar face-processing strategies. Current Biology, 19(6):509-513, 03 2009.

• C. Dahl, N. Logothetis, H. Bülthoff, and C. Wallraven. The thatcher illusion in humans and monkeys. Proceedings of the Royal Society of London B, 277(1696):2973-2981, 10 2010.

• N. Gaissert, H. H. Bülthoff, and C. Wallraven. Similarity and categorization: From vision to touch. Acta Psychologica, 138:219-230, 07 2011.

• A. Shin, S. Lee, H. Bülthoff, and C. Wallraven. A morphable 3d-model of korean faces. In Proceedings of SMC 2012, 10 2012.

• K. Kaulard, D.W. Cunningham, H.H. Bülthoff, and C. Wallraven. The MPI facial expression database - a validated database of emotional and conversational facial expressions. PLoS One, 7(3):e32321, 01 2012.

• A. Aubrey, D. Marshall, P .Rosin, J. Vandeventer, D. Cunningham, and C. Wallraven. Cardiff conversation database (CCDB): A database of natural dyadic conversations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), V&L Net Workshop, pages 0-0, 06 2013.

• C. Wallraven, L Dopjans. Visual experience is necessary for efficient haptic face recognition. NeuroReport 24 (5), 254-258, 2013.

• C Wallraven. Touching on face space: Comparing visual and haptic processing of face shapes. Psychonomic bulletin & review 21 (4), 995-1002, 2014.

• B. Browatzki, V. Tikhanoff, G. Metta, H. Bülthoff, and C. Wallraven. Active In-Hand Object Recognition on a Humanoid Robot. IEEE Transactions on Robotics, 30: 1260-1269, 2014.

• H. Lee Masson, J. Bulthé, H. Op De Beeck, and C. Wallraven. Visual and Haptic Shape Processing in the Human Brain: Unisensory Processing, Multisensory Convergence, and Top-Down Influences. Cerebral Cortex 2015; doi: 10.1093/cercor/bhv170

• H. Lee Masson, C. Wallraven*, and L. Petit. Can touch this: cross-modal shape categorization performance is associated with micro-structural characteristics of white matter association pathways. Human Brain Mapping, 2017.

• J. Kang, B. Ham, and C Wallraven. Cannot avert the eyes: Reduced attentional blink toward others’ emotional expressions in empathic people. Psychonomic Bulletin and Review, 24: 810-820, 2017

• D Derya, J Kang, DY Kwon, C Wallraven. Facial Expression Processing Is Not Affected by Parkinson’s Disease, But by Age-Related Factors. Frontiers in Psychology 10, 2458, 2019

• I Bülthoff, W Jung, RGM Armann, C Wallraven. Predominance of eyes and surface information for face race categorization. Scientific reports 11 (1), 1927, 2022.

• H Kang, T Kang, C Wallraven. Putting vision and touch into conflict: results from a multimodal mixed reality setup. IEEE Transactions on Visualization and Computer Graphics 29 (12), 5224-5234, 2022.

• Derya, Dilara; Wallraven, Christian. Charting decodability of dynamic facial expressions in young and old adults: similarities and differences. Experimental Neurobiology. 34(5): 224-234, 2024.

• C Malak, C Wallraven. The importance of external features for categorizing ethnicity: Can Koreans identify Korean, Japanese, and Chinese faces? Perception, 2026.

• Bae, Jaehyeong; Wallraven, Christian. Can impersonal touch replace interpersonal touch? An investigation using the rubber hand illusion. PLoS One 20(5): e0319433, 2026.

• Bae, Jaehyeong; Wallraven, Christian. More Than a Material: Social Interpretation Shapes the Hedonic Evaluation of Human Hand Touch. IEEE Transactions on Haptics, 2026.

Clinical AI applications

Here, we apply our experience in understanding the human cognitive system as well as in advanced AI methods in order to validate, improve, or design novel applications for AI in the clinical context.

Publications:

• Ryu, Hyeongsuk; Moon, Hyeongjun; Browatzki, Björn; Wallraven, Christian. Retinal vessel detection using deep learning: a novel Directnet architecture. Korean J Vis Sci, 20(2), 151-159. 2018.

• Ryu, Hyongsok; Wallraven, Christian. Learning how to recognize objects with a simulated scotoma: an eye-tracking analysis. Investigative Ophthalmology & Visual Science, 59(9), 2164-2164. 2018.

• Kang, June; Derva, Dilara; Kwon, Do-Young; Wallraven, Christian. Voluntary and spontaneous facial mimicry toward other’s emotional expression in patients with Parkinson’s disease. PloS one, 14(4), e0214957. 2019.

• Derya, Dilara; Kang, June; Kwon, Do Young; Wallraven, Christian. Facial Expression Processing Is Not Affected by Parkinson’s Disease, But by Age-Related Factors. Frontiers in Psychology, 10, 2458. 2019.

• Kang, Taeho; Chen, Yiyu; Fazli, Siamac; Wallraven, Christian. EEG-based prediction of successful memory formation during vocabulary learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(11), 2377-2389. 2020.

• Browatzki, Björn; Lies, Jörn-Philipp; Wallraven, Christian. Encoder-Decoder Networks for Retinal Vessel Segmentation Using Large Multi-scale Patches. International Workshop on Ophthalmic Medical Image Analysis, 42-52. 2020.

• Ryu, Hyeongsuk; Ju, Uijong; Wallraven, Christian. Myopia-correcting lenses decrease eye fatigue in a visual search task for both adolescents and adults. PLoS One, 16(10), e0258441. 2021.

• Cho, Daehyun; Wallraven, Christian. Do pre-processing and augmentation help explainability? a multi-seed analysis for brain age estimation. International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, 12-21. 2022.

• Seo, Jun-Seok; Chen, Yiyu; Kwon, Do-Young; Wallraven, Christian. Single-view, video-based diagnosis of Parkinson's Disease based on arm and leg joint tracking. 2022 International Conference on Mechanical, Automation and Electrical Engineering (CMAEE), 172-176. 2022.

• Ryu, Hyeongsuk; Ju, Uijong; Wallraven, Christian. Decoding visual fatigue in a visual search task selectively manipulated via myopia-correcting lenses. Frontiers in Neuroscience, 18, 1307688. 2024.

• Seo, Jun-Seok; Chen, Yiyu; Kwon, Do-Young; Wallraven, Christian. Single-View, Video-Based Diagnosis of Parkinson’s Disease via Margin of Stability Gait Analysis. International Conference on Pattern Recognition and Artificial Intelligence, 354-368. 2024.

• D Cho, YH Kim, S Ahn, J Oh, C Wallraven. Multimodality for Diagnosis of Asian Choroidal Vasculopathy: Results from a Novel Dataset and Deep-Learning Experiments. MICCAI Workshop, 2024.

• D Derya, C Wallraven. Resting state network connectivity patterns in early aging: late middle-age adults contrasted with young adults. Experimental Neurobiology 33 (6), 282, 2024.

• T Kang, Y Chen, C Wallraven. I see artifacts: ICA-based EEG artifact removal does not improve deep network decoding across three BCI tasks. Journal of Neural Engineering 21 066036, 2024.

• Ju, Uijong; Wallraven, Christian. Decoding the dynamic perception of risk and speed using naturalistic stimuli: A multivariate, whole‐brain analysis. Human brain mapping, 45:4, e26652, 2024.

• Ryu, Hyeongsuk; Ju, Uijong; Wallraven, Christian. Decoding visual fatigue in a visual search task selectively manipulated via myopia-correcting lenses. Frontiers in Neuroscience 18, 1307688, 2024.

• J Park, C Wallraven. Investigating Dynamics of Subjective Anxiety and Behavior due to Personal Space Violations and COVID-19-related Stressors in a Social VR Simulation. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2025.

• Cho Heeseung; Chen, Yiyu; Wallraven, Christian. Self-reported binary gender prediction from personality traits: Alignment between machine learning importance and classical effect sizes. PloS one 21 (8), e0355193. 2026

• Ryu, Hyeongsuk; Wallraven, Christian; Kim, Junsuk. A multimodal analysis of eye-movement and EEG responses during short-term blue-light filtering lens wear in a reading task. Visual Neuroscience 43,  e008. 2026.

• Cho, Daehyun; You, Sung-Hye; Kim, Bo Kyu; Park, Arim; Wallraven, Christian. Establishing the robustness metric as a scalable proxy for clinical relevance in medical AI explainability. Medical Image Analysis, 104159. 2026.

High-risk decision-making

How do we make high-risk decisions under pressure? How do we choose to cheat in a game? How does this compare with current state-of-the-art models - are these "aligned"? 

This research line combines deep learning, virtual reality, psychophysiological measures, and neuroimaging to chart and model human and AI decision-making in high-risk situations.

We are currently recording several large-scale datasets with a unique combination of psychophysiological measures (including eye-tracking), EEG, as well as detailed behavioral data - stay tuned!!

Publications:

• U. Ju, J. Kang and C. Wallraven. Testing intuitive decision-making in VR: personality traits predict decisions in an accident situation. In Proceedings of IEEE Conference on Virtual Reality (IEEE VR2016), IEEE, 2016.

• U. Ju, J. Kang and C. Wallraven. To brake or not to brake? Personality traits predict decision-making in an accident situation. Frontiers in Psychology, 2019.

• U. Ju, J. Kang and C. Wallraven. You or Me? Personality Traits Predict Sacrificial Decisions in an Accident Situation. IEEE Transactions on Visualization and Computer Graphics, 2019.

• Y. Chen and C. Wallraven. Pop or not? EEG-correlates of risk-taking behavior in the balloon analogue risk task. In Proceedings of the 5th International Winter Conference on Brain Computer Interfaces, IEEE, 2017.

• T Kang, Y Chen, S Fazli, C Wallraven. EEG-Based prediction of successful memory formation during vocabulary learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering 28 (11): 2377-2389, 2020.

• Y Chen, S Fazli, C Wallraven. An EEG Dataset of Neural Signatures in a Competitive Two-Player Game Encouraging Deceptive Behavior. scientific data 11 (389), 2024.

• J Jang, C. Wallraven. Whom to Save? A Novel, Realistic Paradigm for Studying Human Decision-Making in Moral Dilemmas. IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2024.

• HS Ryu, U Ju, C. Wallraven. Predicting Future Driving Decisions in an Accident Situation From Videos: A Combined Behavioral, Eye Gaze, and Computational Analysis. IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2024.

• J Park, C Wallraven. Investigating Subjective Anxiety Dynamics due to Personal Space Violations and COVID-19-related Stressors in a Social VR Simulation. IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2024.

• Ryu, Hoe Sung; Ju, Uijong; Wallraven, Christian. Seeing What Matters: Attentional (Mis-) Alignment Between Humans and AI in VR-Simulated Prediction of Driving Accidents. IEEE Transactions on Visualization and Computer Graphics. 2025.

• Ju, Uijong; Ryu, Hoe Sung; Wallraven, Christian. Predicting emergency decision-making in conditionally automated driving from personality traits and heart rate. Transportation Research Part F: Traffic Psychology and Behaviour, 113, 570-585, 2025.

• Ryu, Hoe Sung; Wallraven, Christian. Mind the Gap: Quantifying and Aligning HumanAI Visual Attention for Accident Anticipation. AAAI, 2026.

Multi-level AI alignment

In this newest one among our research foci, we are looking at how to characterize and improve the alignment between humans and AI - this concerns the development of explainable AI approaches to open the black-box, all the way up to measuring and comparing how humans and AI fare on the exact same task at perceptual, cognitive, decision-making, all the way up to moral levels of analysis. 

Publications:

• Moon, Hyungjun; Browatzki, Björn; Blais, Caroline; Wallraven, Christian. Deep neural networks process similar facial features compared to humans in facial expression recognition. IBRO Reports, 6, S193-S194. 2019.

• Browatzki, Bjorn; Wallraven, Christian. 3fabrec: Fast few-shot face alignment by reconstruction. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6110-6120. 2020.

• Park, Serin; Wallraven, Christian. Comparing facial expression recognition in humans and machines: Using CAM, GradCAM, and extremal perturbation. Asian Conference on Pattern Recognition, 403-416. 2021.

• Kim, Doo Yon; Wallraven, Christian. Label quality in affectnet: results of crowd-based re-annotation. Asian conference on pattern recognition, 518-531. 2021.

• Ryu, Hoe Sung; Ju, Uijong; Wallraven, Christian. Predicting decision-making in the future: Human versus machine. Asian conference on pattern recognition, 127-141. 2021.

• Lee, Hyun Seung; Wallraven, Christian. Visualizing the embedding space to explain the effect of knowledge distillation. Asian Conference on Pattern Recognition, 462-475. 2021.

• Mametkulov, Maxim; Artykbayev, Abay; Koishigarina, Darina; Kenessova, Amina; Razikhova, Kamilla; Kang, Taeho; Wallraven, Christian; Fazli, Siamac. Explainable machine learning for memory-related decoding via TabNet and non-linear features. 2022 10th International Winter Conference on Brain-Computer Interface (BCI), 1-7. 2022.

• Shin, Soomin; Kim, DooYon; Wallraven, Christian. Contextual modulation of affect: Comparing humans and deep neural networks. Companion Publication of the 2022 International Conference on Multimodal Interaction, 127-133. 2022.

• Ryu, Hoe Sung; Ju, Uijong; Wallraven, Christian. Predicting Future Driving Decisions in an Accident Situation From Videos: A Combined Behavioral, Eye Gaze, and Computational Analysis. 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 391-392. 2024.

• Malak, Cansu; Wallraven, Christian. Humans versus machines: Distinguishing Korean, Chinese, and Japanese faces via internal and external features. Journal of Vision, 25(9), 2027-2027. 2025.

• Ryu, Hoe Sung; Ju, Uijong; Wallraven, Christian. Seeing What Matters: Attentional (Mis-) Alignment Between Humans and AI in VR-Simulated Prediction of Driving Accidents. IEEE Transactions on Visualization and Computer Graphics. 2025.

• Ryu, Hoe Sung; Wallraven, Christian. Mind the Gap: Quantifying and Aligning Human-AI Visual Attention for Accident Anticipation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37887-37895. 2026.