Cookies Policy
The website need some cookies and similar means to function. If you permit us, we will use those means to collect data on your visits for aggregated statistics to improve our service. Find out More
Accept Reject
  • Menu
Publications

Publications by Tamas Karacsony

2024

Deep learning methods for single camera based clinical in-bed movement action recognition

Authors
Karácsony, T; Jeni, LA; de la Torre, F; Cunha, JPS;

Publication
IMAGE AND VISION COMPUTING

Abstract
Many clinical applications involve in-bed patient activity monitoring, from intensive care and neuro-critical infirmary, to semiology-based epileptic seizure diagnosis support or sleep monitoring at home, which require accurate recognition of in-bed movement actions from video streams. The major challenges of clinical application arise from the domain gap between common in-the-lab and clinical scenery (e.g. viewpoint, occlusions, out-of-domain actions), the requirement of minimally intrusive monitoring to already existing clinical practices (e.g. non-contact monitoring), and the significantly limited amount of labeled clinical action data available. Focusing on one of the most demanding in-bed clinical scenarios - semiology-based epileptic seizure classification - this review explores the challenges of video-based clinical in-bed monitoring, reviews video-based action recognition trends, monocular 3D MoCap, and semiology-based automated seizure classification approaches. Moreover, provides a guideline to take full advantage of transfer learning for in-bed action recognition for quantified, evidence-based clinical diagnosis support. The review suggests that an approach based on 3D MoCap and skeleton-based action recognition, strongly relying on transfer learning, could be advantageous for these clinical in-bed action recognition problems. However, these still face several challenges, such as spatio-temporal stability, occlusion handling, and robustness before realizing the full potential of this technology for routine clinical usage.

2024

Brain Anterior Nucleus of the Thalamus Signal as a Biomarker of Upper Voluntary Repetitive Movements in Epilepsy Patients

Authors
Lopes, EM; Pimentel, M; Karácsony, T; Rego, R; Cunha, JPS;

Publication
2024 IEEE 22ND MEDITERRANEAN ELECTROTECHNICAL CONFERENCE, MELECON 2024

Abstract
The Deep Brain Stimulation of the Anterior Nucleus of the Thalamus (ANT-DBS) is an effective treatment for refractory epilepsy. In order to assess the involvement of the ANT during voluntary hand repetitive movements similar to some seizure-induced ones, we simultaneously collected videoelectroencephalogram ( vEEG) and ANT-Local Field Potential (LFPs) signals from two epilepsy patients implanted with the PerceptTM PC neurostimulator, who stayed at an Epilepsy Monitoring Unit (EMU) for a 5 day period. For this purpose, a repetitive voluntary movement execution protocol was designed and an event-related desynchronisation/synchronisation (ERD/ERS) analysis was performed. We found a power increase in alpha and theta frequency bands during movement execution for both patients. The same pattern was not found when patients were at rest. Furthermore, a similar increase of relative power was found in LFPs from other neighboring basal ganglia. This suggests that the ERS pattern may be associated to upper limb automatisms, indicating that the ANT and other basal ganglia may be involved in the execution of these repetitive movements. These findings may open a new window for the study of seizure-induced movements (semiology) as biomarkers of the beginning of seizures, which can be helpful for the future of adaptive DBS techniques for better control of epileptic seizures of these patients.

2024

NeuroKinect4K: A Novel 4K RGB-D-IR Video System with 3D Scene Reconstruction for Enhanced Epileptic Seizure Semiology Monitoring

Authors
Karácsony T.; Fearns N.; Vollmar C.; Birk D.; Rémi J.; Noachtar S.; Silva Cunha J.P.;

Publication
Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS

Abstract
Epileptic seizures are clearly characterized by their displayed behavior, the semiology, which is used in diagnosis and classification as a base for therapy. This article presents a novel 4K 3D video recording and reviewing system for epilepsy monitoring, introducing a novel perspective and allowing continuous recording and review of 3D videos in the epilepsy monitoring unit (EMU), providing significantly more detail than the current clinical systems, which can lead to the recognition of more Movements of Interest (MOIs) and may reduce inter-rater variability. To put the system to an initial test in clinical practice the article presents three real-world examples of subtle MOIs, that could only be appreciated on the 4K-video, but not on the VGA-video, recorded as part of the clinical routine. In conclusion, a 4K-RGB recording, 3D cropping, and 3D video playing system was developed, implemented, and tested for realworld clinical scenarios, considering the specific requirements of clinical monitoring in EMUs. The new data acquisition setup can support clinical diagnosis, which may lead to new insights in the field of epilepsy and the development of AI approaches in the future.

2025

Exploring image and skeleton-based action recognition approaches for clinical in-bed classification of simulated epileptic seizure movements

Authors
Karácsony, T; Fearns, N; Birk, D; Trapp, SD; Ernst, K; Vollmar, C; Rémi, J; Jeni, LA; De la Torre, F; Cunha, JPS;

Publication
EXPERT SYSTEMS WITH APPLICATIONS

Abstract
Epileptic seizure classification based on seizure semiology requires automated, quantitative approaches to support the diagnosis of epilepsy, which affects 1 % of the world's population. Current approaches address the problem on a seizure level, neglecting the detailed evaluation of the classification of the underlying action features, also known as Movements of Interest (MOIs), which are critical for epileptologists in determining their classifications. Moreover, it hinders objective comparison of these approaches and attribution of performance differences due to datasets, intra-dataset MOI distribution, or architecture variations. Objective evaluation of action recognition techniques is crucial, with MOIs serving as foundational elements of semiology for clinical in-bed applications to facilitate epileptic seizure classification. However, until now, there were no MOI datasets available nor benchmarks comparing different action recognition approaches for this clinical problem. Therefore, as a pilot, we introduced a novel, simulated seizure semiology dataset carried out by 8 experienced epileptologists in an EMU bed, consisting of 7 MOI classes. We compare several computer vision methods for MOI classification, two image-based (I3D and Uniformerv2), and two skeleton-based (ST-GCN++ and PoseC3D) action recognition approaches. This study emphasizes the advantages of a 2-stage skeleton-based action recognition approach in a transfer learning setting (4 classes) and the multi-scale challenge of MOI classification (7 classes), advocating for the integration of skeleton-based methods with hand gesture recognition technologies in the future. The study's controlled MOI simulation dataset provides us with the opportunity to advance the development of automated epileptic seizure classification systems, paving the way for enhancing their performance and having the potential to contribute to improved patient care.

2026

Video-based epileptic seizure classification: A novel multi-stage approach integrating vision and motion transformer deep learning models

Authors
Aslani, R; Karácsony, T; Fearns, N; Caldeiras, C; Vollmar, C; Rego, R; Rémi, J; Noachtar, S; Cunha, JPS;

Publication
BIOMEDICAL SIGNAL PROCESSING AND CONTROL

Abstract
Automated seizure quantification and classification are needed for semiology-based epileptic seizure diagnosis support. To the best of our knowledge, the 5-class (Hypermotor, Automotor, Complex Motor, Psychogenic Non-Epileptic Seizures, and Generalized Tonic-Clonic Seizures) seizure video dataset (198 seizures from 74 patients) studied in this paper is the largest 5-class dataset ever curated, composed of monocular RGB videos from two university hospital epilepsy monitoring units. 2D skeletons were estimated using ViTPose, a vision transformer deep learning (DL) architecture, and lifted to 3D space using MotionBERT, a multimodal motion transformer architecture. The movements were quantified based on the estimated 3D skeleton sequences. Two approaches were evaluated for seizure classification: (1) classical machine learning methods (Random Forest (RF) and XGBoost) applied to quantified movement parameters, and (2) 2D skeleton-based DL using MotionBERT action, an action recognition DL model, to which we perform transfer-learning. The best model achieved a promising, above literature, 5-fold cross-validated macro average F1-score of 0.84 +/- 0.09 (RF) for 5-class classification. The binary case (Automotor vs Hypermotor) resulted in 0.80 +/- 0.18 (MotionBERT action), and adding a 3rd class (Complex motor) lowered to 0.65 +/- 0.14 (RF). This novel multi-stage classification ensures that the included movement features are traceable, allowing interpretable AI exploration of this novel approach supporting future clinical diagnosis.

  • 2
  • 2