Izvestiya of Saratov University.

Physics

ISSN 1817-3020 (Print)
ISSN 2542-193X (Online)

For citation:

Shusharina N. N. Methodology of collection, recording and markup of biophysical multimodal data in the study of human psychoemotional states. Izvestiya of Saratov University. Physics , 2024, vol. 24, iss. 3, pp. 239-249. DOI: 10.18500/1817-3020-2024-24-3-239-249, EDN: MBIZDN

This is an open access article distributed under the terms of Creative Commons Attribution 4.0 International License (CC-BY 4.0).
Published online: 
30.08.2024
Full text:
download
(downloads: 229)
Language: 
Russian
Heading: 
Biophysics and medical physics
Article type: 
Article
UDC: 
004.78
DOI: 
10.18500/1817-3020-2024-24-3-239-249
EDN: 
MBIZDN

Methodology of collection, recording and markup of biophysical multimodal data in the study of human psychoemotional states

Autors: 
Shusharina Natalia Nikolaevna, Immanuel Kant Baltic Federal University
Abstract: 

Background and Objectives: Effective diagnostics of depression state using instrumental methods of biopotentials measurement is promising both as a tool for increasing the efficiency of work of a diagnostician and for application in automated hardware and software therapy complexes, including those with biofeedback, allowing to create neuroadaptive systems for correction of psychoemotional and cognitive problems of patients. The purpose of this paper is to analyze the requirements to the methodology of biophysical data collection, hardware and software for their primary processing on the basis of open datasets of psychoemotional state determination, to formulate a methodology for the formation of multimodal datasets for the study of mental states, their changes, suitable for use in machine learning algorithms, to describe a possible method of realization of these requirements in hardware and software complexes. Materials and Methods: Open datasets of depressed patients were selected to analyze the main characteristics of datasets characterizing mental states. To formulate the main requirements, publications on the features of multimodal data application for the diagnosis of depressive disorders were reviewed. Results: The result of the work are the requirements to multimodal biopotential data for the study of psychoemotional states, the methodology and functional concept of hardware-software complex for their registration, the synchronization and recording in annotated form. Conclusion: The example of depressive disorder shows the usefulness and possibility of recording multimodal, synchronized annotated data on the psycho-emotional state of the subject for research, diagnostic purposes and application as a training sample in machine learning algorithms. The proposed methodology and the concept of the hardware-software complex allow to level out the main disadvantages of multimodal systems of biopotentials registration realized in the form of separate blocks and to supplement the instrumental data with annotation by state labels and time labels.

Key words: 
biophysics
biomedical signals
non-invasive electroencephalogram
depression diagnosis
multimodal data
synchronization of multichannel physiological data
Acknowledgments: 
This work was performed in the framework of the State Assignment (project No. FZWM-2024-0013).
Reference: 
  1. Korotkov K. G. Electrophotonic Analysis of Complex Parameters of the Environment and Psycho-Emotional State of a Person. Wise Journal, 2015, vol. 4, no. 3, pp. 49–56.
  2. Astramskaitл I., Juodžbalys G. Scales used to rate adult patients’ psycho-emotional status in tooth extraction procedures: A systematic review. International Journal of Oral and Maxillofacial Surgery, 2017, vol. 46, no. 7, pp. 886–898. https://doi.org/10.1016/j.ijom.2017.03.015
  3. Nedelsky N. B., Taylor J. P. Bridging biophysics and neurology: Aberrant phase transitions in neurodegenerative disease. Nature Reviews Neurology, 2019, vol. 15, no. 5, pp. 272–286. https://doi.org/10.1038/s41582-019-0157-5
  4. Stoyanov D., Khorev V., Paunova R., Kandilarova S., Simeonova D., Badarin A., Hramov A., Kurkin S. Resting-state functional connectivity impairment in patients with major depressive episode. International Journal of Environmental Research and Public Health, 2022, vol. 19, no. 21, pp. 14045. https://doi.org/10.3390/ijerph192114045
  5. Hramov A. E., Frolov N. S., Maksimenko S. A., Kurkin S. A., Kazancev V. B., Pisarchik A. N. Functional brain networks: From reconnection to dynamic integration. Physics-Uspehi, 2021, vol. 64, iss. 6, pp. 584–616. https://doi.org/10.3367/UFNr.2020.06.038807
  6. Karel G. M. M., Patrick R., Yvonne V., Diederick E. G., Douglas G. A. Prognosis and prognostic research: What, why, and how? BMJ, 2009, vol. 338, article no. b375. https://doi.org/10.1136/bmj.b375
  7. Karpov O. E., Pitsik E. N., Kurkin S. A., Maksimenko V. A., Gusev A. V., Shusharina N. N., Hramov A. E. Analysis of publication activity and research trends in the field of ai medical applications: Network approach. International Journal of Environmental Research and Public Health, 2023, vol. 20, no. 7, pp. 5335. https://doi.org/10.3390/ijerph20075335
  8. Ristevski B., Chen M. Big data analytics in medicine and healthcare. J. of Integrative Bioinformatics, 2018, vol. 15, no. 3, pp. 20170030. https://doi.org/10.1515/jib-2017-0030
  9. Gudayol-Ferré E., Peró-Cebollero M., González-Garrido A. A., Guàrdia-Olmos J. Changes in brain connectivity related to the treatment of depression measured through fMRI: A systematic review. Frontiers in Human Neuroscience, 2015, vol. 9, pp. 582. https://doi.org/10.3389/fnhum.2015.00582
  10. Pitsik E. N., Maximenko V. A., Kurkin S. A., Sergeev A. P., Stoyanov D., Paunova R., Kandilarova S., Simeonova D., Hramov A. E. The topology of fMRI-based networks defines the performance of a graph neural network for the classification of patients with major depressive disorder. Chaos, Solitons & Fractals, 2023, vol. 167, pp. 113041. https://doi.org/10.1016/j.chaos.2022.113041
  11. Pisarchik A. N., Andreev A. V., Kurkin S. A., Stoyanov D., Badarin A. A., Paunova R., Hramov A. E. Topology switching during window thresholding fmribased functional networks of patients with major depressive disorder: Consensus network approach. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2023, vol. 33, no. 9, pp. 093122. https://doi.org/10.1063/5.0166148
  12. Sawarkar I., Sardesai A., Vachhani A., Samdani K. A Review on Psychophysiological monitoring for depression detection with therapeutic cure using AI enabled smart devices. 2019 IEEE 5th International Conference for Convergence in Technology (I2CT). IEEE, 2019, pp. 1–8. https://doi.org/10.1109/I2CT45611.2019.9033839
  13. Maksimenko V. A., Hramov A. E., Grubov V. V., Nedaivozov V. O., Makarov V. V., Pisarchik A. N. Nonlinear effect of biological feedback on brain attentional state. Nonlinear Dynamics, 2019, vol. 95, no. 3, pp. 1923–1939. https://doi.org/10.1007/s11071-018-4668-1
  14. Baghaei N., Chitale V., Hlasnik A., Stemmet L., Liang H. N., Porter R. Virtual reality for supporting the treatment of depression and anxiety: Scoping review. JMIR Mental Health, 2021, vol. 8, no. 9, pp. e29681. https://doi.org/10.2196/29681
  15. Zander T. O., Kothe C. Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general. Journal of Neural Engineering, 2011, vol. 8, no. 2, pp. 025005. https://doi.org/10.1088/1741-2560/8/2/025005
  16. Rahaman M. A., Chen J., Fu Z., Lewis N., Iraji A., Calhoun V. D. Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness. 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2021, pp. 3267–3272. https://doi.org/10.1109/EMBC46164.2021.9630693
  17. Chen W., Sá R. C., Bai Y., Napel S., Gevaert O., Lauderdale D. S., Giger M. L. Machine learning with multimodal data for COVID-19. Heliyon, 2023, vol. 9, pp. e17934. https://doi.org/10.1016/j.heliyon.2023.e17934
  18. Borovkova E. I., Hramkov A. N., Dubinkina E. S., Ponomarenko V. I., Bezruchko B. P., Ishbulatov Y. M., Kurbako A. V., Karavaev A. S., Prokhorov M. D. Biomarkers of the psychophysiological state during the cognitive tasks estimated from the signals of the brain, cardiovascular and respiratory systems. The European Physical Journal Special Topics, 2023, vol. 232, no. 5, pp. 625–633. https://doi.org/10.1140/epjs/s11734-022-00734-z
  19. Hulsen T., Jamuar S. S., Moody A. R., Karnes J. H., Varga O., Hedensted S., Spreafico R., Hafler D. A., McKinney E. F. From big data to precision medicine. Frontiers in Medicine, 2019, vol. 6, pp. 34. https://doi.org/10.3389/fmed.2019.00034
  20. Oviatt S., Schuller B., Cohen P., Sonntag D., Potamianos G., Krüger A. The handbook of multimodalmultisensor interfaces. Volume 2: Signal processing, architectures, and detection of emotion and cognition. Morgan & Claypool, 2018. 2034 p. https://doi.org/10.1145/3107990
  21. Cimtay Y., Ekmekcioglu E., Caglar-Ozhan S. Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 2020, vol. 8, pp. 168865–168878. https://doi.org/10.1109/ACCESS.2020.3023871
  22. McDuff D., Rowan K., Choudhury P., Wolk J., Pham T., Czerwinski M. A Multimodal Emotion Sensing Platform for Building Emotion-Aware Applications. arXiv preprint arXiv:1903.12133, 2019. https://doi.org/10.48550/arXiv.1903.12133
  23. Qin S., Hermans E. J., van Marle H. J., Luo J., Fernández G. Acute psychological stress reduces working memory-related activity in the dorsolateral prefrontal cortex. Biological Psychiatry, 2009, vol. 66, no. 1, pp. 25–32. https://doi.org/10.1016/j.biopsych.2009.03.006
  24. Peng H., Hu B., Zheng F., Fan D., Zhao W., Chen X., Cai Q. A method of identifying chronic stress by EEG. Personal and Ubiquitous Computing, 2013, vol. 17, no. 7, pp. 1341–1347. https://doi.org/10.1007/s00779- 012-0593-3
  25. Acharya U. R., Oh S. L., Hagiwara Y., Tan J. H., Adeli H., Subha D. P. Automated EEG-based screening of depression using deep convolutional neural network. Computer Methods and Programs in Biomedicine, 2018, vol. 161, pp. 103–113. https://doi.org/10.1016/j.cmpb.2018.04.012
  26. Liu M., Zhou L., Wang X., Jiang Y., Liu Q. Deficient manipulation of working memory in remitted depressed individuals: Behavioral and electrophysiological evidence. Clinical Neurophysiology, 2017, vol. 128, no. 7, pp. 1206–1213. https://doi.org/10.1016/j.clinph.2017.04.011
  27. Honke G., Higgins I., Thigpen N., Miskovic V., Link K., Gupta P., Hajcak G. Representation learning for improved interpretability and classification accuracy of clinical factors from EEG. arXiv preprint arXiv:2010.15274, 2020. https://doi.org/10.48550/arXiv.2010.15274
  28. Valenza G., Nardelli M., Lanata A., Gentili C., Bertschy G., Paradiso R., Scilingo E. P. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate variability analysis. IEEE Journal of Biomedical and Health Informatics, 2013, vol. 18, no. 5, pp. 1625–1635. https://doi.org/10.1109/JBHI.2013.2290382
  29. Healey J. A., Picard R. W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 2005, vol. 6, no. 2, pp. 156–166. https://doi.org/10.1109/TITS.2005.848368
  30. Wijsman J., Grundlehner B., Penders J., Hermens H. Trapezius muscle EMG as predictor of mental stress. ACM Transactions on Embedded Computing Systems (TECS), 2013, vol. 12, no. 4, pp. 1–20. https://doi.org/10.1145/2485984.2485987
  31. Mavridou I., Seiss E., Hamedi M., Balaguer-Ballester E., Nduka C. Towards valence detection from EMG for Virtual Reality applications. 12th International Conference on Disability, Virtual Reality and Associated Technologies (ICDVRAT 2018), 4–6 September 2018, Nottingnam, UK. 4 p.
  32. Carvalho N., Laurent E., Noiret N., Chopard G., Haffen E., Bennabi D., Vandel P. Eye movement in unipolar and bipolar depression: A systematic review of the literature. Frontiers in Psychology, 2015, vol. 6, pp. 1809. https://doi.org/10.3389/fpsyg.2015.01809
  33. Sharma M., Kacker S., Sharma M. A brief introduction and review on galvanic skin response. International Journal of Medical Research Professionals, 2016, vol. 2, pp. 13–17. https://doi.org/10.21276/ijmrp.2016.2.6.003
  34. Sarchiapone M., Gramaglia C., Iosue M., Carli V., Mandelli L., Serretti A., Zeppegno P. The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis. BMC Psychiatry, 2018, vol. 18, no. 1, pp. 22. https://doi.org/10.1186/s12888-017-1551-4
  35. Greco A., Valenza G., Scilingo E. P. Advances in electrodermal activity processing with applications for mental health. Cham, Berlin, Springer, 2016. 138 p. https://doi.org/10.1007/978-3-319-46705-4
  36. Papousek I., Schulter G. Associations between EEG asymmetries and electrodermal lability in low vs. high depressive and anxious normal individuals. International Journal of Psychophysiology, 2001, vol. 41, no. 2, pp. 105–117. https://doi.org/10.1016/S0167-8760(01)00131-3
  37. Mokros A., Habermeyer E., Küchenhoff H. The uncertainty of psychological and psychiatric diagnoses. Psychological Assessment, 2018, vol. 30, no. 4, pp. 556. https://doi.org/10.1037/pas0000524
  38. Corneanu C. A., Simón M. O., Cohn J. F., Guerrero S. E. Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, vol. 38, no. 8, pp. 1548–1568. https://doi.org/10.1109/TPAMI.2016.2515606
  39. Poria S., Cambria E., Bajpai R., Hussain A. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 2017, vol. 37, pp. 98–125. https://doi.org/10.1016/j.inffus.2017.02.003
  40. Zhao S., Wang S., Soleymani M., Joshi D., Ji Q. Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2019, vol. 15, no. 3s, pp. 1–32. https://doi.org/10.1145/3363560
  41. Castanedo F. A review of data fusion techniques. The Scientific World Journal, 2013, vol. 2013, article no. 704504. https://doi.org/10.1155/2013/704504
  42. Raman C., Tan S., Hung H. A Modular Approach for Synchronized Wireless Multimodal Multisensor Data Acquisition in Highly Dynamic Social Settings. Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3586–3594. https://doi.org/10.1145/3394171.3413697
  43. Influxdata. Available at: https://www.influxdata.com/ (accessed January 20, 2024).
  44. Mumtaz W. MDD Patients and Healthy Controls EEG Data (New). Available at: https://doi.org/10.6084/m9.figshare.4244171.v2 (accessed January 20, 2024).
  45. Cavanagh J. EEG: Depression rest. Available at: https://doi.org/10.18112/openneuro.ds003478.v1.1.0 (accessed January 20, 2024).
  46. Cai H., Gao Y., Sun S., Li N., Tian F., Xiao H., Hu B. MODMA dataset: A Multi-modal Open Dataset for Mental-disorder Analysis. arXiv preprint arXiv:2002.09283, 2020. https://doi.org/10.48550/arXiv.2002.09283
  47. Van Dijk H., Van Wingen G., Denys D., Olbrich S., Van Ruth R., Arns M. The two decades brainclinics research archive for insights in neurophysiology (TDBRAIN) database. Scientific Data, 2022, vol. 9, no. 1, pp. 333. https://doi.org/10.1038/s41597-022-01409-z
  48. Maaten L. Visualizing datausing t-SNE. Journal of Machine Learning Research, 2008, vol. 9, pp. 2579–2605.
Received: 
14.02.2024
Accepted: 
21.05.2024
Published: 
30.08.2024
  • 492 reads