Для цитирования:
Шушарина Н. Н. Методика сбора, записи и разметки биофизических мультимодальных данных при исследовании психоэмоциональных состояний человека // Известия Саратовского университета. Новая серия. Серия: Физика. 2024. Т. 24, вып. 3. С. 239-249. DOI: 10.18500/1817-3020-2024-24-3-239-249, EDN: MBIZDN
Методика сбора, записи и разметки биофизических мультимодальных данных при исследовании психоэмоциональных состояний человека
Цель настоящей работы – проанализировать требования к методике сбора биофизических данных на основе открытых наборов данных определения психоэмоционального состояния, аппаратному и программному обеспечению для их первичной обработки. Сформулировать методику формирования мультимодальных наборов данных, пригодную для исследования психических состояний и их изменений, в том числе с использованием алгоритмов машинного обучения. Описать возможный метод реализации этих требований в аппаратно-программных комплексах. Методы. Для анализа основных особенностей наборов данных, характеризующих психические состояния, были выбраны открытые наборы данных пациентов с депрессивными расстройствами. Основные требования были сформулированы на основе изучения публикаций об особенностях применения мультимодальных данных для диагностики депрессивных расстройств. Результатом работы являются набор требований к мультимодальным данным биопотенциалов для исследования психоэмоциональных состояний, методика и функциональная концепция аппаратно-программного комплекса для их регистрации, синхронизации и записи в аннотированном виде. Заключение. На примере депрессивного расстройства показана целесообразность и возможность регистрации мультимодальных, синхронизированных между собой аннотированных данных о психоэмоциональном состоянии испытуемого для исследовательских, диагностических целей и применения в качестве обучающей выборки в алгоритмах машинного обучения. Предложенная методика и концепция программно-аппаратного комплекса позволяют нивелировать основные недостатки мультимодальных систем регистрации биопотенциалов, реализованных в виде отдельных блоков и дополнить инструментальные данные аннотированием метками состояний и времени.
- Korotkov K. G. Electrophotonic Analysis of Complex Parameters of the Environment and Psycho-Emotional State of a Person. Wise Journal, 2015, vol. 4, no. 3, pp. 49–56.
- Astramskaitл I., Juodžbalys G. Scales used to rate adult patients’ psycho-emotional status in tooth extraction procedures: A systematic review. International Journal of Oral and Maxillofacial Surgery, 2017, vol. 46, no. 7, pp. 886–898. https://doi.org/10.1016/j.ijom.2017.03.015
- Nedelsky N. B., Taylor J. P. Bridging biophysics and neurology: Aberrant phase transitions in neurodegenerative disease. Nature Reviews Neurology, 2019, vol. 15, no. 5, pp. 272–286. https://doi.org/10.1038/s41582-019-0157-5
- Stoyanov D., Khorev V., Paunova R., Kandilarova S., Simeonova D., Badarin A., Hramov A., Kurkin S. Resting-state functional connectivity impairment in patients with major depressive episode. International Journal of Environmental Research and Public Health, 2022, vol. 19, no. 21, pp. 14045. https://doi.org/10.3390/ijerph192114045
- Hramov A. E., Frolov N. S., Maksimenko S. A., Kurkin S. A., Kazancev V. B., Pisarchik A. N. Functional brain networks: From reconnection to dynamic integration. Physics-Uspehi, 2021, vol. 64, iss. 6, pp. 584–616. https://doi.org/10.3367/UFNr.2020.06.038807
- Karel G. M. M., Patrick R., Yvonne V., Diederick E. G., Douglas G. A. Prognosis and prognostic research: What, why, and how? BMJ, 2009, vol. 338, article no. b375. https://doi.org/10.1136/bmj.b375
- Karpov O. E., Pitsik E. N., Kurkin S. A., Maksimenko V. A., Gusev A. V., Shusharina N. N., Hramov A. E. Analysis of publication activity and research trends in the field of ai medical applications: Network approach. International Journal of Environmental Research and Public Health, 2023, vol. 20, no. 7, pp. 5335. https://doi.org/10.3390/ijerph20075335
- Ristevski B., Chen M. Big data analytics in medicine and healthcare. J. of Integrative Bioinformatics, 2018, vol. 15, no. 3, pp. 20170030. https://doi.org/10.1515/jib-2017-0030
- Gudayol-Ferré E., Peró-Cebollero M., González-Garrido A. A., Guàrdia-Olmos J. Changes in brain connectivity related to the treatment of depression measured through fMRI: A systematic review. Frontiers in Human Neuroscience, 2015, vol. 9, pp. 582. https://doi.org/10.3389/fnhum.2015.00582
- Pitsik E. N., Maximenko V. A., Kurkin S. A., Sergeev A. P., Stoyanov D., Paunova R., Kandilarova S., Simeonova D., Hramov A. E. The topology of fMRI-based networks defines the performance of a graph neural network for the classification of patients with major depressive disorder. Chaos, Solitons & Fractals, 2023, vol. 167, pp. 113041. https://doi.org/10.1016/j.chaos.2022.113041
- Pisarchik A. N., Andreev A. V., Kurkin S. A., Stoyanov D., Badarin A. A., Paunova R., Hramov A. E. Topology switching during window thresholding fmribased functional networks of patients with major depressive disorder: Consensus network approach. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2023, vol. 33, no. 9, pp. 093122. https://doi.org/10.1063/5.0166148
- Sawarkar I., Sardesai A., Vachhani A., Samdani K. A Review on Psychophysiological monitoring for depression detection with therapeutic cure using AI enabled smart devices. 2019 IEEE 5th International Conference for Convergence in Technology (I2CT). IEEE, 2019, pp. 1–8. https://doi.org/10.1109/I2CT45611.2019.9033839
- Maksimenko V. A., Hramov A. E., Grubov V. V., Nedaivozov V. O., Makarov V. V., Pisarchik A. N. Nonlinear effect of biological feedback on brain attentional state. Nonlinear Dynamics, 2019, vol. 95, no. 3, pp. 1923–1939. https://doi.org/10.1007/s11071-018-4668-1
- Baghaei N., Chitale V., Hlasnik A., Stemmet L., Liang H. N., Porter R. Virtual reality for supporting the treatment of depression and anxiety: Scoping review. JMIR Mental Health, 2021, vol. 8, no. 9, pp. e29681. https://doi.org/10.2196/29681
- Zander T. O., Kothe C. Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general. Journal of Neural Engineering, 2011, vol. 8, no. 2, pp. 025005. https://doi.org/10.1088/1741-2560/8/2/025005
- Rahaman M. A., Chen J., Fu Z., Lewis N., Iraji A., Calhoun V. D. Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness. 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 2021, pp. 3267–3272. https://doi.org/10.1109/EMBC46164.2021.9630693
- Chen W., Sá R. C., Bai Y., Napel S., Gevaert O., Lauderdale D. S., Giger M. L. Machine learning with multimodal data for COVID-19. Heliyon, 2023, vol. 9, pp. e17934. https://doi.org/10.1016/j.heliyon.2023.e17934
- Borovkova E. I., Hramkov A. N., Dubinkina E. S., Ponomarenko V. I., Bezruchko B. P., Ishbulatov Y. M., Kurbako A. V., Karavaev A. S., Prokhorov M. D. Biomarkers of the psychophysiological state during the cognitive tasks estimated from the signals of the brain, cardiovascular and respiratory systems. The European Physical Journal Special Topics, 2023, vol. 232, no. 5, pp. 625–633. https://doi.org/10.1140/epjs/s11734-022-00734-z
- Hulsen T., Jamuar S. S., Moody A. R., Karnes J. H., Varga O., Hedensted S., Spreafico R., Hafler D. A., McKinney E. F. From big data to precision medicine. Frontiers in Medicine, 2019, vol. 6, pp. 34. https://doi.org/10.3389/fmed.2019.00034
- Oviatt S., Schuller B., Cohen P., Sonntag D., Potamianos G., Krüger A. The handbook of multimodalmultisensor interfaces. Volume 2: Signal processing, architectures, and detection of emotion and cognition. Morgan & Claypool, 2018. 2034 p. https://doi.org/10.1145/3107990
- Cimtay Y., Ekmekcioglu E., Caglar-Ozhan S. Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 2020, vol. 8, pp. 168865–168878. https://doi.org/10.1109/ACCESS.2020.3023871
- McDuff D., Rowan K., Choudhury P., Wolk J., Pham T., Czerwinski M. A Multimodal Emotion Sensing Platform for Building Emotion-Aware Applications. arXiv preprint arXiv:1903.12133, 2019. https://doi.org/10.48550/arXiv.1903.12133
- Qin S., Hermans E. J., van Marle H. J., Luo J., Fernández G. Acute psychological stress reduces working memory-related activity in the dorsolateral prefrontal cortex. Biological Psychiatry, 2009, vol. 66, no. 1, pp. 25–32. https://doi.org/10.1016/j.biopsych.2009.03.006
- Peng H., Hu B., Zheng F., Fan D., Zhao W., Chen X., Cai Q. A method of identifying chronic stress by EEG. Personal and Ubiquitous Computing, 2013, vol. 17, no. 7, pp. 1341–1347. https://doi.org/10.1007/s00779- 012-0593-3
- Acharya U. R., Oh S. L., Hagiwara Y., Tan J. H., Adeli H., Subha D. P. Automated EEG-based screening of depression using deep convolutional neural network. Computer Methods and Programs in Biomedicine, 2018, vol. 161, pp. 103–113. https://doi.org/10.1016/j.cmpb.2018.04.012
- Liu M., Zhou L., Wang X., Jiang Y., Liu Q. Deficient manipulation of working memory in remitted depressed individuals: Behavioral and electrophysiological evidence. Clinical Neurophysiology, 2017, vol. 128, no. 7, pp. 1206–1213. https://doi.org/10.1016/j.clinph.2017.04.011
- Honke G., Higgins I., Thigpen N., Miskovic V., Link K., Gupta P., Hajcak G. Representation learning for improved interpretability and classification accuracy of clinical factors from EEG. arXiv preprint arXiv:2010.15274, 2020. https://doi.org/10.48550/arXiv.2010.15274
- Valenza G., Nardelli M., Lanata A., Gentili C., Bertschy G., Paradiso R., Scilingo E. P. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate variability analysis. IEEE Journal of Biomedical and Health Informatics, 2013, vol. 18, no. 5, pp. 1625–1635. https://doi.org/10.1109/JBHI.2013.2290382
- Healey J. A., Picard R. W. Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 2005, vol. 6, no. 2, pp. 156–166. https://doi.org/10.1109/TITS.2005.848368
- Wijsman J., Grundlehner B., Penders J., Hermens H. Trapezius muscle EMG as predictor of mental stress. ACM Transactions on Embedded Computing Systems (TECS), 2013, vol. 12, no. 4, pp. 1–20. https://doi.org/10.1145/2485984.2485987
- Mavridou I., Seiss E., Hamedi M., Balaguer-Ballester E., Nduka C. Towards valence detection from EMG for Virtual Reality applications. 12th International Conference on Disability, Virtual Reality and Associated Technologies (ICDVRAT 2018), 4–6 September 2018, Nottingnam, UK. 4 p.
- Carvalho N., Laurent E., Noiret N., Chopard G., Haffen E., Bennabi D., Vandel P. Eye movement in unipolar and bipolar depression: A systematic review of the literature. Frontiers in Psychology, 2015, vol. 6, pp. 1809. https://doi.org/10.3389/fpsyg.2015.01809
- Sharma M., Kacker S., Sharma M. A brief introduction and review on galvanic skin response. International Journal of Medical Research Professionals, 2016, vol. 2, pp. 13–17. https://doi.org/10.21276/ijmrp.2016.2.6.003
- Sarchiapone M., Gramaglia C., Iosue M., Carli V., Mandelli L., Serretti A., Zeppegno P. The association between electrodermal activity (EDA), depression and suicidal behaviour: A systematic review and narrative synthesis. BMC Psychiatry, 2018, vol. 18, no. 1, pp. 22. https://doi.org/10.1186/s12888-017-1551-4
- Greco A., Valenza G., Scilingo E. P. Advances in electrodermal activity processing with applications for mental health. Cham, Berlin, Springer, 2016. 138 p. https://doi.org/10.1007/978-3-319-46705-4
- Papousek I., Schulter G. Associations between EEG asymmetries and electrodermal lability in low vs. high depressive and anxious normal individuals. International Journal of Psychophysiology, 2001, vol. 41, no. 2, pp. 105–117. https://doi.org/10.1016/S0167-8760(01)00131-3
- Mokros A., Habermeyer E., Küchenhoff H. The uncertainty of psychological and psychiatric diagnoses. Psychological Assessment, 2018, vol. 30, no. 4, pp. 556. https://doi.org/10.1037/pas0000524
- Corneanu C. A., Simón M. O., Cohn J. F., Guerrero S. E. Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, vol. 38, no. 8, pp. 1548–1568. https://doi.org/10.1109/TPAMI.2016.2515606
- Poria S., Cambria E., Bajpai R., Hussain A. A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 2017, vol. 37, pp. 98–125. https://doi.org/10.1016/j.inffus.2017.02.003
- Zhao S., Wang S., Soleymani M., Joshi D., Ji Q. Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 2019, vol. 15, no. 3s, pp. 1–32. https://doi.org/10.1145/3363560
- Castanedo F. A review of data fusion techniques. The Scientific World Journal, 2013, vol. 2013, article no. 704504. https://doi.org/10.1155/2013/704504
- Raman C., Tan S., Hung H. A Modular Approach for Synchronized Wireless Multimodal Multisensor Data Acquisition in Highly Dynamic Social Settings. Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3586–3594. https://doi.org/10.1145/3394171.3413697
- Influxdata. Available at: https://www.influxdata.com/ (accessed January 20, 2024).
- Mumtaz W. MDD Patients and Healthy Controls EEG Data (New). Available at: https://doi.org/10.6084/m9.figshare.4244171.v2 (accessed January 20, 2024).
- Cavanagh J. EEG: Depression rest. Available at: https://doi.org/10.18112/openneuro.ds003478.v1.1.0 (accessed January 20, 2024).
- Cai H., Gao Y., Sun S., Li N., Tian F., Xiao H., Hu B. MODMA dataset: A Multi-modal Open Dataset for Mental-disorder Analysis. arXiv preprint arXiv:2002.09283, 2020. https://doi.org/10.48550/arXiv.2002.09283
- Van Dijk H., Van Wingen G., Denys D., Olbrich S., Van Ruth R., Arns M. The two decades brainclinics research archive for insights in neurophysiology (TDBRAIN) database. Scientific Data, 2022, vol. 9, no. 1, pp. 333. https://doi.org/10.1038/s41597-022-01409-z
- Maaten L. Visualizing datausing t-SNE. Journal of Machine Learning Research, 2008, vol. 9, pp. 2579–2605.
- 187 просмотров