The information is prepared on the basis of data from the information-analytical system RSF, informative part is represented in the author's edition. All rights belong to the authors, the use or reprinting of materials is permitted only with the prior consent of the authors.

Project titleBionic methods of sensorics and ambient intelligence for Internet-based monitoring systems of human resilience in the conditions of Northern Territories

Research area 01 - MATHEMATICS, INFORMATICS, AND SYSTEM SCIENCES, 01-521 - Intellectual Internet-technologies

Annotation
In this project, it is proposed to carry out a study of the fundamental scientific problem of developing new bionic methods of sensorics and ambient intelligence. The methods provide for the development of Internet systems for monitoring human resilience in the conditions of the northern territories. A conceptual model of a bionic suit is being developed that provides constant monitoring of a person in everyday conditions. Monitoring is aimed at analyzing the properties of human resilience based on recognition technologies in the systems of the surrounding intelligence. The developed Internet-based health monitoring systems are designed to build various information services-assistants for: a) medical professionals to help with working with patients, b) patients for motivated participation in maintaining their own health, c) organizations engaged in analyzing the health of the population to obtain analytical information. The properties of human resilience are adaptation to stress, dysfunction, diseases, difficult living conditions and other negative effects on (mental) health. Ambient (artificial) intelligence technologies are based on peripheral computing technologies of the Internet of Things, the construction of personal profiles of interaction objects, contextual orientation and foresight in the construction of services, a variety of options for human-machine interaction with a variety of surrounding computing devices and Internet resources, the possibilities of the tactile Internet for the transmission and perception of human information (including on the basis of bionic principles). Elements of trusted AI include: a) justification of the analytical information provided to medical professionals, b) personalized justification and motivation for the implementation of recommendations for the patient, c) assessment of the quality of analytical information for the expert to perform an analysis of the health status of the population. Elements of a "strong" AI include: a) actions in conditions of uncertainty in monitoring data, b) training based on the results of building information assistant services, c) human-machine interaction in natural language, d) the ability to "feel" the object of health monitoring. The object of monitoring is a process in which the autonomous, cognitive and/or motor functions of a person are involved. Data about the monitoring object: health parameters (operational information, accumulated clinical information), parameters of the physical environment (location, climate, surrounding objects, events occurring nearby), parameters of autonomous, cognitive and/or motor function performed by a person (EMC, interviews, audio, inertial data, video, text, expert assessments, reference books, social media). Not only medical personnel are involved in decision-making, but also the person being monitored. The next stack of computing levels in the tactile Internet is being investigated. - Implementation of sensor methods on peripheral IoT devices, taking into account: a) the low productivity and miniaturization of such devices (the problem of ubiquitous computing); b) the variety of possible data for measurement (the problem of multiparametric monitoring for objects of physical, informational and social human environments). - Implementation of protocols for information-controlled interaction of peripheral IoT devices (i.e. directly at the data collection site), taking into account the need for joint data processing with fast feedback (the problem of distribution of computing load and available resources in the network). - Implementation of event recognition algorithms on peripheral IoT devices, taking into account the need to process streaming measurement data online using neural networks (the problem of building online assistant services for humans). The application of the tactile Internet requires the implementation of calculations at each of these levels. The key problem is the balance between the computing power of the peripheral device and the complexity of the software being executed.

Expected results
А. Решения для применения бионических методов сенсорики и окружающего интеллекта на основе технологий виртуальной и дополненной реальности. - новые виды ПО для предобработки «быстрых» потоковых сенсорных данных с управлением баланса «точность/трудоемкость»; - адаптивные стратегии доступа к приоритетной сенсорной информации из множества источников для организации быстрой обратной связи в условиях северных территорий; - концептуальная модель бионического костюма с применением технологий виртуальной и дополненной реальности для организации человеко-машинного взаимодействия в тактильном интернете. Б. Решения для формирования бионических методов сенсорики и окружающего интеллекта: - Интеллектуализация объекта мониторинга здоровья – способность «рассказать» о себе, в т.ч. персонализация; - Интеллектуализация доступа к информации – способность дистанционного пользователя взаимодействовать онлайн с объектом мониторинга здоровья, в т.ч. на основе бионических методов; - Рекомендательная система для поддержки обоснования и принятия человеком решений по управлению объектом мониторинга, в т.ч. анализируя параметры здоровья (оперативная информация, накопленная клиническая информация), параметры физической среды (местоположение, климат, окружающие объекты, происходящие рядом события), параметры выполняемых человеком автономной, когнитивной и/или двигательной функции (ЭМК, интервью, аудио, инерциальные данные, видео, текст, экспертные оценки, справочники, социальные медиа). В. Прототип персонализированного ассистента человека по сопротивляемости ментального здоровья к стрессам и другим негативным внешним факторам, адаптация к условиям работы и проживания, сопротивляемость к усталости, мотивация к сохранению здоровья в условиях северных территорий. В целом, результаты выполнения проекта приведут к появлению новых научных публикаций в высокорейтинговых российских и международных изданиях в области сопряжения искусственного интеллекта, периферийных вычислений и «умной» бионической микроэлектроники, к формированию отечественной интеллектуальной собственности в виде патентов на изобретения и полезные модели, а также свидетельств о регистрации программ для ЭВМ. На их основе становится возможным формирование передовых российских бионических технологий сенсорики и окружающего интеллекта, что способствует реализации стратегических направлений развития РФ по цифровой экономике, цифровому обществу и искусственному интеллекту. Пилотные внедрения таких технологий могут выполняться на территории Республики Карелия, как референтной северной территории и площадки для исследования.

Annotation of the results obtained in 2022
An exploratory scientific direction of research "intelligent sensorics" has been formed, which assumes that sensorics methods implement not only data collection and measurement functions, but also the following "smart" functions: a) Pattern recognition in sensory data (based on AI and peripheral Internet computing technologies). b) Networking with other participants of the Internet system (based on mobile Internet technologies, data center technologies). c) Virtualization of the object of sensory system observation (based on digital twin technologies). d) Evaluation of the quality of the sensory data obtained - (self) diagnosis, reliability (based on AI technologies and peripheral Internet computing technologies). As part of such research, weekly scientific seminars are organized at the Artificial Intelligence Center at Petrozavodsk State University: - Intelligent Sensorics and Video Analytics in the Internet of Things, https://ai.petrsu.ru/events/iot - Human-like motion analysis tasks and robotic Internet systems, https://ai.petrsu.ru/events/robots Intelligent sensorics tracks three human functions: motor function, cognitive function, and autonomous function. Motor function requires tracking and recognizing human movement and the physical objects around it. Cognitive function requires tracking how a person thinks and reasoning and what (emotional) state they are in. Autonomic function requires tracking how the human body works. The following application areas for the development and application of bionic sensory and ambient intelligence methods in Internet-based human resilience monitoring systems in northern conditions are highlighted. Tracking of human movement, including in relation to other objects of the physical environment (motion sensorics). Related to the work of the human motor function. 2. Development of digital assistants, which accompany a person in everyday life (mobile sensorics). It is connected with work of motor, cognitive and autonomous functions of the person. 3. Robotic sensors providing remote amplification of human senses - remote equipment operator or monitoring of remote objects (mobile robotic sensorics). Related to human motor, cognitive, and autonomous functions. 4. Diagnostics of human health condition (body/organism sensorics). Related to the work of human autonomous function. 5. Tactile Internet technologies based on strain sensors (tactile sensorics). It is connected with the work of human motor and cognitive functions. For each application area, models and algorithms necessary for the implementation of hardware-software prototypes of a bionic suit have been developed.

Publications

1. Alexander Meigal, Liudmila Gerasimova-Meigal, Grigorij Rego, Dmitry Korzun Motor Activity Sensorics for mHealth Support of Human Resilience in Daily Life Conference of Open Innovations Association, FRUCT, Volume 32, Issue 1, pp. 169-177 (year - 2022) https://doi.org/10.23919/FRUCT56874.2022.9953829

2. Andrei Velichko, Mehmet Tahir Huyut, Maksim Belyaev, Yuriy Izotov, Dmitry Korzun Machine learning sensors for diagnosis of covid-19 disease using routine blood values for internet of things application Sensors, Volume 22, pp.1-29 (year - 2022) https://doi.org/10.3390/s22207886

3. Nikita Bazhenov, Egor Rybin, Dmitry Korzun An Event-Driven Approach to the Recognition Problem in Video Surveillance System Development Conference of Open Innovations Association, FRUCT, Volume 32, Issue 1, pp.65-74 (year - 2022) https://doi.org/10.23919/FRUCT56874.2022.9953883

4. - В России создали алгоритм, выявляющий ковид по двум показателям крови ТАСС. Наука, 26.10.2022 (year - )

5. - ПетрГУ на международной конференции UBICOMM 2022 Пресс-служба ПетрГУ, 25.11.2022 (year - )

6. - Проблемы современных информационно-вычислительных систем Пресс-служба ПетрГУ, 07.12.2022 (year - )

Annotation of the results obtained in 2023
The property of human resilience determines the object of constant monitoring – the observation of the process in which the autonomous, cognitive or motor functions of a person are involved. Sensory data about the monitoring object: health parameters (operational information, accumulated clinical information), parameters of the physical environment (location, climate, surrounding objects, events occurring nearby), parameters of autonomous, cognitive and/or motor function performed by a person (EMC, interviews, audio, inertial data, video, text, expert assessments, reference books, social media). The concept of "human intellectual sensorics" is defined based on the expansion of data collection and measurement functions, including for the conditions of the northern territories. Conceptual models and algorithms have been developed for the construction of a bionic suit that ensure constant monitoring of a person in everyday conditions. The bionic suit implements effective human interaction with objects of resilience properties. A digital assistant is being implemented that provides the following types of support for human resilience: a) mental health resistance to stress and other negative external factors; b) adaptation to work and living conditions; c) fatigue resistance; d) motivation to preserve health in the conditions of the northern territories. The concept of "human sensory" involves the use of a complex set of sensors and a variety of sensory systems to track: a) motor function (how a person moves), b) cognitive function (how information is perceived) and c) autonomous function (how the body works) of a person, as well as the situation around a person. Resilience is closely related to the work of these human functions, including to maintain human resilience, it is necessary to give him the opportunity to observe and analyze the work of these functions. The "intelligence" of sensorics manifests itself in the form of the following. a) A trainable sensor. Recognition algorithms (AI) work near the data collection site, i.e. on the sensor itself or a computing device nearby. Corresponds to the approach of peripheral analytics (edge computing). b) Self-tracking (active monitoring of observations). Assessment of the quality of sensory data. The reaction in case of detection of a decrease in quality (the so-called self-* properties of the sensor). c) Mobility of communications (mobile sensors). The sensor determines when, to whom and what data to transfer next. Including, taking into account the available resources of the mobile network (transition to 5G/6G technologies). d) Remote autonomy (robotic sensors). The sensor can be remote from the person himself, able to control monitoring without human intervention (sensor robotization). e) Human-machine interaction (digital expansion of human senses). Personalized delivery of information to a person, including the use of augmented and virtual reality technologies, bionic suit technologies. The following conditions of the northern territories affecting the monitoring of human resilience are highlighted. a) Sensors of the Internet monitoring system should be able to work in harsh environmental conditions (climate, snow, mud, movement, etc.). b) The proposed Internet monitoring systems should focus on the existing production in the territory of the Republic of Kazakhstan, contributing to the digital transformation of production processes. In particular, digital services are implemented in the form of assistants for employees. The employee must be able to work outdoors or in the workshop. c) A possible target group is the elderly. Many of them are single. Low population density and remoteness of the place of residence from large settlements or from aid centers. In particular, digital services are implemented in the form of assistants for elderly single people, helping people in everyday life, in everyday tasks. d) A possible target group is people coming to the Republic of Kazakhstan from the southern territories. A special case is students coming to study at PetrSU and other educational institutions. It is necessary to monitor and strengthen resilience, helping to adapt to new living conditions. e) The territory of the Region is attractive for creating jobs for remote employees of various organizations of the Russian Federation. A possible target group of people to monitor human resilience are operators, dispatchers, IT specialists, and experts. As part of this research, weekly scientific seminars have been organized at the Center for Artificial Intelligence of Petrozavodsk State University: - Intelligent sensors and video analytics in the Internet of Things, https://ai.petrsu.ru/events/iot - Tasks of analyzing humanoid motion and robotic Internet systems, https://ai.petrsu.ru/events/robots The following prototypes for digital assistance have been developed and experimentally investigated. 1. A prototype of a digital assistant that provides tracking of human movement and the formation of a 3D model of movement. The work of the prototype is presented in the report and the article: - Assessment of Operation Quality for Robotic Manipulator in Real-Time / N. Bazhenov, G. Rego, E. Rybin, D. Korzun // Conference of Open Innovations Association, FRUCT. – 2023. – No. 34. – P. 200-203. https://elibrary.ru/item.asp?id=54884910 Sosnovsky, I. V. Visualization of 3D space in the form of a point cloud based on augmented reality technologies / I. V. Sosnovsky, T. S. Kiprushkina // Path to science: applied mathematics, informatics and information technologies : Abstracts of the All-Russian Youth Scientific and Practical Conference, Yaroslavl, April 17-21, 2023. Yaroslavl: Yaroslavl State University named after P.G. Demidov, 2023. – pp. 67-69. https://elibrary.ru/item.asp?id=53843935 2. A prototype of a digital assistant that tracks the results of a person's work when performing physical exercises, including on sports simulators. The work of the prototype is presented in the report and the article: - A Mobile Application for Assessing the Strength Exercises on Sports Training Equipment / K. Smirnov, V. Ermakov, E. Topchiy, D. Korzun // Conference of Open Innovations Association, FRUCT. – 2023. – No. 33. – P. 391-394. https://elibrary.ru/item.asp?id=53943950 - Real-Time Evaluation of Hands Position at Sport Training Machine / K. Smirnov, E. Topchiy, V. Ermakov, D. Korzun // Conference of Open Innovations Association, FRUCT. – 2023. – No. 34. – P. 278-281. https://elibrary.ru/item.asp?id=54884926 3. A prototype of a digital assistant that provides enhancement of human senses through robotic sensors with partially autonomous movement The work of the prototype is presented in the report and the article: Kostin D.A. SENSOR SYSTEM OF A MOBILE ROBOT FOR ORIENTATION, TERRAIN MAPPING AND OBSTACLE RECOGNITION / D.A. Kostin, D.S. Melnikov, D.A. Ustinov, S.D. Yaskelainen, V.A. Ermakov, D.J. Korzun // Proceedings of the XVII All-Russian scientific and practical conference "Digital technologies in education, science, society" / PetrSU Publishing House. - Petrozavodsk, 2023. - pp.55-57. - https://it2023.petrsu.ru /. 4. A prototype of a digital assistant that provides diagnostics of human health (body/body sensors). Parkinson's disease is used as an example of the disease. The work of the prototype is presented in M.Belyaev, M.Murugappan, A.Velichko, D.Korzun. Entropy-Based Machine Learning Model for Fast Diagnosis and Monitoring of Parkinson’s Disease. Sensors 2023, 23, 8609. https://doi.org/10.3390/s23208609. 5. A prototype of a digital assistant that provides monitoring of the territory for human movement and human use of technological equipment. The work of the prototype is presented in the report and the article: - Evaluation of the Human Use for Sport Training Equipment based on Multicamera Video Surveillance / N. Bazhenov, E. Rybin, S. Zavyalov, D. Korzun // Conference of Open Innovations Association, FRUCT. – 2023. – No. 33. – P. 342-345. https://elibrary.ru/item.asp?id=53943941

Publications

1. Belyaev M., Murugappan M., Velichko A., Korzun D. Entropy-Based Machine Learning Model for Fast Diagnosis and Monitoring of Parkinson’s Disease Sensors MDPI, Vol.23, pp.8609 (year - 2023) https://doi.org/10.3390/s23208609

2. Korzun D., Bogoiavlenskaia O., Kulakov K. Применение алгоритма случайной отсрочки при активном управлении обменом информации в интернет-среде Программная инженерия, Т.14, №.5. С.207-216 (year - 2023) https://doi.org/10.17587/prin.14.207-216

3. Meigal A., Gerasimova-Meigal L., Korzun D. A Concept Model of mHealth Sensorics for Digital Assistance of Human Cognitive Resilience 4th Conference of Open Innovations Association (FRUCT), Vol.34, pp.100-107 (year - 2023) https://doi.org/-

4. Rego G.E. Mathematical Modeling Method for Detecting the Fuzzy Occurrence of Dangerus Events Программная инженерия, Vol. 14, No. 9. pp. 442-451 (year - 2023) https://doi.org/10.17587/prin.14.442-451

5. Velichko A., Korzun D., Meigal A. Artificial Neural Networks for IoT-Enabled Smart Applications Basel: MDPI, This book is a reprint of the Special Issue Artificial Neural Networks for IoT-Enabled Smart Applications that was published in Sensors (year - 2023) https://doi.org/10.3390/books978-3-0365-8429-4

6. Velichko A. Korzun D. Meigal A. Artificial Neural Networks for IoT-Enabled Smart Applications: Recent Trends Sensors MDPI, Editorial. Vol.23, pp.4853 (year - 2023) https://doi.org/10.3390/s23104853

7. - Искусственный интеллект поможет жителям Карелии сохранять здоровье Пресс-служба Фонда венчурных инвестиций Республики Карелия, Опубликовано 20 апреля 2023 г. (year - )

8. - Сенсорное поле: болезнь Паркинсона выявят с помощью домашней техники Известия, Мария Недюк. 21 ноября 2023, 10:00 (year - )