Keywords: Human Factors, Human-Centered Transportation, Human Performance Modeling
Abstract: Traffic accidents caused by elderly drivers have become major social problems. We investigated central and autonomic nervous system activities at obstacle events while operating a driving simulator with either normal or simulated cataract vision. Driving with cataract vision acutely increased brain and sympathetic nervous system activities, along with attenuated parasympathetic nervous system activity, in some obstacle events under accumulated eyestrain, which may induce driver distraction. In future research, attention alert systems based on biological features may prove useful for safe driving.

Keywords: Brain-Computer Interfaces, Human-Machine Interface, Medical Informatics
Abstract: Individuals suffering from a progressive neurodegenerative disease, such as amyotrophic lateral sclerosis (ALS), will lose muscle function over time and become completely paralysed. For some time, people with ALS may retain functional tongue movement, despite losing mobility below the neck. These individuals can benefit from using an inductive tongue control interface (ITCI) to control computers or assistive robotic devices to gain independence in their daily lives. Eventually, when the individual can no longer use their tongue, they can rely on a brain computer interface (BCI). However, these require a lot of data to calibrate and function properly. Recording this data while the individual can still use the ITCI can potentially speed up the training process, allowing for an easier transition between interface technologies. This study investigates whether it is possible to create a background data collector for a BCI based on attempted tongue movement by analyzing brain patterns of two individuals with ALS while using an ITCI. The participants used an inductive ITCI in simple cued movement trials while electroencephalogram (EEG) was collected from the motor cortex. The EEG signal indicated that movement-related cortical potentials (MRCP) were generated after the cued movements. After synchronising the signal to the activations recorded on the ITCI, the MRCP became even more apparent. Therefore, it is concluded that it is possible to record MRCPs from individuals with ALS performing tongue movements, that the ITCI can assist in better extracting synchronized MRCP epochs, and that a background data collector for a tongue movement intention-based BCI is very feasible.

Keywords: Other Neurotechnology and Brain-Related Topics, Active BMIs
Abstract: Kinesthetic motor imagery (KMI) based brain-computer interfaces hold great potential for post-stroke motor rehabilitation. However, KMI is a complex task to perform, mainly due to the absence of sensory and kinesthetic feedback. To address this issue, sensory neurofeedback solutions have been proposed. One of them is tactile vibration, which has been used in BCI paradigms, but the optimal design choices of the vibration remain unclear. In this study, we present a novel bimodal stimulus that combines a vibrotactile device for the upper limb with a visual animation of a hand grasping a bottle. We aimed to investigate the effects of four different vibration patterns and four stimulation intensities to study the effects of the stimuli on the electroencephalography (EEG) signals of 17 healthy subjects without performing KMI. Our findings revealed EEG activity in the alpha, beta, and delta frequency bands within the somatosensory cortex during the sensory stimulation. While we did not observe significant differences in brain activity among the vibration patterns, we did observe statistically significant variations based on the vibration intensity. We observed central activity in both brain hemispheres, with stronger activity occurring in the contralateral hemisphere. Additionally, EEG activity seemed to involve the contralateral occipital areas. Based on these results, it is worth considering the delivery of feedback after participants have completed KMI tasks to differentiate and comprehensively study the brain activity resulting from the mental and sensory stimulation tasks.

Keywords: BMI Emerging Applications, Other Neurotechnology and Brain-Related Topics, Passive BMIs
Abstract: Decoding taste from brain activity is of paramount importance for understanding brain responses to gustatory stimuli with applications in the food industry, neuro-marketing, objective evaluation of consumer's preferences, and addiction prevention/treatment. However, progress in this field has been thwarted by challenging data acquisition and the multimodal aspects of gustatory events, which make traditional decoding based on event-related potentials (ERP) very difficult. Additionally, emerging approaches such as Deep Learning come short to give insights into decoding taste because their need of big amounts of data. In this paper, we propose a processing and analytical pipeline based on Multivariate Pattern Analysis (MPA) and Global Field Power (GFP) to analyze gustatory evoked potentials recorded with high-density scalp electroencephalography (EEG). Based on this analysis we show the evolution overtime of neural codes for identification of water-vs-wine and wine-vs-wine dyads.

Keywords: Virtual and Augmented Reality Systems, Virtual/Augmented/Mixed Reality, Information Visualization
Abstract: Hidden background cameras used to realize Diminished Reality (DR), which allows the observer to see through obstacles, are installed behind the obstacles. In order to realize DR, it is necessary to measure the positions of these cameras with respect to the observer's position, but this is difficult because the cameras cannot be seen directly from the observer's position. In this study, two calibration methods, the intersection method and the P4P method, which use the directions of radio waves from radio sources mounted on each camera to measure the positions of the hidden background cameras, are proposed. The intersection method measures the direction from several different points to the hidden background cameras and estimate the 3D position of the hidden background camera as the intersection of the measured directions. The P4P method measures the relative pose between multiple hidden background cameras in advance and uses it as a constraint to estimate the 3D poses of the hidden background cameras. The results showed that the P4P method was more accurate than the intersection method, and able to calibrate the camera with a position error of less than 100 mm.

Keywords: Medical Informatics, Human-Machine Cooperation and Systems, Information Visualization
Abstract: Computer Science is being increasingly used in medicine to improve quality of care and patient outcome. Clinical Guidance Systems (CGSs) codify clinical Best Practice Guide- lines (BPGs) and provide situation-specific advice to physicians. CGSs have shown effectiveness in reducing preventable medical errors during clinical evaluations. However, representing both the BPG and the associated physical processes in software is complex and tedious, making CGSs prone to bugs. This can be mitigated using a modular software architecture that allows encapsulated computational representation of physical processes, organized data management, fine-grained development and verification, in addition to improving comprehensibility and maintainability. This paper discusses the sources of complexity in CGSs, and propose a software architecture that consists of a patient digital twin, a compliance monitor, a Graphical User Interface (GUI) along with a communication middleware. We developed a CGS for Pediatric Sepsis Management co-designed by physicians using our approach. We use the Fluid Resuscitation therapy of this CGS as a case study to illustrate the development process.

Keywords: Virtual and Augmented Reality Systems, Information Visualization, Virtual/Augmented/Mixed Reality
Abstract: Free viewpoint video technology, which provides a high sense of presence and immersion by allowing users to view live video from any viewpoint, is increasingly demanded mainly in the entertainment field, as an application that provides new experiences for users. Although there have been many studies on free viewpoint video generation for large-scale spaces, they have focused on generating video for the entire wide-area space, and challenges still remain in terms of reproduction of subject details and real-time video performance of image　generation. In this study, for the purpose of developing a system which enables immersion into a moving miniature world, a method to generate high-definition free viewpoint video for small-scale spaces in real time was realized and evaluated. It can generate high-definition free viewpoint images by combining free viewpoint depth images applying noise reduction processing and the corresponding color images considering occlusion and the normal direction of the object's surface. The free viewpoint images generated by the proposed system was evaluated. The results confirmed that consideration of occlusion and the normal direction of the subject's surface improved the quality of the generated images. In addition, the realized methods were able to generate clear free viewpoint images even when the free viewpoint approached the subject.

Keywords: Artificial Life, Application of Artificial Intelligence, AI and Applications
Abstract: In the aging society, the number of dementia patients continues to increase, and early detection of dementia is required. However, going to a hospital and being diagnosed by a doctor is burdensome for elderly people. This paper designs a Vision Transformer(ViT)-based gait diagnosis with micro-Doppler radar to diagnose dementia without burden for elderly people. The ViT is optimized by proposed Vertical Rectangle Patching and Adaptive Thresholding, which improve the Attention of ViT. The micro-Doppler radar collects the signal of elderly people, and the signal is transformed into two kinds of signal-analyzed images by two signal-analysis methods (Short-term Fourier Transform: STFT, and Continuous Wavelet Transform: CWT) for diagnosing by optimized ViT. Experiments compare eight kinds of CNN models, and current ViTs with the optimized ViT to evaluate the proposal's performance. The experimental results show that STFT is suitable for analyzing micro-Doppler radar signals, and ViT-56x4s+th, which uses Vertical Rectangle Patching and Adaptive Thresholding, achieves high scores such as an accuracy of 88.9%. Another proposed model ViT-224x1 allows for faster learning convergence for time series data and improved accuracy without fine-tuning. In summary, the possibility of diagnosing dementia by optimized ViT has been provided. The experimentation of Adaptive thresholding also proves there are some unimportant patches and brings an idea for reducing the unimportant computations to achieve a compact ViT as future work.

Keywords: Human-Machine Interface, Human Enhancements
Abstract: In the development of myoelectric prosthetic hands for reliable determination of the user’s intended motion, open-set recognition methods enabling consideration of unexpected inputs help to prevent malfunction. Conversely, some abnormal inputs may encompass new motions that can be used for motion classifier evolution and automatic acquisition of anomaly cluster structures. Against such a background, this paper outlines advanced open-set recognition involving clustering of unknown classes for enhanced ease of prosthetic hand interface usage. The proposed approach involves open-set recognition using the authors’ probabilistic neural network, a novel cluster detection method employing a non-parametric Bayesian model, and structural modification. Results from forearm motion recognition using electromyogram signals demonstrated the effectiveness of the technique.

Keywords: Assistive Technology, Human-Machine Interaction, Human Enhancements
Abstract: The Cable Assistive Rehabilitation Elbow Exoskeleton (CARE) is an active arm assistance device for in-home rehabilitation for individuals who suffered a traumatic brain injury, such as stroke, that experienced elbow flexion/extension impairment. A robotic elbow exoskeleton can provide ongoing therapeutic treatment to retrain arm movement functionality to perform activities of daily living. Providing an accessible, lightweight robotic exoskeleton is an ongoing challenge for wearable devices. CARE introduces a cable-driven design using a bidirectional, dual pulley system mated with a custom 17.5:1 planetary gearbox manufactured using high strength engineering stereolithography resin material. The cable-driven dual pulley system transfers the rotary motion to actuate an arm brace for elbow joint flexion and extension. Benchtop experiments demonstrated a maximum output torque of 26 Nm while lifting a 1kg object. Results exceeded the minimum joint torque and velocity requirement to carry out day-to-day activities. The actuation system used additive manufacturing techniques that decreased the transmission mass by 60% compared to conventional transmission designs, reduced cost for rapid prototyping, and provided formability for a compact elbow exoskeleton design. Overall, the system weighs less than 7 kg that can be translated to an in-home rehabilitation unit for stroke treatment.

Keywords: Agent-Based Modeling, Complex Network
Abstract: This paper introduces the liberating effect under group pressure into the HK model and proposes a novel expressed and private opinion dynamics model. Agents in the group hide their honest opinions because of the group pressure, and each agent has a private and expressed opinion. The liberating effect is divided into two stages. In the first stage, one agent in the group is the first to liberate when the number of times it feels pressure exceeds a specific limit and the cumulative pressure is the maximum. The liberating agent will express its opinion authentically, with private opinion consistent with the expressed opinion. In the second stage, the other agents in the group are influenced by the liberating neighbors and also liberate until the group evolution reaches a stable state. Through simulations, we study the effects of confidence level and pressure threshold on group opinion evolution. The experimental results show that both confidence level and pressure threshold are critical. All agents liberate when they are smaller than the critical value; when they are greater than the critical value, the liberating effect disappears. We also find that the liberating effect can accelerate the group opinion evolution.

Keywords: Cognitive Computing, Design Methods
Abstract: In recent years, quadratic optimizations have become increasingly popular in engineering. However, conventional methods that investigate this problem from the perspective of a canonical form with linear constraints are not effective in dealing with the significant challenges posed by quadratic constraints in practice. This paper proposes a solution framework for the quadratic optimization with quadratic constraints (QOQC) based on innovative artificial societies, computational experiments, and parallel execution (ACP) framework. Then, a gradient projection differential neural solution (GPDNS) is proposed to address this. To illustrate the effectiveness of the GPDNS model in solving the QOQC system, numerical simulations are provided. Overall, this paper presents the potential of innovative approaches like the ACP framework to enhance our capabilities in addressing challenging optimization systems.

Keywords: Assistive Technology, Design Methods, Human-Centered Transportation
Abstract: As ageing populations continue to grow, there is an increasing demand for mobility aids such as wheelchairs. However, conventional wheelchair wheels are designed for level ground and are not suitable for travelling over rough terrain or steps. In addition, with the rapid development of mega-cities, more obstacles such as steps and Braille blocks have been introduced, making outdoor mobility even more challenging for wheelchair users. Although electric wheelchairs have been developed for use in various environments, they are costly and not accessible for all users.To address this issue, the authors focused on developing an affordable wheel that can be used in various driving environments, including flat ground, steps, and rough terrain. They designed a wheel that utilizes a slider-crank mechanism, which can be adjusted by the wheelchair user using a central disc. In this paper, the design of the wheel is presented, and its performance was evaluated through experiments involving step climbing and rough terrain simulations. The results showed a reduction in the force required to overcome steps, improved stability when navigating steps, and enhanced driving performance on rough terrain after deploying the new wheel.

Keywords: Visual Analytics/Communication, User Interface Design, Design Methods
Abstract: Context: Process Mining (PM) aims to discover processes and their characteristics from event logs recorded by information systems. There are dozens of general-purpose tools. The Brazilian judiciary wants to make the technology available to magistrates with little or no knowledge of the field of PM. Problem: The usability and comprehensibility of the available tools prevent their adoption by laypeople. In fact, these are two of the eleven challenges proposed by the IEEE Task Force on PM that are still little explored in the context of non-specialists. Methodology: Applied qualitative research, using User Centered Design (UCD) principles to guide the construction of the JuMP tool, conceived considering anthropological and sociological aspects of the Brazilian judiciary. For a year and six months, a team of PM specialists worked with representatives of the judiciary sector to produce, evaluate and evolve the new product. An experiment was performed to evaluate the usability and understandability of JuMP. Results: The study demonstrated that the use of a PM tool oriented to the application domain is fundamental for domain experts with little or no knowledge in PM to be able to make good use of it. Contribution: Demonstration that, unlike the prevailing practice in the area of PM, which prioritizes the provision of general purpose tools, the design of tools oriented to the application domain are prerequisites to improve usability and comprehensibility attributes. Finally, the work proposes a generic and adaptable methodology for developing specific-purpose PM tools and the experience of its implications to the design of the JuMP tool.

Keywords: Cooperative Work in Design, Human Factors, Design Methods
Abstract: Falls among older adults are a serious global challenge. The World Health Organization strongly recommends gait, balance, and functional training, Tai Chi, or home assessment and modification to prevent falls among older people, but these strategies have not changed over the past few decades. The purpose of the present study is to propose an innovative approach to fall prevention naturally embedded in the environment. We first suggest new methods for simplifying and understanding the relationship between falls and everyday life, and describe this relationship as a knowledge graph using emergency transport data on falls among the aged. As a result of a cluster analysis using knowledge graphs, we identified whole-body balance as a key factor and living environmental determinant of falls. We then propose the concept of a homeostatic spatial system and discuss our development of seven homeostatic products as a proof of concept. Finally, we develop an evaluation system to visualize specific locations in a defined area where people maintain whole-body balance using their hands or other body parts. Our findings confirm that the system is useful for evaluating how people change their behaviors to maintain whole-body balance based on homeostatic products and the living environment as a whole.

Keywords: Wearable Computing, Environmental Sensing,, Human-centered Learning
Abstract: Musculoskeletal diseases such as low back pain may cause physical disabilities and make it difficult to continue working. From this point of view, logging of workers' conditions in the working environment is an important issue. So far, limited information acquisition such as smart watches has been realized, but comprehensive information has not been realized with a single device. Therefore, in this study, we estimated the wearer's state using a device called THINKLET, which can acquire image information and inertial information at the same time. By acquiring the pose of the hand from the image information, we were able to estimate the work. More detailed information can be obtained by calculating the position at the same time. By integrating image information with inertial information, it becomes possible to measure the position and posture of the wearer in the working environment. By complementing each other's sensor information, it is possible to obtain robust and accurate self-location. In this research, we developed these basic technologies and evaluated their performance.

Keywords: Wearable Computing
Abstract: Insole-attached force sensing resistors (FSRs) have been used as an alternative to force plates to estimate ground reaction force (GRF) during gait. Since the FSRs have a narrow sensing area and cannot measure force in the mediolateral and anteroposterior directions, it has been challenging to estimate three components of GRF with high accuracy. This paper reports on a gait measurement system using carbon insoles with attached piezoelectric film sensors and FSRs, and the estimation of three components of GRF using the system. Outputs of piezoelectric film sensors reflect the force that deforms the carbon insole in the mediolateral and anteroposterior directions. Using support vector regression (SVR) models trained with the piezoelectric film sensor and FSR data, the estimation accuracy of the three components of GRF was improved.

Keywords: Assistive Technology, Networking and Decision-Making, Environmental Sensing,
Abstract: One of the purposes of smart agriculture is to predict the yield of paddy rice using agricultural data using machine learning. LightGBM, one of the machine learning algorithms, is applied to the yield prediction problem of paddy rice in this paper. Since LightGBM has a large number of hyperparameters, the hyperparameter optimization using the stochastic schemata exploiter (SSE) is used. From the results of comparison with Genetic Algorithm (GA), it is confirmed that SSE has a fast convergence speed. In addition, it is found that the higher the mutation rate of SSE, the more converged to the global optimal solution without falling into the local solution.

Keywords: Human-Machine Interaction
Abstract: With human-machine interaction continuously developing, the consideration of human experience in the design of a machine becomes increasingly relevant. An experience measure that rates how well a work or interaction state is perceived by a user is psychological flow. In this paper we introduce a novel approach to assess flow in human-machine interaction by game-theoretically modeling the interaction. To validate our approach, we perform a user study with 30 participant with a study design that allows for the manipulation of the user’s experience through three experience modes. The results both validate our study design and provide indications that our approach to assess flow is promising to be developed further.

Keywords: Shared Control, Assistive Technology, Human-Collaborative Robotics
Abstract: Shared control is a mode where the user input is combined with a planned motion to achieve a common goal. In navigation, a shared control approach could provide a potential mobility solution for people who have a mobility impairment and find traditional powered wheelchairs unsuitable. While state-of-the-art work in shared control has demonstrated its capability in improving safety, human-machine interaction and reduce confusion, it is still challenging to use shared control in dynamic, crowded scenarios, in a way that is acceptable to users. Learning from recent advances in robot navigation, we present a reinforcement learning based framework, which allows navigation to be achieved in a user-specific shared controlled way. Our approach was trained and tested in a Unity3D based simulator. It achieved 33% fewer collisions, similar high user agreement (≥ 85%) and 27% less completion time when compared with our previous model-based method.

Keywords: Human-Machine Cooperation and Systems, Assistive Technology
Abstract: Many studies have shown that a smart wheelchair could improve the quality of life of people with restricted mobility by providing them with more freedom in the daily activities they can undertake independently. In addition to enhancing independent mobility, it is important to ensure safety for wheelchair users and those around them. To date, previous studies have mostly focused on (semi-)autonomous navigation or obstacle avoidance. By contrast, in this study, we tackle the challenging, but important problem of safely docking to tables. We propose a robotic navigation assistance, applied to electric powered wheelchairs using Time-of-Flight (ToF) sensors to facilitate table-docking for users. To meet this objective, we designed a low-cost sensor system that was integrated into our smart wheelchair prototype, which can detect a table and accurately estimate its height. We then developed a robust algorithm to deliver the manoeuvring assistance. First, we simulated the smart wheelchair system within Unity3D to find the best positions for the ToF sensors and evaluate the accuracy of the docking system, employing different table styles. Then, we experimentally validate the system on our physical wheelchair, using varying angles of approach, which demonstrate its feasibility.

Keywords: Shared Control, Human-Machine Cooperation and Systems, Human-Machine Interface
Abstract: Automated vehicle control beyond friction limits is an emerging technology that can further improve vehicle safety. This paper focuses on the driver's understanding of steering behavior during automated counter-steering. We hypothesize that: 1) motorsports fans or racing game players (GP) have a correct mental model of automated counter-steering, while 2) ordinary drivers (Non-GP) do not, and 3) Non-GP incorrectly estimate rotation direction as being always consistent between the steering wheel and vehicle. We first propose a method to examine the accuracy of a driver's steering mental model. The method measures the participants' ability to estimate steering wheel rotation from a video only showing vehicle rotation and vice versa. We compared the performance of GP and Non-GP using videos with and without counter-steering. The results only support hypothesis 2. This indicates that automated counter-steering likely confuses most drivers no matter their prior knowledge, which makes the improvement of driver-vehicle communication critical.

Keywords: Human-Machine Cooperation and Systems, Shared Control, Human-Machine Interaction
Abstract: Telemanipulation has become a promising technol- ogy that combines human intelligence with robotic capabilities to perform tasks remotely. However, it faces several challenges such as insufficient transparency, low immersion and limited feedback to the human operator. Moreover, the high cost of haptic interfaces is a major limitation for the application of telemanipulation in various fields, including elder care where our research is focused. To address these challenges, this pa- per proposes the usage of nonlinear model predictive control for telemanipulation using low-cost virtual reality controllers, including multiple control goals in the objective function. The framework utilizes models for human input prediction and task related models of the robot and the environment. The proposed framework is validated on an UR5e robot arm in the scenario of handling liquid without spilling. Further extensions of the framework such as pouring assistance and collision avoidance can easily be included.

Keywords: Human-Machine Cooperation and Systems, Shared Control, Augmented Cognition
Abstract: The current advancement of technology with not only physical, but increasing cognitive functions widely coined as Artificial Intelligence (AI) leads to multiple situations where humans willingly or unwillingly accept the decisions of algorithms and Neural Network (NN) Models resulting in humans being decreasingly involved in the decision-making process. Human-machine cooperation and Human Autonomy Teaming (HAT) are an answer to this problem, joining the best of both sides into joined cognitive systems. Meaningful Human Control (MHC) is the concept of ensuring that humans have enough influence on the outcome of actions in HAT and hence stay morally responsible and legally accountable for their actions. This paper extends the concept of using an interaction mediator that facilitates the interaction between an automated and a human agent with interaction patterns to maintain MHC. The concept is applicable in almost any domain, and will be shown with the example of guiding a highly automated vehicle.

Keywords: Brain-Computer Interfaces
Abstract: Identifying and minimizing physiological artifacts in EEG is challenging because these artifacts may corrupt the underlying brain activity severely. In this work, we proposed a hybrid approach to detect/reduce EEG artifacts by combining the MediaPipe face mesh model and artifact subspace reconstruction (ASR). Four types of artifacts, eye blinking, horizontal/vertical eye movements, and jaw movement during EEG measurement were generated to test our approach. We observed that real-time video-based artifact identification achieved over 95% accuracy in detecting eye blinking, horizontal eye movement, and jaw movement. Moreover, the targeted noise reduction was effective in analyzing the signal-to-noise ratio (SNR) for each specific artifact. This work may contribute to improving the reliability and accuracy of EEG data analysis in the real-world and online scenarios by providing a practical and effective approach to identifying and reducing physiological artifacts in real-time.

Keywords: Brain-Computer Interfaces, Wearable Computing, Medical Informatics
Abstract: Analysis of Electroencephalography (EEG) signals for everyday in-situ applications is hindered by many challenges including artifacts induced from physiological and environmental sources as well as anatomical factors. Recent studies have proposed analytical frameworks for the assessment of scalp EEG quality and identification of artifacts using rule based methods. These methods are typically used in an offline processing manner, employed before signal analysis when abundant data is available. With the advent of wearable devices, it is important to build techniques that work on short term signals giving us the ability of intervention based on the signal quality. Further, ear-EEG is a new modality that requires assessment and calibration with short term signals. To support this new modality we conducted a detailed study of scalp and ear-EEG data from the perspective of different measures as well as time duration. An algorithm is developed to identify the epochs with EOG and EMG artifacts in the ear-EEG using a set of metrics based on the characteristics of EEG. Thresholds of these metrics are determined by training on a dataset containing synchronous ear and scalp EEG with good results. The algorithm obtained an accuracy of 76.7% and 76.84% in classifying artifact EEG for scalp referenced ear-EEG and re-referenced ear-EEG, respectively.

Keywords: Brain-Computer Interfaces, Cognitive Computing, Human-Machine Interface
Abstract: There is a critical demand for BCI systems that can swiftly adapt to a new user and at the same time function with any user. We propose a fine-tuning approach for neural networks that serves a dual purpose; first, to minimize calibration times through requiring considerably less data - up to one-sixth - from the target subject than training from scratch, and second, to alleviate cases of user illiteracy by providing a substantial performance boost of over 11% in absolute accuracy from the features learned from other subjects. Ultimately, our adaptation method surpasses standard within-subject performance by a large margin in all subjects. We present ablation studies across three datasets, in which we demonstrate that fine-tuning outperforms other adaptation methods for BCI systems and that what matters most is the quantity of pre-training subjects, rather than their BCI-ability, achieving over 8% absolute increase in classification accuracy when scaling up the order of magnitude. Finally, we compare our approach to the state-of-the-art in EEG-based motor imagery and find it comparable, if not superior, to methods employing far more complex neural networks, obtaining 82.60% and 85.64% within-subject accuracy in the four-class BCIC IV-2a and binary MMI datasets respectively.

Keywords: BMI Emerging Applications, Other Neurotechnology and Brain-Related Topics, Passive BMIs
Abstract: Although the high performance of the convolutional neural networks (CNNs) for brain-computer interface (BCI) tasks based on raw electroencephalography (EEG) signals, the explanation of the prediction result remains challenging owing to their complex structure and numerous parameters. We propose a novel framework for explaining CNNs for EEG-based BCI tasks by using the perturbation-based influence scores. The method supports the interpretation of CNN classification for EEG signals at both the example-level and the feature-level. The experiments on the BCIC III-IVa dataset demonstrate that the proposed method is effective for not only the interpretation of the predictive models, but also for the improvement of the classification accuracy.

Keywords: Active BMIs, Other Neurotechnology and Brain-Related Topics
Abstract: Brain-computer interface (BCI) enables the communication between humans and devices by reflecting intentions and status of humans. Endogenous BCI is the imagined-based BCI and it has the advantage that the fatigue level of the body, especially the eyes, is relatively low and no additional equipment for offering stimulation is required. When conducting imagined speech, one of the endogenous BCI paradigms, the users imagine the pronunciation as if actually speaking. In contrast, overt speech is that the users directly pronounce the words. We proposed the transfer learning-based method from overt speech- to imagined speech-based electroencephalogram (EEG) signals (TOINet). The proposed method utilizes an encoder to extract the feature vector of imagined speech from EEG signals, which is subsequently reconstructed into overt speech signals using the decoder. Through this process, the model can identify the significant and common features present in EEG signals for both overt and imagined speech, facilitating the classification of EEG signals associated with imagined speech. Eight subjects participated in the experiment. The average accuracy of the TOINet was 0.4841 for classifying four words and the EEG features of overt speech improved the performance by 0.0742. Hence, we demonstrated that EEG features of overt speech could improve the decoding performance of imagined speech.

Keywords: Passive BMIs, BMI Emerging Applications, Other Neurotechnology and Brain-Related Topics
Abstract: The Steady-State Visually Evoked Potential (SSVEP) is a robust paradigm used to build a high-speed Brain- Computer Interface (BCI). This technology can benefit disabled subjects, allowing them to interact with their surroundings without using their peripheral nerves. However, one challenge to address is reducing the time calibration of BCI for a new subject (target subject) because of the high brain EEG variability among subjects and within subjects in different sessions. This constraint restricts the application of SSVEP-based BCI in natural environments; thus, some approaches to endeavor this constraint propose a linear transformation of existing subjects over some trials of the target subject. In this paper, we propose an approach to a nonlinear transformation (NLT) using an Extreme Learning Machine Autoencoder (ELM-AE) of SSVEP trials to improve a cross-subject classification reducing the calibration time for the target subject. Our results reported that the recognition accuracy improved by 6.58% for all subjects using NLT. Also, these results exhibit the feasibility of NLT that using a few templates from the target subject can enhance the recognition accuracy over cross-subject classification without NLT.

Keywords: Active BMIs, BMI Emerging Applications
Abstract: Brain-computer interfaces (BCIs) are a promising method for users to interact with machines using brain signals, primarily electroencephalography (EEG). Motor imagery (MI)- BCI, which decodes EEG signals induced by the user's imagination of moving body parts, has gained great attention due to its applicability in various fields such as robotics and rehabilitation. MI-EEG signals exhibit class-discriminative patterns such as event-related de/synchronization across spectral-spatio-temporal (SST) domains. Numerous studies adopted deep learning frame- work, especially convolutional neural network (CNN), for learning SST feature representations automatically in a data-driven manner. In particular, most of the CNN-based methods adopted temporal convolution to learn the spectral information, as it can act as a band-pass filter. In this paper, we propose SAT-Net, a SincNet-based attentive temporal convolutional network for motor imagery classification. The proposed method utilizes Sinc convolution from SincNet for explicit extraction of spectral information from the input EEG signals with high interpretability. Moreover, we adopt an attentive temporal convolutional network to effectively learn SST feature representations while making full use of temporal information. We evaluate our proposed SAT-Net on the public BCI Competition IV-2a dataset, comparing it not only to conventional CNN-based approaches but also to the state-of-the-art method. The experimental results, supported by statistical analysis, demonstrate that our approach outperforms the competing methods.

Keywords: Human Perception in Multimedia, Visual Analytics/Communication, Multimedia Systems
Abstract: Impressive advancements in capturing, display, and broadcasting technologies significantly elevate image and video quality, and with that the need for designing new reference and no-reference image and video quality metrics. One of the latest and perceptually accurate video quality metrics is the Video Multi-Method Assessment Fusion (VMAF) method. However, VMAF considers the temporal nature of video using basic average temporal pooling, an approach that falls short from human perception. In this paper, we introduce a new no-reference video quality metric that uses deep learning to extract spatial features and a unique temporal pooling approach to accurately predict the visual quality score. To this end, first we created a video quality dataset that consists of high-resolution, 20s-long test video clips compressed at several different bitrates. These videos were labeled based on subjective evaluations and were used to determine the perceptual importance of frames in our temporal pooling scheme. Evaluations showed that our proposed approach achieved correlation of 90.55% with human perception and outperformed the state-of-the-art VMAF approach by 15.63% accuracy.

Keywords: Human-Computer Interaction, Multimedia Systems, Affective Computing
Abstract: Technology users will eventually face death in the offline world. However, their data may survive for a long time, recorded in a wide variety of digital services and devices. This digital legacy must be handled and legally abide by the owners’ wishes, which may vary from inheritance to deletion. By carrying out a literature review on the subject, the authors noticed there is a lack of user studies on these issues. We, therefore, conducted a focus group with technology users of different profiles to understand the volunteers’ acceptability of this topic. During the group discussions, an unexpected factor was raised: to what extent might a user's fame influence their digital immortalization? This study aims to discuss digital immortality in terms of its acceptability among users and to contribute to the field of Human-Computer Interaction (HCI) with important insights on death in the digital world. In our findings regarding acceptability, users can feel some comfort in interacting with their deceased loved ones. Regarding fame, while influential individuals may not need digital immortality, there are concerns regarding the utilitarian nature of their legacies.

Keywords: Multimedia Systems, Human-Computer Interaction, Design Methods
Abstract: Designing attractive images for object display can be a time-consuming and skill-intensive process. The emergence of advanced algorithms, particularly the Diffusion Model, has made it possible to synthesize attractive images using AI. However, the existing diffusion models are mostly used to generate entire images and lack control over specific objects for object display. Here, to the best of our knowledge, we pioneers to extend the application of the diffusion model to synthesize novel images for specific objects. By encoding the input images of objects into NeRF representation and synthesizing the desired backgrounds using diffusion models with the input of rendered object images and text prompts, our method can generate 3D aware object display images at arbitrary angles and arbitrary backgrounds. We have conducted extensive experiments to demonstrate that our method is capable of generating high-quality and photo-realistic images, which are >6 times faster than the conventional photomontage approach. Moreover, our generated images have higher compositional scores, image quality scores, and aesthetics scores in our user experiments. By significantly reducing the need for human effort and producing higher quality generated images, our approach opens up exciting possibilities for creating versatile novel images of specific objects.

Keywords: Kansei (sense/emotion) Engineering, Multimedia Systems, Interactive Design Science and Engineering
Abstract: Recently, cross-modal correspondences, which is compatibility effects between attributes or dimensions of stimuli in different sensory modalities, have been drawing attention. In the case of the correspondences between sound and shape, the effect of “angularity” of shapes on sound pitch has been widely studied while there has been little research on the other features of shapes. Previous studies have shown that the human emotion mediates the cross-modal correspondences. Therefore, this study focuses on the complexity and order of shapes, which is said to influence human emotion, and aims to examine the cross-modal correspondences induced by its sensation. First, indices of complexity and order are selected. Next, shapes are selected based on each index. Finally, an experiment was conducted to investigate the correspondence between shapes and sound pitches. As a result, the correspondence between the complexity of shapes and higher pitches were found. On the other hand, the correspondence between the order of shape and lower pitches were not found.

Keywords: Cognitive Computing, Human-Centered Transportation, Systems Safety and Security
Abstract: Autonomous transportation systems such as road vehicles or vessels require the consideration of the static and dynamic environment to dislocate without collision. Anticipating the behavior of an agent in a given situation is required to adequately react to it in time. Developing deep learning-based models has become the dominant approach to motion prediction recently. The social environment is often considered through a CNNLSTM- based sub-module processing a social tensor that includes information of the past trajectory of surrounding agents. For the proposed transformer-based trajectory prediction model, an alternative, computationally more efficient social tensor definition and processing is suggested. It considers the interdependencies between target and surrounding agents at each time step directly instead of relying on information of last hidden LSTM states of individually processed agents. A transformer-based sub-module, the Social Tensor Transformer, is integrated into the overall prediction model. It is responsible for enriching the target agent’s dislocation features with social interaction information obtained from the social tensor. For the awareness of spatial limitations, dislocation features are defined in relation to the navigable area. This replaces additional, computationally expensive map processing sub-modules. An ablation study shows, that for longer prediction horizons, the deviation of the predicted trajectory from the ground truth is lower compared to a spatially and socially agnostic model. Even if the performance gain from a spatial-only to a spatial and social context-sensitive model is small in terms of common error measures, by visualizing the results it can be shown that the proposed model in fact is able to predict reactions to surrounding agents and explicitely allows an interpretable behavior.

Keywords: Assistive Technology
Abstract: Our society is built on the benefits of technological development, but this development also entails risks. When the convenience of a new technology outweighs the sense of anxiety about its risks, we can consider that this technology is socially accepted. In this study, we use the social media to analyze the general public's anxiety feelings toward various technologies and model the social acceptability of the new technologies based on this analysis. In this study, 6,452,730 tweets about three technologies: “automated driving,” “electronic currency,” and “drones” were collected, and the social acceptability of the technologies was examined using emotion classification, text categorization, and topic analysis techniques. As a result, we concluded that the anxiety about unemployment can be one perspective for analyzing the social acceptability of a technology.

Keywords: Human-Machine Interaction, Medical Informatics
Abstract: Tremors are typically experienced by patients at the beginning of Parkinson’s disease (PD). Clinicians evaluate clinical symptoms based on scale and experience, but mild tremors do not have significant characteristics and are difficult to observe with the naked eye. Implementing an intelligent and objective method to identify PD patients with early tremor symptoms and healthy controls (HC) is necessary. This study used wearable sensors to collect 9-axis inertial signals and 2-channel sEMG signals at the wrists of 13 HC and 24 PD patients from Dalian Municipal Central Hospital. Based on a Long Short-Term Memory Network (LSTM), an attention mechanism, and a Fully Convolutional Network (FCN), we develop a model to classify and recognize data from PD patients. Compared the proposed method with several classification methods, the results showed that the proposed method achieved higher classification accuracy (91.78%), precision (100%), recall (87.50%), and F1-score (93.33%) of PD class than Support Vector Machine, FCN, and LSTM. The computing time of the proposed method is approximately 1 second. The proposed method identifies early PD patients by pre-clinical assessment of mild upper limb tremors, which is valuable for early treatment and rehabilitation.

Keywords: Medical Informatics, Biometrics and Applications,
Abstract: The chest X-ray (CXR) is commonly employed to diagnose thoracic illnesses, but the challenge of achieving accurate automatic diagnosis through this method persists due to the complex relationship between pathology. In the past few years, numerous approaches based on deep learning have been proposed to address this issue but confounding factors such as image resolution or noise problems often damage model performance. In this paper, we focus on the chest X-ray classification task and proposed an interpretable instrumental variable (IV) learning framework, to eliminate the spurious association and obtain accurate causal representation. Specifically, we first construct a structural causal model (SCM) for our task and learn the confounders and the preliminary representations of IV, we then leverage electronic health record (EHR) as auxiliary information and we fuse the above feature with our transformer-based semantic fusion module, so the IV has the medical semantic. Meanwhile, the reliability of IV is further guaranteed via the constraints of mutual information between related causal variables. Finally, our approach’s performance is demonstrated using the MIMIC-CXR, NIH ChestX-ray 14, and CheXpert datasets, and we achieve competitive results.

Keywords: Wearable Computing, Medical Informatics, Human-centered Learning
Abstract: Chronic Obstructive Pulmonary Disease (COPD) is one of the leading causes of chronic diseases and deaths worldwide. When acute exacerbation of COPD (AECOPD) occurs, the frequency and severity of malignant attacks are highly correlated with mortality rate. The purpose of this study is to use wearable devices to collect physiological parameters of patients for early warning and prevention of complications of possible AECOPD attacks in the future. The subjects used wearable devices to measure Heart Rate Variability (HRV) at home. Physiological data and physiological assessment scales of 13 COPD patients were collected during the 6-month study period. According to the scale responses, the severity of the condition was classified into mild AE and no AE. If the subject needed emergency medical treatment due to COPD, it was classified as AE. With the scale classification method, a machine-learning Random Forest (RF) algorithm is used to predict the occurrence of AECOPD in the next 7 days, so as to prevent the deterioration of the disease in advance. The results of the study show that the accuracy of the model is more than 92% according to different classification methods, and using the mixed-parameter model as a feature for the prediction can improve the sensitivity of the original warning mechanism. In order to provide predictive results to the nursing staff at any time, the user interface of our system would transmit a warning message to remind the nursing staff to ensure early medical intervention for patients to avoid the occurrence of AECOPD.

Keywords: Human-Machine Interaction, Assistive Technology, Human-Computer Interaction
Abstract: Manual assembly workers face increasing complexity in their work. Human-centered assistance systems could help, but object recognition as an enabling technology hinders a sophisticated human-centered design of these systems. At the same time, activity recognition based on hand poses suffers from poor pose estimation in complex usage scenarios, such as wearing gloves. This paper presents a self-supervised pipeline for adapting hand pose estimation to specific use cases with minimal human interaction. This enables cheap and robust hand pose-based activity recognition. The pipeline consists of a general machine learning model for hand pose estimation trained on a generalized dataset, spatial and temporal filtering to account for anatomical constraints of the hand, and a retraining step to improve the model. Different parameter combinations are evaluated on a publicly available and annotated dataset. The best parameter and model combination is then applied to unlabeled videos from a manual assembly scenario. The effectiveness of the pipeline is demonstrated by training an activity recognition as a downstream task in the manual assembly scenario.

Keywords: Cognitive Computing, Human Factors, Intelligence Interaction
Abstract: The intermittency and randomness of wind power output have a negative impact on the stable operation of the power grid. Accurately modeling the uncertainty of wind power output is essential, and the primary method to achieve this is through scenario generation. Traditional scenario generation methods suffer from limitations such as low accuracy and high computational complexity. In this paper, a novel generation framework based on the denoising diffusion probabilistic model is presented and proposed for scenario generation of wind power. This method can overcome the limitations of traditional methods and learn the distribution of real data to generate reliable wind power scenarios. Compared to a homogeneous generative model, the proposed method shows improved performance in precisely capturing features of wind power scenarios.

Keywords: Cognitive Computing, Intelligence Interaction, Multi-User Interaction
Abstract: This paper introduces a novel algorithm, refer to NEWDQN, which is based on the deep Q-network (DQN) framework. The primary objective of this algorithm is to optimize the successful rate both in autonomous drone obstacle avoidance and target tracking tasks, while this algorithm can also improve the drawbacks of the previous algorithm in convergence. Furthermore, the algorithm endows the drone with environment perception capabilities and incorporates a direction-based reward-penalty function into the reward function, enhancing the drone’s generalization ability and overall performance. Extensive simulations demonstrate that compared to conventional DQN and Double DQN (DDQN) algorithms, NEWDQN exhibits faster convergence speed, shorter tracking paths, and more robust adaptability to different environments.

Keywords: Intelligence Interaction, Multimedia Systems, Medical Informatics
Abstract: Ultrasound image segmentation is crucial for early disease detection and treatment planning but remains a challenging task due to the low contrast of organ boundaries and varying image quality. Current methods often require manual intervention or have limited accuracy. In this paper, we propose a novel hybrid framework that combines an automatic option polygon segment (AOPS) algorithm and a distributed- and memory-based evolution (DME) algorithm for precise ultrasound organ segmentation. Our pipeline consists of two cascaded stages: (1) a coarse segmentation step using the AOPS algorithm, which determines the number of vertices/clusters without human intervention, and (2) a refinement step using the DME algorithm for hunting for the optimal neural network, which is then used to represent a smooth, explainable mathematical expression of the organ boundary. We employ the fractional backpropagation learning network with L2 regularization (FBLN) for training and use the scaled exponential linear unit (SELU) activation function to address the vanishing gradient problem. This is a new attempt such a hybrid framework is applied to ultrasound organ segmentation tasks, and it demonstrates significant contributions in terms of accuracy, smoothness, and computational efficiency.

Keywords: Virtual and Augmented Reality Systems, Virtual/Augmented/Mixed Reality, Interactive Design Science and Engineering
Abstract: Virtual Reality (VR) is increasingly considered a valuable therapeutic intervention for people with dementia (PwD). However, it has not yet been widely implemented or rigorously evaluated for use in private residences, where it has potential for significant impact on quality of life for both PwD and their family caregivers. This paper describes results from the VRx@Home Pilot study, which is among the first to explore the potential benefits of immersive VR experiences delivered through a head-mounted display when compared to a standard two-dimensional display (handheld tablet). This was a prospective mixed methods study involving seven PwD-caregiver dyads (n=14) who took part in a four-week home-based intervention (two weeks VR, two weeks Tablet-Only). We evaluated the feasibility, usability, and impact of 360-degree videos on the quality of life of PwD and their caregivers. These outcomes were assessed through in-app metrics, questionnaires, observations, and interviews conducted at a) baseline, b) after each phase of the intervention, and c) at the end of the study. Results revealed that the VR and Tablet-Only conditions were comparable in terms of ease-of-use, session length, and frequency. Both conditions appeared to positively affect in-the-moment mood and quality of life of PwD and their caregivers. Improvements to VR-content, system navigation, and evaluation measures were identified as factors that will increase the likelihood of VR-therapy being adopted in the home setting by PwD and their caregivers.

Keywords: Virtual and Augmented Reality Systems, Telepresence, Virtual/Augmented/Mixed Reality
Abstract: This study proposes a simplified telepresence system based on extended reality for increasing the number of remote-able tasks. The system provides an immersive virtual environment that duplicates remote real spaces to enable communication among system users in different geographic locations through their digital avatars. The motion of the avatars corresponds with that of the mobile devices manipulated by the system users, and the avatars’ position and orientation in the real space represent the viewpoint and viewing angle of the users. The three experiments showed that the proposed system creates a high-fidelity immersive environment in a short period of time and positively supports users' remote collaboration in the virtualized real space. The system successfully demonstrated its potential to contribute to the expansion of remotely enabled operations.

Keywords: Human-Computer Interaction, Team Performance and Training Systems
Abstract: As Computer Mediated Communications (CMCs) advance, businesses have sought alternatives to face-to-face (F2F) meetings to increase productivity for geographically dispersed teams while saving time and money. However, critical differences between CMCs and F2F impact multiple aspects of communication performance. To explore these differences, the present study examined communication performance in three conditions: video conferencing (VC), virtual reality (VR), and F2F. The study utilized an electrical circuit repair task and multiple surveys to collect data from 104 participants on four dependent variables: shared situational awareness, usability, mental workload, and performance confidence. For all the variables, results showed significantly better scores in VR and F2F conditions than in VC, but there was no significant difference between the VR and F2F conditions. These findings can inform technology developers in improving communication performance in computer mediated contexts, especially by using VR.

Keywords: Virtual and Augmented Reality Systems, Telepresence, Human-Machine Interaction
Abstract: The presence (or telepresence) feeling and virtual reality (VR) sickness affect the task execution in teleoperation. Most teleoperation works have assessed these concepts using objective (physiological signals) and subjective (questionnaires) measurements. However, these works did not include social tasks. To the best of our knowledge, there was no physiological approach in teleoperation of social tasks. We measured presence and VR sickness in a simulation of teleoperated social tasks by questionnaires and analyzed the correlation between their scores and multimodal biomarkers. The results showed some different correlations from the findings of non-teleoperation studies. These correlations were between presence and neural biomarkers in the frontal-central and central regions (for the beta and delta bands) and between VR sickness and brain biomarkers in the occipital region (for the alpha and beta bands) and the mean temperature. This work revealed significant correlations to support some biomarkers as predictors of the trend of presence and VR sickness in simulated teleoperated social tasks. These biomarkers might also be valid to predict the trend of telepresence and motion sickness in teleoperated social tasks in a remote environment.

Keywords: Virtual and Augmented Reality Systems, Medical Informatics
Abstract: It is estimated that 3.5-10 percent of the population are afraid of injections. These people are at risk of vasovagal syncope, a condition in which the patient faints during the injection. To help overcome this response to the perceived pain of injections, this paper proposes a diffuse noxious inhibitory control effect that can be applied through Augmented Reality (AR) smartphone applications. Furthermore, we confirm that presenting ARs that evoke noxious stimulation in participants can influence pain perception. We investigate the pain perception and negative affect when presented with noxious ARs and a distraction AR. The results show that pain thresholds are significantly increased in each AR condition when a painful stimulation is presented to participants. However, no difference in pain reduction between the noxious and distraction ARs is observed. One reason for this result may come from a manipulation problem: exposure to ARs does not make all participants feel pain. For the group of participants who felt pain when exposed to the ARs, the magnitude of pain tended to be smaller in a noxious AR (fire) condition than in the distraction condition. There was no significant increase in negative affect in the fire AR condition compared with the other conditions. These results may imply that the proposed method could reduce pain through diffuse noxious inhibitory control without producing any extra discomfort.

Keywords: Supervisory Control, Information Visualization, Resilience Engineering
Abstract: End-product quality prediction is crucial in smart manufacturing, where reliable evaluation and parameter optimization are essential for ensuring high-quality outputs. This study presents a novel approach that combines adaptive machine learning and nonlinear regression to accurately predict the quality of highly customized end products using limited supply-chain data through digital transformation. The research was conducted in collaboration with a major power transformer manufacturer and its supply chain partners. The adaptive model was trained and validated using real datasets from key components provided by the supply chain, resulting in accurate predictions of end-product quality. The model incorporates the core loss parameter, obtained from the power transformer's key component, as an input dataset for training and testing. The proposed approach, called AdaBoost-Regression, combines adaptive boosting (AdaBoost) and Regression machine learning techniques. Experimental results demonstrate that the AdaBoost-Regression model outperforms simple AdaBoost and Regression models in predicting transformer quality. The model also exhibits superior performance in terms of mean absolute percentage error (MAPE) and root mean square error (RMSE) during real-data verification. This approach has the potential to significantly reduce overall production costs by accurately predicting the quality of complex, expensive, and highly customized industrial products. It can be applied across various industrial sectors.

Keywords: Kansei (sense/emotion) Engineering
Abstract: Contemporary society is inundated with a plethora of information, which can induce anxiety in individuals. However, continuous exposure to various types of information generates a tolerance known as habituation. The dynamics of anxiety share similarities with allergic symptoms in terms of immune reactions to foreign substances. Immunotherapy, a well-established treatment for allergies, suppresses allergic symptoms by properly training the immune system. Anxiety tolerance exhibits parallels to allergic phenomena. This study proposes a KANSEI model based on a mathematical model for allergies focused on immunotherapy to explain anxiety dynamics using differential equations. We conducted numerical simulations using input-induced anxiety and provided a phenomenological discussion of the numerical results.

Keywords: Design Methods, Interactive Design Science and Engineering, Medical Informatics
Abstract: Several clinical decision support systems (CDSSs) have been developed to help practitioners in their diagnostic procedures in order to achieve the best care. This article presents the construction work of a knowledge graph within the framework of a CDSS dedicated to non-hematologist physicians whose main objective is to determine the urgent situations in hematology. First, we recall the basic notions concerning the different approaches of CDSSs. After having identified the skills issues and specified the need, we describe our first work on the construction of this graph based on the latest scientific data concerning the management of anemia and the proposed topology. We finally pass to the validation of the graph as well as its visualization.

Keywords: Design Methods, Medical Informatics, Interactive Design Science and Engineering
Abstract: This work is part of a project aiming to identify a profile of patients with an indolent form of chronic lymphocytic leukemia (CLL) using the MTS assay as a predictive marker. We describe the first results related to the construction of a knowledge graph representing data from heterogeneous data sources. After identifying the competency questions defining the prediction of the evolution of CLL, we propose a model to represent patient data in a knowledge graph, and we write the first expert rules to predict the disease progression.

Keywords: Human Factors
Abstract: The process of recycling and remanufacturing begins with disassembly. Through disassembly, the components with recycling value are decomposed. However, with the rapid development of production automation, designers often ignore the fact that manual operation is flexible but fails to achieve maximum production efficiency and profit. Therefore, the consideration of human factors in disassembly lines holds significant importance. This study delves into the multi-objective optimization of a U-shaped disassembly line balancing problem involving multiple products. A comprehensive objective function is developed, taking into account various factors including employee fatigue and other factors. To address the aforementioned problem, this study uses a collaborative resource allocation strategy within a multi-objective evolutionary algorithm based on decomposition. By comparing the results of different experimental cases, this paper shows that the proposed algorithm is more competitive than the carnivorous plant algorithm, fruit fly optimization algorithm, and Pareto archiving evolutionary strategy.

Keywords: Human Factors, Cooperative Work in Design, Information Systems for Design and Marketing
Abstract: In this paper, we continue our investigations on digital nomadism and the impact of COVID-19 pandemic on the work-related aspects and lifestyle of digital nomads (DN). The findings presented in this empirical study reflect the analysis of the impact of COVID-19 outbreak (and its waves) on the market economy and work-life boundaries of DNs as perceived from posts and comments gathered from a Reddit community during the period of early March 2020 until the end of 2022. From this point, our results indicate that the massification of remote work among formal workers in response to COVID-19 pandemic has impacted both the formal labor market and the DN ecosystem. As a consequence, we argue that digital nomadism tends to play a critical role beyond work from (almost) anywhere (WFA) in a post-COVID-19 era taking into account the novel facets of nomadic work-lifestyle.

Keywords: Information Systems for Design, Networking and Decision-Making, Human-Computer Interaction
Abstract: After decades of massive digitization, a substantial amount of documents exists in digital form. The accessibility of these documents is strongly impacted by the quality of document indexing. Most of these documents are indexed in noisy versions that include numerous errors. The noise can be due to manual input mistakes or optical character recognition process and results in errors like spelling mistakes, missing characters, and others.

This paper presents a study of the impact of noise on document ranking, an essential task in natural language processing (NLP) with wide-ranging practical applications. We provide a deep and quantitative analysis of the impact of recognition errors on document ranking by testing two popular ranking models on several noisy versions of a subset of the MS MARCO passage ranking dataset, with various levels and types of noise. Our study provides insights into the challenges of document ranking under noisy conditions and advocates for developing ranking models that are more robust to noise.

Keywords: Information Systems for Design and Marketing, Networking and Decision-Making, Information Visualization
Abstract: In today’s data-rich environment, recommender systems play a crucial role in decision support systems. They provide to users personalized recommendations and explanations about these recommendations. Embedding-based models, despite their widespread use, often suffer from a lack of interpretability, which can undermine trust and user engagement. This paper presents an approach that combines embedding-based and semantic-based models to generate post-hoc explanations in recommender systems, leveraging ontology-based knowledge graphs to improve interpretability and explainability. By organizing data within a structured framework, ontologies enable the modeling of intricate relationships between entities, which is essential for generating explanations. By combining embedding-based and semantic based models for post-hoc explanations in recommender systems, the framework we defined aims at producing meaningful and easy-to-understand explanations, enhancing user trust and satisfaction, and potentially promoting the adoption of recommender systems across the e-commerce sector.

Keywords: Cooperative Work in Design, Networking and Decision-Making, Interactive Design Science and Engineering
Abstract: Crisis situations can present complex and multifaceted challenges, often requiring the involvement of multiple organizations and stakeholders with varying areas of expertise, responsibilities, and resources. Acquiring accurate and timely information about impacted areas is crucial to effectively respond to these crises. In this paper, we investigate how collaborative and social technologies help to contextualize crises, including identifying impacted areas and real-time needs. To this end, we define CORec-Cri (Contextulized Ontology-based Recommender system for crisis management) based on existing work. Our motivation for this approach is two-fold: first, effective collaboration among stakeholders is essential for efficient and coordinated crisis response; second, social computing facilitates interaction, information flow, and collaboration among stakeholders. We detail the key components of our system design, highlighting its potential to support decision-making, resource allocation, and communication among stakeholders. Finally, we provide examples of how our system can be applied to contextualize crises to improve crisis management.

Keywords: Human-Computer Interaction, Assistive Technology, Design Methods
Abstract: Robotic rehabilitation for paralyzed hands utilizes exoskeletons and soft gloves equipped with active mechanisms to provide support for hand motion. From a safety and control perspective, it is imperative to measure finger joint angles during motion support provided by a soft robotic wearable system. However, embedding sensors into these devices can be inconvenient as it may lead to bulkiness or structural difficulties. This study aims to develop a machine learning model for estimating finger joint angles from images utilizing data created with computer graphics (CG), and to validate the feasibility of this method through basic experiments. The three-dimensional CG (3DCG) hand model includes bones corresponding to the major joints of the fingers, and a wearable system of the index finger imported from a computer-aided design software was attached to the 3DCG hand model. After rendering the integrated motion between the hand model and the wearable system for finger flexion and extension, images in conjunction with the finger joint angles were recorded as the training data. The finger joint angle estimator was based on a pre-trained vision transformer model and was learned using the created 3DCG training dataset. The basic experiments showed that the developed machine learning model enabled the estimation of finger joint angles with reproducibility for four types of hand postures with the wearable system. Furthermore, the differences between the joint angles measured in the real world and those estimated by the developed model were smaller than those for an existing hand pose estimation model. The developed machine learning model, which utilizes 3DCG, has the potential to estimate finger joint angles with the wearable system using image videos.

Keywords: Brain-Computer Interfaces, Design Methods, Information Visualization
Abstract: Numerous studies suggest that learning related but different tasks prior to a new task makes it easier, possibly because of our brain's neural pattern alignment mechanism. Specifically, the neural patterns in the new task align with those in the learned task, enabling the reuse of knowledge from the previous task to aid learning in the new task. Brain-machine interface (BMI) is an excellent tool for analyzing the dynamics of neural population patterns during new task learning by directly recording neural signals from the brain. If we can repeat the neural pattern alignment process through an alignment algorithm using the recorded neural signals, it would provide a computational tool to help us understand the brain mechanism during task learning. Additionally, the pre-trained decoder parameters from the old task can be reused to expedite learning in the new task. However, the existing Iterative Closest Point (ICP) method easily fails as it is sensitive to neural data distribution. This paper proposes a pair-wise Kullback Leibler (KL) divergence optimizing framework for stable neural pattern alignment. The KL divergence measures the difference between the data distribution of the previous task and the aligned new task. The alignment process is formulated as an optimization problem by minimizing the KL divergence. The proposed algorithm is tested in a simulated experiment where a rat learns a two-lever discrimination task from a one-lever pressing task. Three scenarios are designed to test the feasibility of our algorithm, including non-Gaussian neural pattern shapes, noisy neural data, and different alignment angles. The results demonstrate that the proposed method is more robust than ICP, indicating its potential to discover the brain's alignment mechanism more accurately.

Keywords: Systems Safety and Security
Abstract: Microsoft Office documents have become hackers’ preferred tool to construct malicious documents. However, current research on detecting malicious Office documents has not covered all document formats and various types of malicious attacks. To address this issue, this paper proposes a Static Multi-class Malicious Office document Detection Method (SM2ODM) for multiple versions of Office documents. The focus of this research is to design a unified static feature representation method for multiple versions of Office documents via multi-feature fusion, including VBA (Visual Basic for Applications) code keywords, DDE (Dynamic Data Exchange) instructions, embedded files, OLE (Object Linking and Embedding) objects, external links, and other relevant features. In addition, this research identifies eight new types of malicious features and embedding locations. Then, this paper proposes a multi-class detection method for malicious Office documents that can detect five common types of malicious documents. Through analyzing 20,000 samples provided by Topsec Technologies Group, the proposed SM2ODM achieves high accuracy in multi-classification detection and identifies 185 malicious Office samples that common antivirus software failed to detect.

Keywords: Human-Computer Interaction, Human-Machine Interaction, Human-Machine Interface
Abstract: While there has been renewed interest in the use of myoelectric control for general-purpose applications, the burden of training and maintaining robust models still limits its real-world viability. Online unsupervised adaptation has been proposed to solve this issue by updating the model using predicted pseudo-labels in real time during regular device use. Until now, however, these unsupervised strategies have been limited as they rely on the very classifier outputs they are adapting, making them ill-suited when there is a drastic shift in the input space (e.g., after donning and doffing a device) or there is insufficient training data. In such situations, leveraging context (i.e., task-specific information that can help understand or assess a circumstance) could provide additional guidance for adaptation and improve its robustness. Although difficult to extract in traditional prosthesis control use cases without additional sensors, context may be more readily available in other general-purpose applications, such as in human-computer interaction. In this study, we explore leveraging context, both positive (i.e., reinforcing correct actions) and negative (i.e., correcting poor actions), for conditioning pseudo-label predictions within an adaptive gamified target acquisition setting. The results show that leveraging this additional context significantly outperforms the current state-of-the-art high-confidence unsupervised adaptation (p<0.05) using both offline and online performance metrics. This pilot work contributes novel findings and contextual approaches that do not rely on additional sensors, and thus outlines a promising direction of study for myoelectric control as a reliable and effective interaction technique.

Keywords: Assistive Technology, Human-Computer Interaction
Abstract: People with metamorphopsia suffer from perceiving things in a distorted way. Various methods for examining metamorphopsia have been suggested in the current literature, with the most advanced techniques demonstrating the ability to yield quantitative measurements. However, these cutting-edge methods necessitate extended examination durations and impose challenging manipulations on patients. In this study, our objective is to enhance the time efficiency of the inspection process and alleviate the burden placed on the user. We propose a novel user-friendly quantitative inspection system which utilizes interactive reinforcement learning. Instead of having users directly operate the system, we ask them to evaluate the stimuli generated by the system. Based on their evaluations, the system gradually refines the deformation map representing the distortion perceived by the user. The reinforcement learning scheme is implemented using relevance feedback approach based on optimum-path forest classifier. To evaluate the effectiveness of the proposed system, subjective evaluation experiments involving simulated and real metamorphopsia participants were conducted in this study. The experimental findings reveal that, when compared to the state-of-the-art method, our proposed system yields comparable inspection outcomes while significantly reducing both the inspection duration and the mental workload.

Keywords: Human-Centered Transportation, Human-centered Learning, Human-Machine Interaction
Abstract: Advanced Driver Assistance Systems (ADAS) are increasingly important in improving driving safety and comfort, with Adaptive Cruise Control (ACC) being one of the most widely used. However, pre-defined ACC settings may not always align with driver's preferences and habits, leading to discomfort and potential safety issues. Personalized ACC (P-ACC) has been proposed to address this problem, but most existing research uses historical driving data to imitate behaviors that conform to driver preferences, neglecting real-time driver feedback. To bridge this gap, we propose a cloud-vehicle collaborative P-ACC framework that incorporates driver feedback adaptation in real time. The framework is divided into offline and online parts. The offline component records the driver's naturalistic car-following trajectory and uses inverse reinforcement learning (IRL) to train the model on the cloud. In the online component, driver feedback is used to update the driving gap preference in real time. The model is then retrained on the cloud with driver's takeover trajectories, achieving incremental learning to better match driver's preference. Human-in-the-loop (HuiL) simulation experiments demonstrate that our proposed method significantly reduces driver intervention in automatic control systems by up to 62.8%. By incorporating real-time driver feedback, our approach enhances the comfort and safety of P-ACC, providing a personalized and adaptable driving experience.

Keywords: Companion Technology, Assistive Technology, Affective Computing
Abstract: This study aimed to develop a counseling robot to improve the quality of life (QOL) of older adults and verify its effectiveness. QOL is a comprehensive indicator that includes physical, mental, and social aspects, and gerontechnology aims to improve the autonomy and QOL of older adults. In recent years, utilizing robots to address the shortage of caregivers has been increasingly studied. We developed a counseling robot that estimates the QOL of older adults in real time, and generates appropriate and empathetic responses according to the estimated QOL. The results obtained from a one-week interaction experiment with a counseling robot targeted toward older adults indicated a significant improvement in the mental aspect of QOL, and the use of cognitive behavioral therapy and empathetic responses was inferred to facilitate self-disclosure and enhance the effectiveness of counseling. Additionally, advice based on the QOL estimation results contributed to organizing the thoughts of older adults and further improved their mental health. This study not only contributes to improving the QOL of older adults but also suggests that robots that understand and appropriately respond to individuals can facilitate continuous relationship building. This study may also serve as a guide for promoting the introduction of information and communication technology into welfare facilities.

Keywords: Affective Computing, Human-Machine Interaction, Intelligence Interaction
Abstract: To satisfy the required recognition accuracy from a far distance, an adjusted attention-based YOLOX for face recognition (AA-YOLOX-FR) is designed by an appropriate segmentation of the original image, so that faces’ pixels (e.g.,20x20 at 15m) effectively train, validate, and test. Based on an edge computing platform (e.g., NVIDIA Jetson-AGX), the processing time for the image with 8 segmentations of 640x640 equals 296.3ms in comparison to 95.4ms for its down-sampling to 640x640. Although the down-sampling technique achieves a faster processing speed, its recognition rate decreases as a face is at a far distance or with different lighting conditions. The average online video-based recognition rate of AA-YOLOX-FR for a distance from 10m to 15 m and different lighting conditions is 92.5%.

Keywords: Assistive Technology, Wearable Computing, Haptic Systems
Abstract: Patients with hand paralysis have motor and sensory dysfunction in their hands. They can neither flex/extend their fingers nor feel gripping force or finger posture. To allow patients with hand paralysis to perform gripping motions, it is necessary to assist finger motion and provide sensory information about the gripping force and posture at the residual sensory area. We previously developed a wearable finger-motion assist system that assists finger motion and measures the gripping force necessary to present sensory information. Therefore, the purpose of this study is to propose and develop a wearable system that provides multiple types of sensory information based on gripping force and finger posture estimated by finger-motion assist system, and to confirm the basic performance of the system through experiments. We developed a band-type sensory feedback system that tightens the forearm in response to gripping force using tendon-driven wires. A vibrator is installed in it, and the intensity of vibration changes in response to finger posture. As basic experiments, identification of gripping force and finger posture using band squeezing and vibration was tested. The results showed the overall identification rate for the combination of gripping force and finger posture that were both correct was 75.6%. The identification rates for gripping force and finger posture when stimulations were presented simultaneously were 93.3 % and 80.9 %, respectively. In conclusion, we confirmed the feasibility of this system that presents multiple types of sensory information.

Keywords: Assistive Technology
Abstract: Although the average life expectancy in Japan has been increasing in recent years, the problem of the large gap between healthy life expectancy and average life expectancy is still unresolved. Among the factors that lead to the need for nursing care, injuries due to falls account for a certain percentage of the total. In this paper, we developed boots that can provide external disturbance to the ankle with pneumatic gel muscles (PGM). We conducted an experiment using electromyographic potentials (EMG) and center of foot pressure (COP) as evaluation indices to evaluate the effectiveness of fall prevention training using this device, which is smaller and lighter than conventional devices and reported on the usefulness of that one.

Keywords: Assistive Technology, Human Performance Modeling
Abstract: The authors have developed a cane gait-training machine that enables stroke and paraplegic patients to safely rehabilitate on their own. This training machine is pulled by a wire connected to a harness on the patient’s waist; thus, the training machine can follow the patient without the use of hands. However, the machine’s ability to follow the walker is an issue. Therefore, we motorised the casters and set them to follow the patient’s movements. To cope with the transmission, processing, and mechanical delays that occur in this process, and for predictive control, we used long short-term memory, a machine learning method, which predicted the future waist gait speed from the acceleration measured at multiple sites on the body. In this study, we examined the effects on prediction error of varying the combination of acceleration measurement sites used for learning and the prediction horizon, which is the target time to be predicted. Prediction errors under certain conditions enabled the prediction of each subject’s average gait speed with an accuracy of 3–5%. Overall, the prediction error increased with longer prediction horizon but temporarily decreased at 0.4 s. Although prediction is possible using only one site on the lower body, we believe that prediction using multiple sites will reduce the error due to noise.

Keywords: Intelligence Interaction, Assistive Technology
Abstract: The rapid increase in the elderly and disabled population has been identified as a growing socioeconomic problem. Due to reasons such as a lack of reliable caretakers and the need to empower the elderly and disabled population, it is important to have assistive devices. The interactive capabilities of these devices should match the nature of the interaction that prospective users would have with their companions. Humans communicate with each other in many modalities, such as speech, hand gestures, head gestures, gaze, etc. Hand gestures have been a popular modality that has been used in these interactive devices for speech and mobility-impaired wheelchair users. There have been many gesture models that have been developed recently for hand gesture-controlled wheelchair navigation. Natural hand gestures that are used in human-human interactions include both static and dynamic gestures. Therefore, it was logical to include both of these gestures in a navigational gesture model or hand gesture-controlled navigational system. However, hand tremors that are prevalent among the elderly and disabled community could affect the nature of the hand gesture. These tremors can vary from person to person, and hence the fixed ranges cannot be used for hand features. Hand features such as palm velocity, fingertip velocity, and others will have different ranges from person to person. Due to these reasons, a static hand gesture intended by the human user could be identified as a dynamic gesture. Further, this could lead to the misrecognition of gestures defined in gesture models. Therefore, a system is proposed in this paper to validate the gestures by considering the activity of hand features in 3D regions defined for the gesture. The accuracies of the improved system for static and dynamic gestures were 0.9849 and 0.9840, which were improvements from the accuracies of 0.8994 and 0.8479.

Keywords: Brain-Computer Interfaces, Human-Machine Interaction, Biometrics and Applications,
Abstract: The rise of security concerns has spurred ongoing research into Brain-Computer Interfaces (BCI) based authentication. These applications utilize electroencephalogram (EEG) signals, due to their properties that can enhance security systems. In previous studies, EEG data has been incorporated into various authentication systems to compare the performance of new and existing classification methods. However, using EEG data to compare the performance of distance metrics in a P300-based BCI authentication system has not been explored yet. In this study, EEG data is used to determine the most effective distance metric for authenticating users in a closed-loop system. To accomplish this task, we conducted a longitudinal study to evaluate three distance metrics (Cosine, Correlation and Chebyshev) while participants interacted with our BCI authentication system. Our results indicated that the Cosine similarity outperformed all other distance metrics for each user.

Keywords: Brain-Computer Interfaces
Abstract: Invasive Brain Computer Interface (BCI) systems through Electrocorticographic (ECoG) signals require efficient recognition of spatiotemporal patterns from a multi-electrodes sensor array. Such signals are excellent candidates for automated pattern recognition through machine learning algorithms. The importance of these patterns can be highlighted through feature extraction techniques. However, the signal variability due to non-stationarity is ignored while extracting features, and which features to use can be challenging to figure out by visual inspection. In this study, we introduce the signal split parameter to account for the variability of the signal and increase the accuracy of the machine learning classifier. We use genetic selection, which allows the selection of the optimal combination of features from a pool of 8 different feature sets. Genetic selection of features increases accuracy and reduces the BCI’s prediction time. Along with Genetic selection, we also use a reduced signal length, which leads to a higher Information Transfer Rate. Thus this approach enables the design of a fast and accurate motor-related EcoG BCI system.

Keywords: Brain-Computer Interfaces
Abstract: The concept of barrier certificate has been developed recently in control theory to give formal guarantees on safety of a dynamical system. Neural mass models (NMMs) simulate the aggregated activity of neurons in the brain and have been used to model phenomena such as epilepsy. With a view to move towards novel treatments for epilepsy by investigating the application of control theory to epilepsy, we take one such NMM, the Wilson-Cowan (WC) model, and show that it is possible to automatically generate barrier certificates in both deterministic and non-deterministic cases, where the parameters of the model belong to an uncertainty set.

Keywords: Brain-Computer Interfaces, Human Performance Modeling, Multi-User Interaction
Abstract: Emerging research has shown that interbrain synchronization can occur when individuals interact on a shared task or receive shared stimuli. While the mechanisms producing such effects are not fully understood, interbrain synchronization has been observed in multiple contexts. Analyses of such synchrony, however, are often limited to individual, bottom-up features such as activity at individual cortical sites and frequency bands, or cross-projections of the data into a common space that maximizes correlation. It is an open question whether synchronization can also be observed using more top-down notions of latent state composed of multiple, simultaneous features that are not fit to the individual or task at hand? Here, we investigate whether latent state estimation methods, borrowed from the domain generalization toolboxes of the brain-computer interface community can be used to assess interbrain synchrony. Domain generalization methods learn models of neural activity that transfer from one domain to another without the need for participant- or task-specific training data. We use data from single individuals in a controlled task to train a domain-generalized model to detect latent state changes previously described as relating to alertness or vigilance. We apply the model to a novel, target domain in which teams of two worked together to locate and defuse simulated improvised explosive devices. Our results revealed significantly greater correlation between the latent states of dyadic partners compared to randomly paired individuals, establishing that interbrain synchronization can be observed using top-down methods that are not fit to the individuals, or task, in question.

Keywords: Brain-based Information Communications
Abstract: This study aims to clarify how functional network hubs change during hierarchical visual processing in the human brain through the estimation of brain states from features extracted using a convolutional neural network (CNN), a hierarchical model of image processing. We used representational similarity analysis for brain states predicted through encoding models based on feature representations at each layer of the CNN, and applied the PageRank algorithm to matrices converted from the generated representational dissimilarity matrices to capture the hub characteristics of brain region-related systems. This succeeded in capturing changes in the hubness of interregional brain coordination during hierarchical information processing in the human cerebral cortex in visual processing. Specifically, we found that the hubness of the occipital visual cortex increased in the early phase of visual processing, and that the hubness of the prefrontal cortex and temporal lobe increased in the late phase of visual processing. From the above, we found that our proposed method allows us to capture hierarchical changes in the hubness of interregional coordination.

Keywords: Other Neurotechnology and Brain-Related Topics, BMI Emerging Applications, Active BMIs
Abstract: Electroencephalography (EEG) signals capture the electrical activity of the brain and have traditionally been used in clinical settings. More recently, with the development of mobile, wearable EEG devices their use has been explored for other applications, including emotion recognition, quality of experience monitoring, or fatigue detection, just to name a few. EEG signals are known to have a 1/f-noise like structure and are very low amplitude, thus making them highly susceptible to artefacts, such as power line interference, muscle movement, and eye blinks. Moreover, prototyping and development of new devices to record EEG signals may introduce additional sources of artefacts generated by different instrumentation settings. As such, automated quantification of the quality of the EEG signals has become important and the focus of recent research. Here, we propose a new quality metric based on the 1/f-noise structure of the EEG signal. Experimental results show the proposed metric classifying clean versus noisy EEG segments in a subject-independent setting with an accuracy of 86.0% for the AF7 electrode location and 64.6% for AF8, two electrode locations known to be highly degraded by artefacts. Additionally, the proposed metrics are shown to generalize well to unseen electrode locations. For example, a quality model trained on AF7 noisy EEG data achieved an accuracy of 61.4% when tested on data collected from the AF8 location.

Keywords: Passive BMIs, BMI Emerging Applications
Abstract: For Brain-Computer Interfaces to be affordable and efficient, it is worth analyzing the importance of individual EEG channels and finding the smallest subset that still provides adequate results. In this work, we applied five different feature importance approaches to three different datasets focused on internally and externally directed attention. The methods used were: Random Forest Importance, Mutual Information, Permutation Importance, Shapley Additive Explanations, and an Ablation Study. We determined the importance of EEG channels, compared the results among the algorithms through correlation analysis, and evaluated the classification performance using different subsets of channels to validate the importance rankings. The results indicate that, in line with the existing literature, electrodes located on the right parietal cortex with the alpha frequency band appear to be the most important, followed by several channels covering the left parietal and frontal lobes. For the first dataset, we were able to reduce the number of channels from 32 to 2 while increasing the accuracy from approximately 60% to 62%. In the second dataset, we reduced the number of channels from 12 to 1 while improving the accuracy from approximately 59.5% to 64%. In the third dataset, we reduced the number of channels from 19 to 1 while only slightly decreasing the accuracy from approximately 60% to 58.5%. The results of the individual feature importance methods generally exhibit positive correlations with each other. Furthermore, we demonstrated that the rankings of channels between subjects are largely positively correlated, suggesting the presence of shared patterns of neural activity across subjects. The substantial reduction in EEG channels, identification of crucial brain regions and application of channel feature importance methods to internal/external attention data, collectively advance the development of cost-effective and efficient Brain-Computer Interfaces, paving the way for future advancements in the field.

Keywords: BMI Emerging Applications, Other Neurotechnology and Brain-Related Topics, Active BMIs
Abstract: Numerous studies have explored the neural correlates of musical rhythms using various neuroimaging modalities. Non-invasive neuroimaging modalities lack either the spatial or temporal resolution to reveal the nuances the neural processes involved in perception of musical rhythms. Intracranial recordings of electrophysiological activity such as electrocorticography (ECoG) can jointly provide spatial and temporal resolution for improved characterization and modeling of the underlying processes. The present study examines anticipatory and perceptual models that use ECoG recordings to estimate simple perceived and imagined musical rhythms in human participants. The resulting models are characterized and compared across participants. The results show that the anticipatory and perceptual models can reconstruct the auditory stimulus envelope with statistically-significant correlations when trained and tested on independent listening data. However, these models are unable to reliably reconstruct the expected rhythm pattern when trained on listening data and applied to imagining data. This suggests, similar to recent findings in overt and imagined speech decoding using intracranial signals, that there are likely distinct neural substrates activated during listening and imagining of musical rhythms.

Keywords: BMI Emerging Applications, Active BMIs, Other Neurotechnology and Brain-Related Topics
Abstract: Speech neurprostheses have the potential to provide severely paralyzed patients with a means of communication. To enable the best possible decoding of speech processes from neural data, it is important to chose a representation of speech that is both meaningful to the decoding process and represented well in the neural recordings.

Previously, acoustic, articulatory, and textual representations of speech have been decoded from neural recordings. Semantic representations of speech could add additional information about the content of the produced speech. In this study, we show that semantic embeddings for individual words, as extracted by a emph{word2vec}-model, can be used to reconstruct neural activity during speech production across wide-spread cortical and subcortical areas. We elucidate the temporal dynamics of reconstruction quality and show that a slight right hemisphere preference exists. These findings could be used to add semantic information into speech neuroprostheses in the future.

Keywords: Passive BMIs, BMI Emerging Applications
Abstract: The detection of an individual's attentional state via a Brain-Computer Interface (BCI) holds significant promise, offering a multitude of possibilities, including enhancing the usability of applications and enabling timely alerts in hazardous situations. However, the variability of EEG data between individuals and the dynamic nature of recordings pose practical challenges for achieving reliable results with BCIs. Thus, conventional methods often require the collection of each person's training data prior to usage, which is then used to train an individual model for detection. Such training data collection , makes it difficult to achieve practical use. To overcome this challenge, we propose a self-improving online learning system that personalizes a person-independent model for detecting attentional state in real-time during runtime. This eliminates the need for collecting individual training data prior to usage and instead generates the necessary labels for adaptation using automatically detected error-related potentials. The system was developed based on pre-trained models of two classifiers and used to evaluate different strategies of adaptation and label generation. A statistically significant accuracy improvement of 0.088 was achieved across all available subjects, based on simulations with pre-recorded data. These results suggest that person-dependent models for attentional state detection could in the future be substituted by self-improving classifiers that do not require a dedicated training data collection.

Keywords: Passive BMIs, Other Neurotechnology and Brain-Related Topics
Abstract: The assessment of mental workload from electroencephalogram (EEG) data for brain-computer interfaces (BCI) poses some challenges due to the interaction of spatial and temporal features within the data. Similar challenges are well known in the analysis of turbulent flows, which exhibit complex space-time correlations resulting from large-scale structures. The similarities motivated us to conduct this feasibility work of applying analytic methods of fluid dynamics – i.e., a scheme combining spectral proper orthogonal decomposition (SPOD) and spherical harmonics as basis functions – to EEG data for identifying features representing mental states. Based on EEG data of an existing BCI Hackathon, the scheme yielded some relevant features across subjects and sessions by relying only on a Fourier transform in time and the basis functions in space. The features were then classified by employing a conventional support vector machine algorithm to produce an accuracy comparable to those reported in a previous study on the same EEG data. This performance comparability indicates the scheme’s potential for analyzing EEG data in BCI applications. Nevertheless, future work is needed to select specific features as general indicators of mental workload.

Keywords: Human Factors
Abstract: Recently, multiple naturalistic traffic datasets of human-driven trajectories have been published (e.g., highD, NGSim, and pNEUMA). These datasets have been used in studies that investigate variability in human driving behavior, for example for scenario-based validation of autonomous vehicle (AV) behavior, modeling driver behavior, or validating driver models. Thus far, these studies focused on the variability on an operational level (e.g., velocity profiles during a lane change), not on a tactical level (i.e., to change lanes or not). Investigating the variability on both levels is necessary to develop driver models and AVs that include multiple tactical behaviors. To expose multi-level variability, the human responses to the same traffic scene could be investigated. However, no method exists to automatically extract similar scenes from datasets. Here, we present a four-step extraction method that uses the Hausdorff distance, a mathematical distance metric for sets. We performed a case study on the highD dataset that showed that the method is practically applicable. The human responses to the selected scenes exposed the variability on both the tactical and operational levels. With this new method, the variability in operational and tactical human behavior can be investigated, without the need for costly and time-consuming driving-simulator experiments.

Keywords: Affective Computing, Biometrics and Applications,, Human Factors
Abstract: Industry 5.0 promotes the development of human-centered industrial operations fueled by a fresh wave of disruptive technologies that encourage synergistic human-machine integration. Its focus is on understanding how human cognition contributes to a more secure and harmonious coexistence between humans and machines in industrial scenarios, employing solutions that prioritize fundamental worker demands while preserving or enhancing industrial productivity. In this context, the ability to assess fatigue objectively is crucial for occupational health and safety because it can reduce cognitive and motor function, ultimately lowering productivity and raising the risk of harm to human operators. To this end, wearable systems provide a promising solution for continuous, non-intrusive, and long-term monitoring of biological signals for fatigue detection. However, the adoption of these devices presents unique challenges, such as inter-individual variability that renders traditional one-size-fits-all machine learning models unsuitable. This paper provides an analysis of the current state-of-the-art for wearable device monitoring, including ongoing issues and current knowledge gaps. In addition, an experimental analysis is presented, employing a pattern discovery pipeline based on unsupervised learning on a real-world dataset. Our analysis provides experimental evidence of the limitations of one of the classical approaches to fatigue assessment, thus highlighting the need for more advanced models.

Keywords: Team Performance and Training Systems, Human Factors
Abstract: In this paper we describe the intervention framework of ATLAS, an artificial socially intelligent agent that advises teams. The framework treats interventions as atomic components, and manages the lifecycle of each intervention through presentation, as well as followups to interventions. The key benefit of this framework is that it allows for rapid development of scenario-specific Interventions that leverage scenario-agnostic team models. The implementation of this framework is reported for three player teams in a Search and Rescue task simulated in Minecraft. Low competence teams advised by ATLAS improved more between first and second trials than those with a human advisor while the reverse was found for high competence. Four times as many interventions were proposed as were presented. 15% of advice was withheld to avoid repetitive advice, excessive rate of advice, and needlessly advising high performing teams, while a Theory of Mind model and delay for confirmation mechanism filtered out other unnecessary advice.

Keywords: Human-Centered Transportation, Human Factors, Human-Computer Interaction
Abstract: Lack of transparency has been demonstrated to be a stumbling block in building Air Traffic Controller (ATCO) trust towards intelligent decision aids. To address this issue, a runway exit prediction decision aid, with explainability, was developed with the trait of providing explanations involving the top three contributing features to its predictions. To evaluate the influence of the intelligent decision aid’s explanations on ATCO Trust, the decision aid was used in a human-in-the-loop study during off-nominal runway exits, utilizing 12 participants and a total of 67 trials. A multi modal approach was adopted with three types of data (questionnaire, behavioural, physiological) being collected in this study to ensure a more comprehensive understanding of the effects of explainability on ATCO trust. The results indicated that higher levels of perceived transparency led to an increase in trust levels, with an accompanied increase in cognitive load and complacency, even with low prediction accuracy by the intelligent decision aid. As such, these effects must be accounted for in designing XAI decision aids, which are defined as decision aids that rationalize their recommendations, for off-nominal events, when attempting to enhance trust levels by increasing transparency.

Keywords: Human-Centered Transportation, Human Factors
Abstract: The application of self-driving technology to personal mobility vehicles (PMVs) is being investigated. Passengers of self-driving PMVs will likely be engaged in activities with visual processing, such as operating smartphones. Thus, self-driving PMVs need to consider the possibility that passengers will perform these activities and the ease of performing these activities. However, little research has considered such passengers’ convenience when using self-driving PMVs as a service. It is known that visual information is particularly dominant among input information to human beings. As was observed in previous works, changes in the surrounding environment due to moving may negatively impact the passenger’s main activities with visual information processing. The traveling speed affects these changes in the surrounding, and it is a parameter of self-driving methods. Therefore, this paper aims to grasp the effect of traveling speed on passengers’ visual processing during self-driving of PMVs to realize self-driving PMVs on which passengers can easily process information. The effect of the traveling speed of self-driving PMV on task performance and mental workload of passengers’ visual processing was investigated by conducting a simulator experiment. The results showed that although the task performance is maintained, the mental workload of visual processing increases when the PMV moves compared to the stopping condition. Moreover, it was suggested that the visual input of the changing environment and the passenger’s gaze behavior influence the increase in mental workload.

Keywords: Haptic Systems, Human-Machine Interface, Human Factors
Abstract: Teleoperation makes it possible to work in inaccessible environment. To do so safely and effectively, some means of force feedback is necessary. However, force feedback methods that involve a reaction force often require bulky equipment. Instead, our study has utilized force feedback using vibrotactile stimulation. Although the modulation method of force information to vibrotactile stimulation has previously been demonstrated in magnitude (AM) and frequency (FM), we propose a method called AFM that uses amplitude and frequency modulation simultaneously to improve performance. In particular, we adjusted the amplitude using the sensation level and used logarithmized frequency for the AFM. First, the vibrotactile detection threshold of participants was measured to determine the amplitude to be equivalent in the same sensation level among participants. Then, the performance of discrimination was evaluated through stair-case psychophysical experiments. The results showed that our proposal AFM method significantly improved the discrimination sensitivity for the force information as compared with AM and FM methods.

Keywords: User Interface Design, Human-Machine Interface, Human-Computer Interaction
Abstract: The graphical user interfaces (GUIs) for operating heavy machinery, such as cranes, vary significantly even for the same type of machines depending on machine manufacturers or third-party suppliers who develop the GUIs. This situation leads to diverse GUIs, which require operators to train themselves every time they use GUIs from different machine manufacturers or third-party suppliers. Using significantly different GUIs may also increase the risk of human error, since the GUIs may have different rules or mechanisms that operators should follow. To improve the design consistency across different machine manufacturers and third-party suppliers, there is a need for a design system that crane manufacturers and third-party suppliers can use when developing their own GUIs. This paper presents the process of designing user interface elements for operating remote rubber-tired gantry (RTG) cranes, which will be offered as part of OpenCrane Design System.

Keywords: Human-Machine Interaction, Human-Machine Interface, User Interface Design
Abstract: We present a novel method for using embedded optoelectronics in existing laminated printed circuit board (PCB) structures to achieve human interface as well as rudimentary sensing of orientation, obstruction, and failure in robotic systems using fiberglass PCBs as foundational structural elements. The ability to use these existing fiberglass structures for dimensionally stable positioning of electronics, mechanical support to broader robotic elements, as well as visualization and sensing, presents an expanded opportunity for PCB utilization in robotics without additional system cost or mass. We demonstrate using a stacked denoising autoencoder (SDAE) with the capacity to use a low feature density infrared emitter and receiver arrays implemented with embedded optoelectronic elements to detect small deviations in the position of human-actuated mechanical elements incorporated into robotic systems.

Keywords: Human-Machine Interaction, Telepresence, Human-Machine Interface
Abstract: Intuitive robotic tele-operation is a necessary but often difficult task for users because the structure of a robot is different from that of a human body. Our previously developed novel interface allows a user to operate a robot intuitively because the user manipulates this interface according to the Cartesian coordinate system. However, interference between the axes in the interface causes unintended input by the user and reduces operability. In this study, we develop a model that identifies whether each input is the user’s intended or unintended input. We obtained the displacement and angle of the interface’s grip in each direction and investigated the interference relationships between axes. As a result, the large interferences were in the pairs of pitch and Z-axis, and yaw and Y-axis. Finally, we constructed an identification model using a recurrent neural network. The models for all axes achieved F1 scores above 0.97.

Keywords: Human-Machine Interface, Human-Machine Interaction, Human-Computer Interaction
Abstract: Action Recognition in videos is a topic of interest in the area of computer vision, due to potential applications such as multimedia indexing and surveillance in public areas. In this research, we first propose spatial and temporal Convolutional Neural Network (CNNs), based on transfer learning using ResNet101, GoogleNet and VGG16, for undertaking human action recognition. Besides that, hybrid networks such as CNN-Recurrent Neural Network (RNN) models are also exploited as encoder-decoder architectures for video action classification. In particular, different types of RNNs such as Long Short-Term Memory (LSTM), Bidirectional-LSTM (BiLSTM), Gated Recurrent Unit (GRU), and Bidirectional-GRU (BiGRU), are exploited as the decoders for action recognition. To further enhance performance, diverse aggregation networks of CNN and CNN-RNN models are implemented. Specifically, an Average Fusion method is used to integrate spatial and temporal CNNs trained on images, as well as CNN-RNN trained on videos, where the final classification is formed by combining Softmax scores of these models via a late fusion. A total of 22 models (1 motion CNN, 3 spatial CNNs, 12 CNN-RNNs and 6 fusion networks) are implemented which are evaluated using UCF11, UCF50, and UCF101 datasets for performance comparison. The empirical results indicate the significant efficiency of Average Fusion of multiple Spatial-CNNs with one Motion-CNN, and ResNet101-BiGRU, among all the networks for undertaking realistic video action recognition.

Keywords: Human-Machine Interface, Medical Informatics, Wearable Computing
Abstract: Given the recent surge of significant interest in implementing intelligent hand gesture recognition methods in human-machine interface systems, a wide variety of Deep Neural Networks (DNNs) have been proposed in the literature. In this paper, we introduce a novel and compact Spiking Neural Network (SNN) model for hand gesture recognition using High-Density surface Electromyogram (HD-sEMG) signals. Capitalizing on their ability to extract spatiotemporal features of HD-sEMG signals along with their proven strength in imitating human brain's neural activity using event-driven data processing, we used SNNs as the main building block of our proposed hand gesture recognition model. We show that our proposed model can efficiently differentiate 14 hand movements by considering each sample of the HD-sEMG data as a single time step for the SNN architecture. Moreover, we show that the proposed SNN model does not require huge pre-processing, spike encoding and feature extraction tasks and works effectively on Min-Max normalized continuous-value sEMG signals. We evaluate our SNN model using a 5-fold cross-validation scheme and categorize different participants based on the range of classification accuracy we obtained for them. The following results are acquired by segmenting HD-sEMG signals into windows of size 62.5ms with no overlap. The proposed method led to 6 out of 19 subjects achieving average classification accuracy of greater than 80% with maximum accuracy of 98% associated with 3rd session of the sEMG dataset as the test set.

Keywords: Human-Machine Cooperation and Systems, Networking and Decision-Making, Human-Machine Interaction
Abstract: In this paper, we propose a novel personalized decision support system that combines Theory of Mind (ToM) modeling and explainable Reinforcement Learning (XRL) to provide effective and interpretable interventions. Our method leverages DRL to provide expert action recommendations while incorporating ToM modeling to understand users' mental states and predict their future actions, enabling appropriate timing for intervention. To explain interventions, we use counterfactual explanations based on RL's feature importance and users' ToM model structure. Our proposed system generates accurate and personalized interventions that are easily interpretable by end-users. We demonstrate the effectiveness of our approach through a series of crowd-sourcing experiments in a simulated team decision-making task, where our system outperforms control baselines in terms of task performance. Our proposed approach is agnostic to task environment and RL model structure, therefore has the potential to be generalized to a wide range of applications.

Keywords: Human-Machine Cooperation and Systems, Virtual and Augmented Reality Systems, Human Factors
Abstract: To help enhance mission effectiveness for future dismounted soldiers in the Canadian Armed Forces (CAF), a Soldier-Robot Teaming (SRT) concept has been developed and accepted as a force multiplier to extend operational abilities of dismounted soldiers in the battlefield, such as reducing the number of soldiers in dangerous environments and empowering them with advanced robot technologies. To support this endeavour, the Defence Research and Development Canada (DRDC) Toronto Research Centre (TRC) has led a research effort `Concept of Operations (CONOPS) for SRT in the CAF' since 2019. In the first research phase, key stakeholders within the Department of National Defence (DND) and CAF were engaged in a series of meetings and interviews to help identify future SRT concepts across a broad range of missions and operational environments. During the stakeholder analysis, five SRT use cases were developed and validated for the CAF to capture the intended SRT CONOPS, operational priorities, operational contexts, functionality, interactions, and expected mission performance, and to support Land Operations at section and platoon levels. To further facilitate SRT concept development and experimentation activities at DRDC TRC, a synthetic modeling and simulation environment was proposed and developed, in support of future SRT effectiveness research on team communication, coordination, collaboration and trust. In the meantime, this effort can also support studies on human-machine interface and human-systems integration, as well as help carry out SRT human-in-the-loop experiments and trials, for the CAF needs of reducing soldier workload and improving team performance and effectiveness in future SRT operations.

Keywords: Human-Machine Interaction, Human-Centered Transportation, Human Factors
Abstract: In the automotive industry, contact and surface heating systems are designed to enhance thermal comfort in vehicles. The maximum temperature thresholds for such systems are integrated in DIN EN ISO 13732-1. This paper presents a study on the effects of contact surface heating on thermal comfort in automotive interiors, with a particular focus on individuals with heat sensitivity impairments caused by polyneuropathy. Two groups of individuals, one with normal sensitivity and the other with impaired heat sensitivity due to polyneuropathy, were tested for their thermal perception and response to an overwarmed seat and steering wheel heating system. The study aimed to measure the temperature and time at which individuals in a Control and Experimental group would stop using an overwarmed contact surface heating on the steering wheel and seat. In addition, the thermal load on passengers in a vehicle environment was investigated to define time limits and temperature thresholds for comfortable and uncomfortable states when using contact surface heating systems. A BMW driving simulator was used to simulate a realistic in-vehicle experience. The results indicate that both groups could feel the heat but had significant differences in their threshold temperature and time to discontinue heating. The heat sensitivity impaired group had a more sensitive thermal perception compared to the normal sensitive group. The outcomes of this research offer valuable insights for the design and advancement of contact and surface heating systems in vehicles, specifically for individuals who suffer from heat sensitivity impairments.

Keywords: Intelligence Interaction, Human-centered Learning, Interactive Design Science and Engineering
Abstract: Companies increasingly need to know the skills and competences they have developed to cope with market skills and to tackle projects. Yearly evaluations cannot put on operational and technical competences developed by actors during their work. In fact, competence generally refers to the experiences, knowledge, attitudes, abilities and behavior that enable effective action in a work environment. In this paper, an interaction analysis is proposed in order to emphasize the manifestation of actors’ competences. This approach is mainly based on action verbs and interaction linguistics studies.

Keywords: Team Performance and Training Systems, Multi-User Interaction, Interactive Design Science and Engineering
Abstract: Understanding the dynamic nature of emergent team properties such as collaboration and role differentiation could inform intervention methods that aim to improve team outcomes. Often such emergent properties are studied in heavily controlled experiments that constrain how these properties develop. In this study, we use a novel paradigm for which to study emergent team properties in a dyadic team setting. Participants are granted extensive autonomy to determine their strategy as they navigate an immersive environment to pursue their objective of locating and defusing improvised explosive devices. We use Granger causality methods to identify when a participant’s behaviors are predictive of their teammate’s behaviors, which we refer to in this study as Granger leadership. Participants that did the most leading in their team were identified as leaders while their teammates were identified as followers. Results indicated better task performance for teams that more evenly shared leadership responsibilities. We also found that gaze fixation rates increased and blink rates decreased with increasing Granger leadership values. This indicates that Granger leadership may be a surrogate for task engagement, and that interventions aimed at improving team performance should attempt to encourage engagement of the followers within a group.

Keywords: Affective Computing, Human-Collaborative Robotics, Human-Machine Interaction
Abstract: Many attempts have been made at estimating discrete emotions (calmness, anxiety, boredom, surprise, anger) and continuous emotional measures commonly used in psychology, namely `valence' (The pleasantness of the emotion being displayed) and `arousal' (The intensity of the emotion being displayed). Existing methods to estimate arousal and valence rely on learning from data sets, where an expert annotator labels every image frame. Access to an expert annotator is not always possible, and the annotation can also be tedious. Hence it is more practical to obtain self-reported arousal and valence values directly from the human in a real-time Human-Robot collaborative setting. Hence this paper provides an emotion data set (HRI-AVC) obtained while conducting a human-robot interaction (HRI) task. The self-reported pair of labels in this data set is associated with a set of image frames. This paper also proposes a spatial and temporal attention-based network to estimate arousal and valence from this set of image frames. The results show that an attention-based network can estimate valence and arousal on the HRI-AVC data set even when Arousal and Valence values are unavailable per frame.

Keywords: Human-Collaborative Robotics, Human-Machine Interaction, Human-Computer Interaction
Abstract: We introduce three new multi-modal data sets. They contain physiological and/or emotional information about human interactions with robotic arms in proximity to completing a task in an industrial setting. The data sets provide data from human subjects engaged in the assistive task of assembling a PVC joint pipe with robots. These data streams were collected to analyze and improve the comfort and safety of humans collaborating with robots in proximity in an industrial setting. These data sets can appeal to researchers studying human-robot collaboration, robot adaptation, and affective computing. Our data is stored in various formats, including images and human-readable Comma-Separated Values (CSV) or JavaScript Object Notation (JSON) files.

Keywords: Brain-Computer Interfaces
Abstract: Brain-computer interface (BCI) technology has attracted a lot of attention in recent years. Emotion recognition which based on electroencephalography is a typical application of BCI. Traditional methods on emotion recognition are mainly focusing on time domain feature and frequency domain feature while spatial information is often been ignored. In this paper, to make use of spatial feature, we propose a new convolutional neural network using not only temporal feature but also person related feature and event related feature. Depthwise convolution and separable convolution are also used for feature extraction. To verify the effectiveness of our method, we conduct extensive experiments on the public dataset DEAP and DREAMER. Compared with other methods, our method has achieved the state-of-the-art effect.

Keywords: Haptic Systems
Abstract: In recent years, the need for guide dogs has increased, and a more efficient methodology for training guide dog trainers is accordingly required. One challenge is that haptic information, which is a significant part of guide-dog training, is difficult to explain and visualize. Virtual reality (VR) systems have attracted attention owing to their ability to support high-level immersive interactions with the presence of multisensory experiences. A haptic-enabled VR system could address the limitations of the conventional method and support novice trainers in practicing in an immersive and remote transmission manner. This study presents a handle-based sensing setup for recording haptic experiences of trainers during guide dog training for use in a VR system. Considering trainers perceive and apply forces via handle to obtain and control dog motion status, the applied forces are decoded as haptic information and recorded using corresponding sensors. While accuracy of proposed setup for collecting required data is checked to be high(average errors of validating forces and yaw angles are 0.4N and 0.93° respectively), we believe that the proposed recording system is able to measure haptic experiences for further haptic experience re-establishments in proposed VR guide-dog training system and could finally facilitate the education of a large number of dog trainers.

Keywords: Image Processing and Pattern Recognition, Fuzzy Systems and their applications, Machine Learning
Abstract: Medical image retrieval is a highly discussed topic, and it includes an efficient classification of diagnoses based on the similarity search in large databases of medical images. It is very important for early and correct diagnosis and treatment. In this paper, we focus on detecting four diagnoses of treatable retinal diseases in optical coherence tomography (OCT) images. The fuzzy medical image retrieval model (FMIR) is applied to transfer images to fuzzy signatures organized in Fuzzy S-tree, as it was previously successfully used for breast cancer detection and COVID-19 chest X-ray detection. The paper examines and compares the performance of the FMIR method on 4-class and binary classification models built on an OCT dataset and compares the impact of two metrics, Euclidean and Hamming fuzzy distances. The experiments show a clear dominance of Hamming fuzzy distance. The best accuracy is achieved for binary classification (61.16 - 93.8%), while the performance of the 4-class model is worse (51.7%). The distribution of signature space and classification performance are analyzed in detail.

Keywords: Image Processing and Pattern Recognition, Transfer Learning, Neural Networks and their Applications
Abstract: Edge Detectors are one of the most fundamental tools used in Computer Vision applications. Using deep learning, state-of-the-art (SOTA) models are able to produce sharp, fine edges, mirroring human-level performance. However, most of these SOTA models are trained solely on color images. We experimentally determined that this skewed data severely hinders performance on images from other domain spaces. In this paper, we propose a way to adapt SOTA models to single-channel inputs from other domains with minimal overhead, using small-scale feed-forward networks. Our model, which serves as an additional layer to existing SOTA models, greatly boosts their performance on out-of-domain data, allowing the edge detector to generalize to new domain spaces without undergoing extensive, resource- or data-heavy retraining. Using a single off-the-shelf GPU and a small 30-image dataset, we were able to train this low-complexity (less than 13k parameters) model in half an hour to boost the F-score of the produced edgemaps for chest x-rays by over 0.2.

Keywords: Machine Vision, Deep Learning, Machine Learning
Abstract: We have developed a CT guided needle puncture robot (Zerobot) to assist in interventional radiology surgery. Currently, Zerobot is operated remotely, and the next goal is to perform automatic needle puncture surgery. There is a challenge that automatic detection of puncture needle from CT images for first step of automatic puncture surgery. Because there is the case that the form of puncture needle is curved, it is necessary to detect the needle from CT image instead of estimating the needle position from arm of Zerobot. First, the method detects ROI with ResNet. Next, difference of the CT value is calculated for each pixel in the ROI, and a linear approximation is performed for to detect the needle shape. Images from animal experiments were used to evaluate the learner and image processing. We confirmed that the proposed method can detect needles in a single image and in multiple images.

Keywords: Representation Learning, Neural Networks and their Applications, Complex Network
Abstract: Many natural and engineering systems can be modeled and represented in the forms of graph data, and then studied using graph theory and network analysis tools. Graph representation learning aims at generating lower-dimensional representations from higher-dimensional graph data, which is a crucial step that facilitates the follow-up tasks, such as node and graph classifications. In this paper, we present a simple but effective graph representation learning method, namely the pyramid pooling-based local profile (PPLP), which enables local nodal profiles to be transformed into a graph representation, with multi-scale features extracted. PPLP can be either embedded into a graph neural network as the readout layer, or perform independently as a graph embedding algorithm. The resultant representations of PPLP are for graph-level tasks. PPLP is experimentally tested by performing graph classification tasks on ten representative datasets, either as the readout layer of different graph neural networks, or as an independent graph embedding algorithm. Experimental results demonstrate that: 1) when embedded into graph neural networks, PPLP outperforms the widely-used global pooling-based readout methods; 2) as an independent graph embedding algorithm, PPLP performs fairly good, especially on the social network datasets. The investigation confirms PPLP as a simple but promising method for graph-level tasks.

Keywords: Neural Networks and their Applications, Machine Learning, Computational Life Science
Abstract: Multifunctionality describes the capacity for a neural network to perform multiple mutually exclusive tasks without altering its network connections; and is an emerging area of interest in the reservoir computing machine learning paradigm. Multifunctionality has been observed in the brains of humans and other animals: particularly, in the lateral horn of the fruit fly. In this work, we transplant the connectome of the fruit fly lateral horn to a reservoir computer (RC), and investigate the extent to which this 'fruit fly RC' (FFRC) exhibits multifunctionality using the 'seeing double' problem as a benchmark test. We furthermore explore the dynamics of how this FFRC achieves multifunctionality while varying the network's spectral radius. Compared to the widely-used Erdös-Renyi Reservoir Computer (ERRC), we report that the FFRC exhibits a greater capacity for multifunctionality; is multifunctional across a broader hyperparameter range; and solves the seeing double problem far beyond the previously observed spectral radius limit, wherein the ERRC's dynamics become chaotic.

Keywords: Neural Networks and their Applications, Biometric Systems and Bioinformatics, Application of Artificial Intelligence
Abstract: Semantic segmentation of bone structures requires pixel-wise classification and high accuracy to build a precise bone model capable of diagnoses. Various segmentation methods have been developed, and state-of-the-art models, such as DeepLabv3+, are generally based on a Convolutional Neuronal Network (CNN). A CNN requires many images for training, and high performance is not expected when few training images are available. In general, it is costly to obtain training images, known as ground truth for bone segmentation, because ground truth images are usually handcrafted by experts. Considering the lack of training images, we aim to develop a robust method in an environment with few images. U-Net is a well-known CNN and is often adopted for medical image segmentation because it can train with very few images compared to other models, thanks to its U-shaped encoder-decoder architecture. However, it is reported that U-Net cannot capture complex shapes in spine images. In this paper, we propose combing the advantages of both U-Net and DeepLabv3+. Specifically, we incorporate atrous spatial pyramid pooling (ASPP), which is used in DeepLabv3+, into U-Net to adopt varying-sized convolutions to capture precise bone structure. Because applying ASPP to all convolution layers of U-Net would be computationally expensive, it was applied after the encoder as the first step. It is known that the last convolution layer before the fully connected layer expresses visual features in a classification model; therefore, we hypothesized that the layer after the encoder would provide more inherent specific features than other layers in U-Net. Moreover, the Gaussian conditional random field (CRF) was utilized to refine the output semantic segmentation map. We evaluated the proposed model on spine and femur image datasets from Cancer Imaging Archive. We confirmed that the proposed method outperforms U-Net, and observed that both ASPP and CRF are effective for the performance improvement of U-Net.

Keywords: Neural Networks and their Applications, Evolutionary Computation
Abstract: This paper studies a multiobjective optimization problem in continuous-time recurrent neural networks. For simplicity, we use a simple recurrent neural network: a hysteresis associative memory characterized by a binary hysteresis activation function and ternary cross-connection parameters. The optimization problem is based on two objectives. The first objective evaluates the memory accuracy and the second objective evaluates connection sparsity. In order to analyze the optimization problem, we present a simple evolutionary algorithm with growing connection structure. Applying the algorithm to typical examples, we have obtained a Pareto front that guarantees existence of a trade-off between the two objectives. The trade-off provides basic information for the system dynamics and becomes a criterion to optimize the system parameters.

Keywords: Neural Networks and their Applications, Swarm Intelligence, Deep Learning
Abstract: Over the last few decades, significant attention has been devoted to the coordinated motion of swarm systems, which involve multiple autonomous robots. These systems have a range of potential applications, such as creating formations with physical or non-physical bonds, navigating to desired locations while maintaining formation, and preventing collisions. To achieve coordinated motion, recent researches have focused on the potential function method. We previously validated novel potential functions and developed a sliding-mode speed controllers to promote collective behavior. In this study, we implement an accepted neural network with one hidden and one output layer to update potential function weights with decentralized manner in real-time, thereby enhancing swarm behavior for formation creation and maintenance. The effectiveness of this approach is demonstrated through simulation and processor-in-the-loop tests using two different micro-controllers as well as measuring execution time of neural networks’ models. By successfully validating neural networks implementation in this context, our study extends the current understanding of swarm behavior and opens up new possibilities for real-time swarm applications.

Keywords: Intelligent Internet Systems, Cloud, IoT, and Robotics Integration, Neural Networks and their Applications
Abstract: Current interactions of network traffic through cloud data centers have become an important process of network services. Precise and real-time detection and prediction of network traffic can assist system operators in effectively allocating resources, and assessing network performance based on actual service requirements, and analyzing network health. However, sources and distribution of network traffic are different, which makes accurate warnings of network attack traffic become a difficult problem. In recent years, neural networks have been proven to be effective in predicting time series data, particularly long short-term memory networks for capturing temporal features and convolutional methods for capturing spatial features. This work proposes a Deep Hybrid Spatio-Temporal (DHST) network method for abnormal traffic detection in cloud data centers, which combines a cooperative temporal convolutional network, an attention mechanism and a random inactivation method to capture the network traffic data’s spatio-temporal features. It improves accuracy of abnormal traffic detection, and realizes classification of normal traffic and abnormal one. It achieves higher accuracy than typical detection methods when applied to a real-life dataset collected from Yahoo Webscope S5.

Keywords: Evolutionary Computation, Swarm Intelligence, Intelligent Internet Systems
Abstract: Current mobile edge computing (MEC) owns cloud resources at the network edge, which enables low-latency mobile services. In addition to fixed MEC servers, MEC proxy servers with certain mobility and limited computing, e.g., flying unmanned aerial vehicles (UAVs), and vehicles, have emerged as competitors in providing services. In this work, aiming at a task offloading problem of a UAV-assisted MEC system, a hybrid network environment with multiple mobile devices (MDs) and multiple UAVs is established. A constrained mixed integer nonlinear program of the UAV-assisted hybrid cloud-edge system is formulated. A novel hybrid metaheuristic algorithm called Genetic Simulated annealing-based Particle Swarm Optimization (GSPSO) is presented to solve the program. Then, a task offloading and resource scheduling method is designed to intelligently minimize the total energy consumption of the hybrid system. Simulation results verify superiority of GSPSO over its three benchmark algorithms, thus demonstrating the proposed method significantly improves the energy efficiency of the UAV-enabled hybrid system.

Keywords: Cloud, IoT, and Robotics Integration, Swarm Intelligence, Evolutionary Computation
Abstract: Edge computing is a new architectural model that aims to offer computing, storage, and networking resources to support Internet of Things. Its primary strategy involves transferring computational tasks to the edge of network, which is closer to end-users. This paradigm facilitates offloading of computation, resulting in reduced latency and improved system performance. However, nodes located at the network edge have restricted energy and resources. As a result, running tasks entirely at the edge leads to higher energy consumption. This work proposes a novel three-tier offloading framework comprising of multiple mobile vehicles (MVs), a base station (BS), and a cloud data center (CDC). It jointly optimizes offloading rates of tasks, CPU computation rates of MVs, BS, and CDC, and the allocation of wireless bandwidth resources at MVs during partial computation offloading of tasks. It also considers limits of maximum computational resources and maximum delay of task execution. To further reduce the total system energy consumption, this work actively caches execution codes of tasks in MEC servers to reduce data transmission energy of MVs, which minimizes the total system energy consumption. This work develops a mixed integer nonlinear program and designs a mixed metaheuristic algorithm with a multi-strategy adaptive particle swarm optimizer. Simulation results demonstrate that it outperforms various state-of-the-art algorithms by achieving lower energy consumption in fewer iterations.

Keywords: Intelligent Internet Systems, Metaheuristic Algorithms, Evolutionary Computation
Abstract: Fog computing provides lower latency and higher bandwidth compared to cloud computing and is widely used in Internet of Vehicles (IoV). Vehicles cannot compute all tasks locally due to their limited computing power and battery capacity. Thus, it is a useful way to offload some tasks of vehicles to other resource-rich servers. However, due to the high mobility of vehicles, there may be a failure of returning computing results. Thus, it is a challenge to minimize the latency of tasks while meeting the constraint of energy consumption. Thus, this work proposes a vehicle-fog offloading system that offloads tasks to fog servers or idle vehicles. This work proposes an improved optimization algorithm called an adaptive L ́evy flight-based Whale optimization algorithm with Hierarchical learning (LWH) to solve this problem. Simulation experiments show that LWH has a strong global search capability and outperforms its five typical and widely used algorithms.

Keywords: Intelligent Internet Systems, Neural Networks and their Applications, Deep Learning
Abstract: Nowadays, rapid development of Internet has brought a sharp increase in traffic data. Abnormal traffic haS serious impact on network security. Traffic anomaly detection can be achieved by extracting characteristics of network traffic to detect anomalous intrusions, and therefore, anomaly detection algorithms are of great significance to maintenance of network security. This work proposes a hybrid spatio-temporal neural network with attention named CTGA to effectively identify anomalous traffic. CTGA combines a Convolutional neural network (CNN), a Temporal convolutional network (TCN), a bidirectional Gated recurrent unit network (BiGRU), and a self- Attention mechanism. It automatically extracts temporal and spatial features of sequences from raw data by sliding window preprocessing followed by CNN, TCN, BiGRU, and the selfattention mechanism to detect anomalous data. CNN is used to extract spatial features of time sequences and reduce the loss of spatial information. In the sequence, TCN obtains shortterm features. Long-term dependencies in the data are captured by BiGRU, and the self-attention mechanism obtains important information in the sequence. Finally, experiments with the reallife Yahoo S5 dataset prove that CTGA outperforms other approaches substantially.

Keywords: Cloud, IoT, and Robotics Integration
Abstract: Mobile edge computing boosts the application performance on mobile devices by collaborating with cloud platforms. This paper studies the task offloading and computing resource allocation problem in a multibase, multiserver, and multiuser scenario subject to resource constraints. The goal is to maximize the users’ task offloading utility, including improvements in task completion time, energy consumption, and communication cost. The addressed problem is formulated as a mixed integer nonlinear programming (MINLP) model. In this paper, we decompose the MINLP and the optimal computing resource allocation policy under a deterministic offloading strategy obtained by the Karush-Kuhn-Tucker conditions. Then, a hybrid adaptive large neighborhood search (HALNS) algorithm is proposed to conduct task offloading. The adaptive large neighborhood search and the variable neighborhood de-scent stages are jointly employed in HALNS. The proposed algorithm, an improved simulated annealing algorithm, and a modified variable neighborhood search algorithm are executed to evaluate their performances. Digital experimental results show that our proposed algorithm achieves higher system utility, lower delays, and less energy consumption.

Keywords: Heuristic Algorithms, Hybrid Models of Computational Intelligence, Optimization and Self-Organization Approaches
Abstract: By using an autoencoder as a dimension reduction tool, an Autoencoder-embedded Teaching-Learning Based Optimization (ATLBO) has been proved to be effective in solving high-dimensional computationally expensive problems through several widely used function problems. However, the following two crucial issues have not been resolved, 1) ATLBO should be verified by solving real-life optimization problems; and 2) how autoencoder parameters and structures impact AEO’s performance. In this work, ATLBO is verified by an energy consumption minimization problem (ECM) in mobile edge computing systems. To design an effective autoencoder for ATLBO, this work proposes a parameter tuning optimization strategy for autoencoders. By using the proposed Autoencoder Parameter Tuning (APT) strategy, ATLBO can enjoy higher robustness than those without it. The experimental results show that it is three to six times better than state-of-the-art methods in solving ECM. We consider the strategy-induced overhead and take the execution time as the primary criterion to evaluate them. In addition, the experimental results show that, against the conventional wisdom that higher-accuracy autoencoders bring higher system performance, lower-accuracy ones can actually assist ATLBO in locating the best solutions. This work promotes a novel application of autoencoders in optimization theory and practice.

Keywords: Deep Learning, Neural Networks and their Applications, Expert and Knowledge-Based Systems
Abstract: Digital relays are critical components of smart power grids, therefore, their security is paramount for the proper operation of the grid. This paper proposes a cyber-attack detection method for digital relays using a modified generative adversarial network combined with the extra tree classifier for dimensionality reduction. The proposed method is evaluated on a IEEE 39-bus transmission network. The results shows that the proposed method can detect false data injection attacks and civil-attack with more than 97 percent of accuracy, f1 score, and more than 96 percent of sensitivity.

Keywords: Deep Learning, Image Processing and Pattern Recognition
Abstract: Emotion detection has become an increasingly important area of research in recent years, as it has numerous applications in fields such as psychology, marketing, and human computer interaction. Deep learning has shown success in emotion recognition due to the availability of large amounts of data and the ability to learn complex patterns in facial expressions, speech, and physiological signals when recognizing emotions. However, there are challenges associated with variations in lighting, pose, and facial expressions. This study introduces a novel deep-learning approach for emotion detection, leveraging the power of VGGFace2 to classify six emotional poses in children. The proposed approach outperforms the state-of-the-art in the field, achieving a success rate of 96.3% on the CAFE dataset. A comprehensive evaluation of the findings and a detailed discussion of their potential implications is offered as part of the study.

Keywords: Deep Learning, Machine Learning, Image Processing and Pattern Recognition
Abstract: Deep Neural Networks (DNNs) are dominant in the field of machine learning. However, because DNN models have large computational complexity, implementation of DNN models on resource-limited equipment is challenging. Therefore, techniques of compressing DNN models without degrading their accuracy is desired. Pruning is one such technique that removes unimportant neurons (or channels). In this paper, we present Pruning with Output Error Minimization (POEM), a method that performs not only pruning but also reconstruction to compensate the error caused by pruning. The strength of POEM lies in its consistent criteria for neuron selection and reconstruction based on the output error of the activation function, whereas the previous methods use ad hoc criteria or relax the problem to minimize the error before the activation function. The experiments with well-known DNN models (VGG-16, ResNet-18, MobileNet) and image recognition datasets (ImageNet, CUB-200-2011) were conducted. The results show that POEM significantly outperformed the previous methods in maintaining the accuracy of the compressed models.

Keywords: Deep Learning, Machine Vision, Representation Learning
Abstract: Compared with grid-based 2D images, processing 3D point clouds is more challenging due to their irregular distribution and intricate spatial information. Most prior works introduce delicate designs on either local feature aggregators or global geometric architecture, but few combine two scales effectively. Therefore, to better incorporate the advantages of both local and global processing, we propose DS-Point, a dual-scale 3D framework for point cloud understanding. DS-Point firstly disentangles 3D features from channel dimension for concurrent dual-scale modeling, i.e., point-wise convolution for local fine-grained geometry parsing, and voxel-wise attention for global long-range spatial exploration. Upon that, an HF-fusion module is proposed to enhance the cross-modal interaction and thoroughly blend the dual-scale features. Then, with task-specific heads for different downstream tasks, DS-Point serves as an effective 3D framework for feature extraction. By the dual-scale paradigm, DS-Point achieves superior performance on multiple downstream tasks, e.g., 93.8% for shape classification on ModelNet40, 84.9% on ScanObjectNN, and 84.3% on ShapeNetPart.

Keywords: Metaheuristic Algorithms, Intelligent Internet Systems, Cloud, IoT, and Robotics Integration
Abstract: Nowadays, smart mobile devices (SMDs) support various computation-intensive and delay-sensitive applications, e.g., online games, and figure compression. However, SMDs have limited computing resources and battery energy and cannot execute all tasks of the above applications in a real-time manner. Cloud computing provides enormous computing resources and energy that can easily execute tasks offloaded from SMDs. However, could data centers (CDCs) are often located in remote sites, which leads to long transmission time. Small base stations (SBSs) offer high-bandwidth and low-latency services for SMDs, which solves the problem of cloud computing. However, it becomes a challenge to achieve the lowest cost in such a heterogeneous architecture including multiple SMDs, SBSs, and the CDC while meeting delay requirements of tasks. This work proposes a cost-minimized computation offloading strategy to minimize the total cost of the system. A constrained optimization problem is first formulated based on the hybrid architecture. Afterward, a two-stage optimization algorithm called a Levy flights and Simulated Annealing-based Grey wolf optimizer (LSAG) is developed to optimize the total cost of the system. In the first stage, the optimal edge selection policy is determined given multiple available SBSs. In the second stage, task offloading and resource allocation among SMDs, SBSs, and the cloud are determined in the second stage. LSAG integrates L´evy flights and simulated annealing into the grey wolf optimizer to improve the global exploration ability and avoid early trapping into local optima. Experiments with real-life tasks prove that LSAG significantly achieves lower cost with faster convergence speed than state-of-the-art peers.

Keywords: Metaheuristic Algorithms
Abstract: As systems become larger and more complex, real-world problems, such as system operation, require that quasi-optimal solutions with sufficient engineering optimality be obtained in practical time. Meta-heuristics have attracted attention as a framework for methods that seek quasi-optimal solutions. We focus on the issue of degeneracy in the Combinatorial optimization method with search strategy based on hierarchical interpretation of solution space, which has high performance and versatility compared to existing basic methods. The degeneracy is a phenomenon in which multiple search points follow the same path, which has negative effects such as wasting computational resources and weakening the interaction between search points. A new stochastic process is introduced with the main objective of improving search performance by dealing with the degeneracy. The occurrence of the degeneracy and search performance are compared and verified with the original method. We use the basic benchmark problems: Knapsack Problem (KP), Traveling Salesman Problem (TSP), Flow-shop Scheduling Problem (FSP), and Quadratic Assignment Problem (QAP).

Keywords: Metaheuristic Algorithms
Abstract: Differential Evolution (DE) is one of the effective metaheuristics for solving unconstrained optimization problems. Constraint Handling Technique (CHT) is needed to extend DE to constrained optimization. Feasibility Rule (FR) is one of the typical CHT. FR addresses constraints by using a simple rule that considers the objective function and constraint violation when comparing search individuals. However, since the solutions with small constraint violation, i.e., feasible solutions, are preferentially selected, the set of search individuals may be biased toward feasible regions and the improvement of the objective function value may stagnate. This paper overcomes this challenge by proposing a DE that uses a superior infeasible solution in an external archive. The external archive in the proposed DE stores search individuals that are superior in both objective function value and constraint violation and utilize them to generate mutant individuals. Finally, we verify the effectiveness of the proposed method using a benchmark problem where the feasible region is a convex set.

Keywords: Metaheuristic Algorithms, Heuristic Algorithms
Abstract: The capacitated p-center problem (CpCP) is an extension of the classical p-center problem. It consists of choosing p centers from a set of candidate centers and assigning each client to a center such that the total client demand assigned to each center does not exceed its given capacity. The objective of the CpCP is to minimize the maximum distance between each client and its assigned center. In this paper, we propose a two-stage iterated local search algorithm called TS-ILS to solve the CpCP. The first stage uses a tabu search procedure to select centers and greedily assign clients to centers, while the second stage adopts a variable neighborhood search procedure to perform the fine-grained assignment of clients. Tested on 39 commonly studied instances in the literature, TS-ILS improves the best known results of the state-of-the-art metaheuristic algorithms on 18 instances and matches the records for the remaining ones within less run time.

Keywords: Metaheuristic Algorithms, Evolutionary Computation, Heuristic Algorithms
Abstract: Multi-depot vehicle routing problem (MDVRP) is a variant of the classical VRP, which includes several depots with a fleet of homogeneous vehicles to serve each customer exactly once while satisfying the vehicle capacity and duration constraints. We propose a memetic algorithm called GVTS-DPX which hybridizes the granular variable tabu search (GVTS) with the depot partition crossover (DPX) for solving the MDVRP, where GVTS combines tabu search and the granular neighborhoods with variable neighborhood descent, while DPX treats the solution as the collection of depots and partitions the depots into two groups covering the most customers. The main contributions of this study include proposing the DPX operator, reforming several existing move types used for the VRP and its variants, and designing a granular variable neighborhood consisting of a total of 21 kinds of move types. Experimental results on 33 public MDVRP instances indicate that GVTS-DPX is competitive with the state-of-the-art algorithms in the literature.

Keywords: Metaheuristic Algorithms, Neural Networks and their Applications, Image Processing and Pattern Recognition
Abstract: Deep learning methods have shown state-of-the-art results in various application areas such as computer vision, NLP, etc. However, their practical use presents many challenges, including those caused by the large size of the models, especially in the context of model weight storage and transmission. One of the possible solutions to this problem is Neural Network (NN) compression, which is a process of obtaining a derived model serving the same task with a smaller number of parameters or with parameters of lower precision. The most common NN compression techniques include pruning, sparse representation, quantization, and knowledge transfer. In this article, the compression of Convolutional Neural Networks (CNNs) using fractional differentiation is investigated. A formulation of this task as a large-scale continuous optimization problem is then proposed, and its resolution is performed through a new optimization algorithm, called the Improved Fractal Decomposition Algorithm (IFDA), based on space geometric fractal decomposition. The results obtained show that MobileNetV3, for instance, is compressed by 18.5% with only a 2.5% decrease in accuracy. Additionally, the proposed IFDA algorithm outperforms all other competing metaheuristics in solving this problem.

Keywords: Deep Learning, Neural Networks and their Applications
Abstract: Recently, graph neural networks (GNNs) have become a popular approach to deal with machine learning tasks for graph-structured data. To achieve reliable performance with a GNN-based approach, obtaining high-quality graph structures is crucial. However, the graph data in the real-world often contain noise from data themselves or during the collecting procedure, which leads to the performance degradation of GNNs. In this paper, we propose a novel approach to enhance graph structures for performance improvement of GNNs by reversely applying the concept of adversarial attacks on graph data. Experimental results demonstrate the effectiveness of our method in improving performance of GNNs. Furthermore, we investigate the changes in the graph structure induced by our method, taking into account the connectivity of both inter-class and intra-class edges and measuring the extent of over-smoothing.

Keywords: Deep Learning, Representation Learning, Image Processing and Pattern Recognition
Abstract: To make content-based image retrieval (CBIR) technology for magnetic resonance (MR) images of the brain practical and useful for diagnosis and research, it is important to obtain low-dimensional representations that embody pathological attributes. However, recent evidence suggests that variations in domains resulting from differences in imaging equipment and protocols at each imaging facility can overshadow pathological attributes. In this study, we propose a novel approach known as multidecoder adversarial domain adaptation (MD-ADA) to obtain low-dimensional representations of brain MR images that preserve pathological features while mitigating domain differences. This method combines adversarial domain adaptation techniques with convolutional autoencoders that have distinct decoders for each domain and employs adversarial learning to prevent domain discrimination from the produced low-dimensional representations. Experimental evaluations on two datasets, ADNI and PPMI, comprising 4,168 brain images demonstrate that the proposed MD-ADA significantly reduces domain differences between datasets without compromising the recoverability of brain images or the accuracy of disease classification.

Keywords: Deep Learning, Image Processing and Pattern Recognition, Machine Vision
Abstract: It is difficult to train a trustworthy transformer model on a small image classification dataset. This research proposes a sophisticated structured knowledge distillation algorithm that uses CNNs as Transformer's sophisticated teachers, significantly lowering the number of training datasets needed. To better to develop the potential for CNN tutors, this research configures a public data set for CNN teaching as an enlightenment textbook to guide Transformer’s training and avoid falling into local optimization prematurely. The distillation process then employs a ``learn-digest-self-distillation'' learning strategy to enable the Transformer to assimilate CNN knowledge in a structured manner. Sufficient experiments show that the proposed method is significantly better than the direct training Transformer under the condition of limited data sets. Moreover, in order to show the practical application value, this research contributed a practical data set for the classification of smoking and calling. The corresponding code and dataset will be released at https://gitee.com/wustdch/surpass-teacher if this paper is accepted.

Keywords: Deep Learning, Big Data Computing,, Neural Networks and their Applications
Abstract: Challenges in recent recommender systems include how to model high-order feature interaction and how to exploit user-item interaction, particularly for neural network-based recommender systems. While previous approaches have focused only on one aspect, this paper attempts to address both simultaneously by extracting augmented embeddings for users and items with feature interaction and modeling user-item interaction using graph neural networks. Real-world experimental results show that the proposed method outperforms state-of-the-art methods considering one type of interaction.

Keywords: Neural Networks and their Applications
Abstract: In neuroscience research, in vivo imaging techniques for mice are used to observe brain activity and link it to their behavior. Brain activity can often only be associated with observed behavioral outcomes. In other words, it is difficult to speculate on unmanifested behavior due to factors such as "hesitation" in humans. When a prediction model can predict mice behavior, if brain activity is observed in a specific brain region during incorrect predictions, that would be strong evidence of unmanifest behavior. In this study, we developed a trajectory prediction model to predict the walking trajectory of mice as a prelude to the behavior prediction model. The prediction model was applied to the behavioral analysis of mice administered an anxiolytic drug (diazepam) or saline, revealing significantly different outcomes.

Keywords: Virtual/Augmented/Mixed Reality, Human-Machine Cooperation and Systems, Human-Machine Interaction
Abstract: Descending stairs is a difficult task for the elderly and disabled, and it is difficult to rehabilitate using real stairs because of the high degree of difficulty and the anxiety it causes. Therefore, rehabilitation using real stairs is difficult. Therefore, stair training on a flat surface using motor imagery is expected to be a new rehabilitation training method that eliminates the fear of falling. This paper proposes a new rehabilitation training method that simultaneously provides a visual presentation using virtual reality (VR) and a force sensation to the muscular senses using a pneumatic gel muscle (PGM). The system is designed to allow users to perceive the sensation of descending stairs. To verify the effectiveness of the system, we confirmed the muscle activity of the quadriceps muscle using electromyography (EMG) and evaluated the presence of the sensation of descending stairs using a psychological evaluation.

Keywords: Human-Computer Interaction
Abstract: The pupil center and glints have been used in conventional eye-gaze estimations, and the optical axis and point-of-gaze are calculated using these as keys. However, the pupil center used as a basis gets shifted when the diameter of the pupil varies, and the variance in illumination condition leads to degradation of accuracy. Therefore, we propose a semi-automated annotation for iris detection and a model-based gaze estimation method that uses an iris center. The iris area was extracted using segmentation trained with annotated data. The optical axis is determined using the iris center, which does not shift, and is therefore implemented instead of the pupil center. We evaluated the robustness of the proposed gaze estimation through an experiment under changing illumination conditions and confirmed the effectiveness of the iris-based estimation.

Keywords: Human-Computer Interaction
Abstract: 人机交互（HCI）和机器学习（ML）技术具有对自闭症儿童进行行为筛查的潜力，但如何设计工具并可靠地分析行为具有挑战性。在本文中，我们提出了一种交互式行为知觉分析模型，为自闭症筛查寻找客观定量的行为指标。我们介绍了基于自闭症儿童非典型特征设计的多场景反应行为范式。我们记录了91名参与者的眼动数据和面部数据，并进行多模态特征提取，使用机器学习训练分类模型。我们进行了对比实验，实验结果验证了多场景范式和多模态特征群的优势，表明我们的分析方法和筛选模型有效可靠，具有实际研究意义。

Keywords: Human-Computer Interaction
Abstract: The 3D point-of-gaze has been calculated using the binocular visual axes in the model-based eye gaze estimation. However, the 3D point-of-gaze is often shifted largely when the visual axis includes certain errors. The distance between the estimated point-of-gaze and the visual axis can be used as an error metric; however, horizontal errors cannot be evaluated, and an additional error metric is required to assess the reliability of the estimated point-of-gaze. Therefore, we propose a novel error metric using the correlation between binocular corneal images. We hypothesize that binocular corneal images extracted around the reflection of the estimated point-of-gaze exhibit a high correlation when the 3D point-of-gaze is correctly estimated. The two approaches were implemented using the corneal images extracted directly and generated based on binocular corneal imaging, and these approaches were evaluated through comparative experiments. We confirmed the feasibility of the proposed error metric and the effectiveness of binocular corneal imaging.

Keywords: Human-Computer Interaction, Supervisory Control, Human Space Flight
Abstract: Autonomous systems show promise as teammates for human operators in safety- and performance-critical environments. Humans’ cognitive states (like trust, mental workload, and situation awareness) change as they work in demanding environments with autonomous systems. Providing adaptive autonomous systems with information about human teammates’ cognitive states is a current gap in human-autonomy teaming research. In this paper we present results from a human-autonomy teaming experiment in which participants completed a spaceflight-relevant task in a supervisory, “on-the-loop” role. To complete the task, participants worked with an autonomous system that had five distinct modes of autonomy. Our experiment results show that unobtrusive measures (based on actions participants take and eye tracking data) could inform statistical models that accurately predicted three different subjectively reported cognitive states at the same time (RMSE from 11% to 14% of questionnaires’ ranges). Our work builds upon previous research and demonstrates that our model-building approach can extend to different human-autonomy teaming scenarios. This also represents the first time that three cognitive states were predicted using unobtrusive measures from the same supervisory human-autonomy teaming task. Our research enables future experiments investigating an autonomous system which adapts according to cognitive state predictions.

Keywords: Optimization and Self-Organization Approaches, Computational Intelligence
Abstract: The purpose of this paper is to propose a method for analyzing the period sensitivities and bifurcations for rhythm phenomena. The authors have already proposed a method for analyzing the sensitivities of the period with respect to parameters, an important characteristic of rhythm phenomena, and computationally efficient algorithms for this purpose. The period sensitivities of rhythm phenomena depend on bifurcations of that phenomena. In this paper, we propose a method and computation algorithms for investigating variations of the period sensitivities and bifurcations for rhythm phenomena by incorporating those algorithms. We also show an example of investigating variations of the period sensitivities and bifurcations by using the developed algorithm and demonstrate what can be revealed by it.

Keywords: Optimization and Self-Organization Approaches, Evolutionary Computation
Abstract: Real-world optimization problems are often computationally expensive and feature multi-modal objective functions. Surrogate-assisted evolutionary optimization has proven to be an effective approach for addressing expensive black-box optimization challenges, but the technique has not been adequately studied in multi-modal situations. In this paper, we propose a simple but effective multi-output surrogate-based approach for empowering surrogate-assisted evolutionary optimization to address expensive multi-modal optimization problems. Specifically, our proposed approach employs a multi-output Gaussian process to capture correlations between data collected from different local areas. Experiments on synthetic benchmark test problems demonstrate the effectiveness of our proposed algorithm against five state-of-the-art peer algorithms.

Keywords: Intelligent Power Grid, System Modeling and Control, Control of Uncertain Systems
Abstract: In this paper, a new method is proposed to minimize network losses in active distribution networks (ADNs). In recent years, studies on ADNs have been prevalent because of the emergence of distributed energy resources (DERs) such as Photovoltaic (PV) systems, wind power generation, etc. One of the challenges is how to handle the uncertainties of such renewables affected by weather conditions. This paper proposes an Evolutionary-Computation (EC)-based Robust Optimization (RO) method for minimizing the network losses in ADNs. This paper aims at reducing the risks of the obtained solutions in ADNs with parameter uncertainties in nonlinear combinatorial optimization problems. RO evaluates feasible solutions by considering the worst scenario through data sampling while EC works to evaluate better solutions that escape from local minima. As a method of EC, this paper makes use of Brain Storm Optimization (BSO) to obtain better solutions. It plays a key role to separate solution candidates into several clusters to evaluate better solutions by selecting one of four rules randomly. The effectiveness of the proposed method is demonstrated in the IEEE 69-node distribution network.

Keywords: Smart Buildings, Smart Cities and Infrastructures, Smart Metering
Abstract: With the widespread implementation of smart meter infrastructure, various applications have been proposed and implemented beyond energy usage monitoring and savings. In particular, there has been a growing interest in using smart meter data for daily life monitoring and remote care systems by visualizing human activities based on electricity, gas, and water usage. Previous research has demonstrated the feasibility of activity detection using an individual smart meter dataset. However, research on the effectiveness of using multiple smart meter datasets is limited. This paper proposes a model for Activities of Daily Living (ADLs) detection using multiple smart meter datasets (i.e., gas and electricity). To evaluate the performance of the proposed approach, a case study was conducted with 47 households for one month, collecting electricity data, gas data, and ground truth data about ADLs. The case study results showed that our proposed approach, which uses multiple smart meter datasets, enables more accurate activity detection.

Keywords: Smart Buildings, Smart Cities and Infrastructures, Infrastructure Systems and Services, Decision Support Systems
Abstract: Water quality prediction can accurately reflect the development trend of water quality, and it is an important means to prevent the water environment from being polluted and maintain the health of the water environment. Existing prediction methods generally cannot accurately capture nonlinear characteristics of water quality, and suffer from issues of gradient disappearance and gradient explosion. This work designs a water quality prediction model called SMF2 to effectively solve these problems and increase the accuracy of prediction. SMF2 combines the Savitsky-Golay filter, seasonal-trend decomposition using loess for multiple seasonal components, Fourier transform frequency-enhanced block and frequency-enhanced attention, serving for noise smoothing, extraction of exact seasonal components, time domain-frequency domain interconversion, feature extraction, and time series prediction by frequency domain lowrank approximation transform, respectively. Experimental results based on a real-life water environment data set show that the proposed SMF2 outperforms other advanced algorithms in terms of prediction accuracy.

Keywords: Decision Support Systems, Infrastructure Systems and Services, Smart Buildings, Smart Cities and Infrastructures
Abstract: Water quality prediction provides timely management to solve possible water environmental problems, which is of great importance. However, the following challenges exist: 1) The existence of noise in the water quality time series can lead to overfitting of nonlinear models; 2) It is difficult to capture temporal dependencies in complex time series data; 3) Long-term forecasting is difficult to achieve. To address the above difficulties, this work proposes a multi-step water quality prediction model, called SG-Autoformer, which combines the Savitzky-Golay filter, the inner series decomposition, and an autocorrelation mechanism. First, SG-Autoformer performs noise reduction on the water quality time series to suppress overfitting of nonlinear models. Second, it embeds series decomposition inside the encoder and decoder, which obtains more predictable components from complex time series for long-term prediction. Third, SG-Autoformer utilizes the auto-correlation mechanism to capture the time dependence and improve information utilization. Extensive experiments with real-world datasets show that SGAutoformer outperforms other advanced prediction methods in terms of prediction accuracy.

Keywords: Swarm Intelligence, Complex Network, Cybernetics for Informatics
Abstract: The shepherding problem refers to guiding a group of agents (called sheep) to a specific destination using the repulsive forces of an external agent (called a shepherd). Although various movement algorithms for the shepherd to achieve the guidance task can be found in the literature, there is a scarcity of methodologies for selective guidance, which is a key technology for realizing finer swarm control. Therefore, in this study, we investigate the problem of guiding a single target sheep within a large swarm to a given destination using a shepherd. We first present our model of the dynamics of sheep agents and the interaction between sheep and shepherd agents. We then show that the model is well-defined in the sense that no collision occurs if the magnitude of the interaction between sheep and shepherd is reasonably small. Based on this analysis and using Lyapunov stability principles, we design a shepherd control law to guide the target to the origin while avoiding collisions. Our experimental results demonstrate that the proposed method is effective in guiding the target sheep in both small and large scale swarms.

Keywords: Swarm Intelligence, Evolutionary Computation, Application of Artificial Intelligence
Abstract: This paper investigates a swarm shepherding problem which aims to herd multiple sub-swarm of robot agents (sheep) in a large-scale cluttered environment to a specific goal area using multiple distance-constrained robots (sheepdogs) located at different depots. We propose to formulate this challenging problem as a Multi-depot, Distance-constrained Close-Open Mixed Vehicle Routing Problem (MDCOMVRP). We also design a Two-phase Ant Colony Optimisation to address it by decomposing MDCOMVRP into a Multi-depot Open Vehicle Routing Problem~(MOVRP) and a split problem. In the first phase, the Max-Min Ant System algorithm is employed to find open routes for all robots by transforming the MOVRP into a standard Travelling Salesman Problem using the proposed transformation method. In the second phase, a Modified Split algorithm is presented to construct a set of close or open distance-constrained routes, which are further optimised by the 2-opt local search method to generate the optimised sequence for each sheepdog robot to collect/drive sheep sub-swarms. Experiments are conducted to demonstrate that the proposed algorithm can solve MDCOMVRP successfully and assist the robots to complete the swarm shepherding mission efficiently.

Keywords: Swarm Intelligence, Evolutionary Computation, Metaheuristic Algorithms
Abstract: Cooperative co-evolutionary algorithms (CCEAs) have witnessed giant success in solving large-scale optimization problems (LSOPs). However, most existing CCEAs use low-dimensional EAs to optimize the decomposed sub-problems. Such utilization of low-dimensional EAs may limit the effectiveness of CCEAs because some of the decomposed sub-problems may still be high-dimensional. Since there exist many non-decomposition based large-scale EAs, it is interesting to investigate the optimization effectiveness of CCEAs by using these non-decomposition based large-scale EAs to solve the decomposed sub-problems. To this end, this paper incorporates two state-of-the-art large-scale swarm optimizers into CCEAs with five state-of-the-art decomposition strategies to solve LSOPs. Experiments conducted on the CEC'2010 and CEC'2013 LSOP benchmark sets have shown that the two large-scale swarm optimizers help CCEAs with the five decomposition strategies achieve much better performance than the most widely used low-dimensional EA.

Keywords: Swarm Intelligence, Evolutionary Computation, Computational Intelligence
Abstract: This paper proposes a stochastic dominant cognitive experience-guided learning framework for particle swarm optimization (SDCEGPSO) to enhance its search ability in complex environment. Specifically, different from classical PSOs, SDCEGPSO randomly selects dominant cognitive experiences to guide the learning of particles. To this end, the cognitive experiences of all particles, namely their personal best positions, are sorted from the best to the worst. Then, each particle randomly chooses a personal best position better than its own to learn. For the cognitive experience selection, this paper designs three selection methods, namely the random selection, the roulette wheel selection, and the tournament selection. With this learning framework, particles have diverse guiding exemplars to learn from and thus high search diversity is expectedly maintained. Experiments conducted on the 50-D and 100-D CEC2014 problem suite have verified the effectiveness of SDCEGPSO. Compared with the classical global PSO (GPSO) and local PSO (LPSO), SDCEGPSO with the three selection schemes achieve significantly better performance. Besides, among the three selection schemes, the binary tournament selection is the most effective one to help SDCEGPSO solve optimization problems.

Keywords: Swarm Intelligence, Cybernetics for Informatics
Abstract: The composite velocity and acceleration constraints suffered by nonholonomic mobile robots during motion severely hamper the stability and smoothness of the cluster. This paper presents an applicable control framework for the robust and smooth implementation of distributed formations based on nonholonomic mobile robot clusters against velocity and acceleration saturation. The decoupled feedback linearization control of cascaded position, heading and wheel velocity is used to achieve distributed time-varying formations with modular and scalable stabilization of the triple-level errors. The adaptive auxiliary dynamics are incorporated into the distributed coincident errors to reduce velocity and acceleration saturation, which improves the overall transient results and stability range. The adaptive saturation extended observer in the substrate dynamics is conceived to simultaneously estimate the lumped disturbances and suppress undesired peaks, smoothly enhancing the robustness of wheel velocity control. Ultimately, comparative simulations and extensive experiments for mobile robots are performed to test validity and practicality.

Keywords: Human Performance Modeling, Human-Machine Interaction, Human Factors
Abstract: Previous research has shown how statistical model checking can be used with human task behavior modeling and human reliability analysis to make realistic predictions about human errors and error rates. However, these efforts have not accounted for the impact that design changes can have on human reliability. In this research, we address this deficiency by using similarity theory from human cognitive modeling. This replicates how negative transfer can cause people to perform old task behaviors on modified systems. We present details about how this approach was realized with the PRISM model checker and the enhanced operator function model. We report results of a validation exercise using an application from the literature. We discuss the implications of our results and describe future research.

Keywords: Human Performance Modeling, Human-Machine Interaction, Human Factors
Abstract: Mental workload extremes are associated with poor human performance and safety problems across safety critical domains. Mental workload is a complex, difficult-to-predict phenomenon, where issues may only arise due to concurrency between resource-conflicting tasks. This research addresses this deficiency by presenting a novel method for using model checking (for performing formal proofs about concurrent systems) to predict mental workload. Our method combines multiple resource theory and formal methods based on hierarchical task analysis to identify mental workload extremes in a complex system. This paper presents this method and shows preliminary validation using a texting and driving task. Implications of our results and future research are discussed.

Keywords: Medical Informatics, Human-Computer Interaction, Human-Machine Cooperation and Systems
Abstract: Computer-aided-diagnosis (CAD) systems have become an important utensil in today’s radiologist’s toolbox. The new age of computer vision transformers could further increase their value, although domain-specific limitations and adaptations should be studied first. Here we show that with the new adoption of the set prediction paradigm in transformer-based object detection, the Hungarian loss’ applicability to medical imaging could benefit from specialized modifications. A new dataset of 50,000 chest radiographs was used to study the Hungarian set matching’s ability to model the detection of 17 classes of pathologies. Consequently, five new targeted matching schemes were derived accordingly. The proposed strategies increased the overall mean average precision (mAP) score from 47.17 to 50.82 (+3.65). A subsequent reader study involving four radiologists showed the physician’s mean overall sensitivity improved by 6.9 ± 7.1% (95% CI, P = 0.008) while the specificity remained non-inferior (P < 0.001) when assisted by the AI model in the diagnosis of 200 patients. The results show how some of the set prediction matching’s shortcomings could be remodelled to fit chest x-ray pathology detection and make a case for transformed-based computer-aided diagnosis (CAD).

Keywords: Medical Informatics
Abstract: Diagnosing thyroid nodules with computer-aided techniques remains a challenging task. Using ultrasound videos for the classification of benign and malignant nodules can provide valuable timing and change information that is consistent with clinical diagnosis. In this paper, we propose a novel thyroid nodule classification model based on ultrasound video. To capture different semantic information, we sample video frames at various time intervals and extract local and global features using two different feature extraction branches. Our experimental results show that our method outperforms existing state-of-the-art methods, demonstrating its effectiveness in accurately diagnosing thyroid nodules with ultrasound videos.

Keywords: Conflict Resolution

Keywords: Intelligence Interaction, Networking and Decision-Making
Abstract: Embodied AI, where the agent learns to accomplish tasks through interaction with its surrounding environment, is drawing increasing attention in the community. As a challenging Embodied AI task, visual room rearrangement aims to restore the initially misplaced objects in a room to the target state. Existing approaches usually use a single policy to learn a mapping from visual observation to action. Those methods may be capable of accomplishing tasks with simple goals such as visual navigation. However, the agent in the rearrangement task has to explore various types of interaction for a long time. Only considering a single policy may easily get stuck in local optimum. In this paper, we propose a Phase Discriminated Multi-Policy (PDMP) model, decomposing the task into specific phases and tackling them with customized policies. In particular, we first introduce the graph representation of object relationships providing scene layout knowledge, which is discriminated to task phases. Then based on the knowledge a hierarchical actor-critic module is proposed to dynamically call the policies capable of navigation or object interaction. Each policy is trained with narrowed action space and dense rewards so that they can better converge and cooperate to reach long-term goals. Comprehensive experiments based on the AI2-THOR platform, show that the proposed model achieves better performance than baselines.

Keywords: Information Visualization, Kansei (sense/emotion) Engineering, Visual Analytics/Communication
Abstract: Recently, there has been a growing interest in developing recommendation systems that capture users’preferences and interests, while focusing on visual styles has gained increasing attention. However, understanding interior-style scenes is challenging because of the complex interplay of various objects that must be correctly interpreted. To properly understand interior-style scenes, extracting appropriate image features and providing visual explanations for the objects that determine the style is necessary. This study proposed a model that can simultaneously extract image features at multiple scales and provide visual explanations for the objects that characterize each class. Specifically, we adopted a hierarchical attention branch network (ABN) for visual explanations and applied atrous convolution to the feature extractor. We hypothesized that atrous convolution could extract various specific objects of each class that differ in size because its algorithm extracts features at an arbitrary resolution. Our evaluation results show that a conventional hierarchical ABN could not extract interior objects accurately, and our proposed model can detect specific objects by incorporating atrous convolution.

Keywords: Human Perception in Multimedia, Human-Computer Interaction, Information Visualization
Abstract: Recommendation algorithms automatically suggest news articles based on past behavioral logs. There has been reported cases of mental health problems caused by continuous consumption of negative articles, besides recommendation algorithms has a problem of over-recommendation which may induce continuous consumption of negative articles. Although research on the relationship between negative news article consumption and mental health has been conducted via small-scale user interviews, large-scale behavioral research on user engagement with recommended news articles has not been carried out. Therefore, we comprehensively investigated how the emotional polarity of articles affects each indicator of user attention by assigning emotional labels to news articles using crowdsourcing and analyzing about 1 million user behavior logs that viewed these articles. To the best of our knowledge, this is one of the first publicly available studies to analyze the impact of negative articles on users' news consumption behavior on an online news platform. The findings indicated that negative articles, irrespective of their category, were more likely to be clicked on, were read for longer durations, and had lower bounce rates. Furthermore, users showed greater interest in negative news related to entertainment and sports. These findings can be used as a first step for news platforms to build safer recommendation algorithms that consider the psychological impact on users.

Keywords: Medical Informatics, Information Visualization, Visual Analytics/Communication
Abstract: In practice, fetal 3D ultrasound can have difficulty in accurately detecting labels for auxiliary standard cut plane localization because of mass loss. Therefore, in this paper, we propose a new segmentation-based reinforcement learning framework for automatically localizing the standard plane of the face: in 3D fetal ultrasound, the initial plane is localized based on anatomical landmarks of mass and geometric relationships, agents navigate through visual segmentation to automatically localize the standard plane, and bound ultrasound views are presented to show the resultant plane. This study was extensively validated on an in-house large dataset. The accuracy of this automatic localization of 3D ultrasound standard planes with sonographer-calibrated median sagittal views of the face, horizontal transverse views of both eyeballs, and coronal views of the nasolabial was 6.64 °/5.65mm, 7.04°/3.58mm, and 5.14 °/4.26mm, respectively, with success rates of 66.67 %, 78.38 %, and 80.41 %, respectively. The experimental results verify that this system can effectively improve navigation performance.

Keywords: Evolutionary Computation, Swarm Intelligence, Metaheuristic Algorithms
Abstract: Ridesharing is a popular transportation mode and has become an important part of smart city development, which helps alleviate the pressure of urban travel. The ridesharing problem (RSP) is mainly to match drivers to suitable pas- sengers. In practice, passengers appear dynamically, and the departure and the destination locations of these subsequent orders are unknown, resulting in the dynamic RSP (DRSP). To solve this dynamic optimization problem, this paper develops a new genetic programming hyperheuristic (GPHH) method to evolve order dispatching rules (ODRs), which can guide drivers to match suitable passengers in real time. The proposed GPHH method contains a heuristic template for simulation-based hyper-heuristic optimization. The experiment results show that the proposed GPHH method outperforms the state-of-the-art methods. Further analysis revealed some valuable insights, such as the generalizability of the generated rules and the impact of some features on the results.

Keywords: Evolutionary Computation, Swarm Intelligence, Computational Intelligence
Abstract: Surrogate-assisted evolutionary algorithms (SAEAs) have become a popular method to solve data-driven optimization problems (DOPs), which are common in industry. However, with the development of the Internet of Things, data are collected, processed, and stored in a distributed manner, leading a new optimization paradigm for SAEAs. To make SAEAs adapt to these distributed DOPs, this paper employs the edge computing paradigm to develop a platform that provides technical support for SAEAs with distributed structures, named ECdo. Specifically, the platform utilizes KubeEdge, an open-source edge computing framework, to mount the cluster and combines microservice interface design with the containerization strategy to offer a flexible deployment approach for distributed SAEAs. In addition, an efficient and stable internal communication mechanism is designed for the interaction between distributed components within the platform. To demonstrate the application of ECdo, we take the examples of a class of distributed DOPs, in which the objective and constraints are expensive and need to be approximated by accumulated data. These problems are known as distributed and expensive constrained optimization problems (DECOPs). We implement a distributed SAEA on ECdo to address DECOPs in real-world scenarios. Experiments show that the ECdo can provide the expected implementation for distributed SAEAs with good network tolerance under tough network conditions.

Keywords: Evolutionary Computation, Swarm Intelligence, Heuristic Algorithms
Abstract: Efficient power inspection is crucial for maintaining a stable power system. During an inspection, unmanned aerial vehicles (UAVs) usually need to be recharged due to the wide geographical range of inspection and the limited battery capacity of UAVs. This limitation makes the problem more challenging that requires not only optimizing the task execution order, but also taking the chargings of UAVs into consideration. In order to address this complex problem, this work first formulates the UAV power inspection planning problem with charging stations. After that, we propose a new heuristic navigation model, in which UAVs can follow a heuristic rule to decide where to go next based on both its own information and task-related information. To obtain the heuristic rule, we design a set of features to describe the status of the UAVs and task completion. Then a genetic programming (GP) algorithm is introduced to evolve and get the heuristic rule. Finally, by applying heuristic navigation rule, the UAV navigation model can automatically prioritize task and charging order, and generate UAV flight routes that satisfy all constraints. The experiment results show that our method significantly outperforms the state-of-the-art algorithms.

Keywords: Machine Vision
Abstract: Depth completion is a crucial computer vision task that aims to fill in missing or incomplete depth values in a depth map. In this paper, we propose SDC: Spatial Depth Completion for Outdoor Scenes. Our approach leverages a two-stage architecture with a spatial feature extractor (SFE) to utilize multi-scale features for accurate depth completion effectively. The proposed method incorporates attention mechanisms, including the Efficient Position Attention Module (EPAM) and Channel Attention Module (CAM), to adaptively fuse depth map features and improve the accuracy of depth completion. Additionally, the Pearson loss function is employed to further enhance the accuracy of the completed depth maps. Experimental results on the KITTI depth completion benchmark demonstrate that our method achieves comparable or better results than traditional depth completion methods while significantly reducing the number of parameters. The proposed SDC model shows great potential in practical applications of depth completion, with its ability to effectively fuse multiscale features and compact model size.

Keywords: Deep Learning, Machine Vision, Representation Learning
Abstract: Inspired by the huge success of generative adversarial networks (GANs), GAN-based night-to-day translation methods have achieved excellent results. However, these methods have not been well visualized and understood, and thus cannot find the characteristics about their night-to-day translation performance. To this end, we present a simple clustering-based parsing approach to effectively understand the internal representations of the GAN-based night-to-day translator. In particular, we first cluster the internal representations of specific layers of the translator into a number of classes. Then, according to the proposed selection strategy, the class that has the most significant impact on the translation performance can be identified. Through experiments on three publicly available datasets, we find the answer of the question (title) is that the characteristics at the junction of bright and dark regions affect the performance of night-to-day translation.

Keywords: Machine Vision, Deep Learning, Machine Learning
Abstract: Zero-shot learning (ZSL) is an important but challenging task in computer vision that aims to identify unseen classes without matching training samples. Current cutting-edge ZSL methods based on locality focus on acquiring the explicit locality of distinguishing characteristics, which could face a lack of adequate supervision at the class attribute level. This paper introduces a novel approach called IAC, which aims to learn Implicit Attribute Composition for ZSL. This method is more comprehensive compared to attribute localization that solely focuses on class-level attribute supervision. IAC utilizes subspace representations that efficiently capture the inherent structure of high-dimensional image features. Then, we learn implicit attribute composition through subspace representation learning. The superiority of the proposed IAC compared to the state-of-the-art is demonstrated through sufficient experiments conducted on three commonly used ZSL datasets, CUB, SUN, and AwA2.

Keywords: Quantum Cybernetics, Quantum Machine Learning, Deep Learning
Abstract: The quantum state generation problem is a major research goal of quantum control and quantum variational algorithms, which use iterative optimization methods to evolve the initial state to the target state. The Twin Delayed Deep Deterministic Policy (TD3) algorithm in reinforcement learning achieves high learning efficiency and better stability in continuous control tasks. Here, using the TD3, we propose a new quantum state preparation method that does not require case-by-case optimization, to find a suitable evolution path to obtain the desired state. Specifically, we input the initial state into the trained actor-network, which can output the parameters of the unitary gates step by step, thus gradually evolving the initial state to the fixed quantum state. According to the reversibility of the unitary transformation, we can obtain a sequence of unitary gates to evolve the fixed state to the desired state. To verify the effectiveness of the algorithm, we perform simulations for one-qubit, two-qubit, and four-qubit cases, and the results show that the trained actor-network can provide appropriate unitary transformations to obtain the fixed state.

Keywords: Human Performance Modeling, Assistive Technology, Human Enhancements
Abstract: The fear of falling due to changes in gait leads to a decrease in quality of life in Parkinson’s patients. Also, Parkinson’s patients require medical treatment due to falling each year. However, the reasons for the changed walking style still remain unknown. The goal of this study is the evaluation of possible reasons for the changed walking pattern in Parkinson’s disease. A pilot study is conducted, which includes patient experiments, data analysis, and biomechanical modeling. Differences between Parkinsonian and healthy gait are detected and replicated by the model. The model represents simplified body dynamics and is optimized for healthy and Parkinsonian gait, respectively. Comparison measures are ground reaction forces, joint torques, and the virtual pivot point (VPP) location. The VPP is the intersection point of all forces throughout the gait cycle and is closely correlated to human balancing and stability. Parkinsonian gait showed different force and torque curves compared to healthy walking and a VPP below the center of mass location (negative VPP). The model represents healthy walking well. Specific Parkinsonian behavior can be explained by the changed modulation of the model. However, a negative VPP location turns the model unstable as, after two steps, the trunk tilts more than 90 degrees forward. Our modeling with negative VPP shows instability effects observed in Parkinson’s patients walking who struggle with frequent stumbles and falls. Such modeling approaches could be used for developing new rehabilitation techniques and gait assistance devices.

Keywords: Complex Network, Cloud, IoT, and Robotics Integration, Evolutionary Computation
Abstract: With the gradual development of unmanned aerial vehicles (UAV) related disciplines such as mechanics, electronics and wireless communication. UAV technology is an important direction for the future development of sixth generation mobile communications (6G) technology. When the BS computing capacity is insufficient or unavailable, UAV-aided 6G mobile edge computing (MEC) is considered a practical solution to maintain the balance between the number of computing tasks and the number of base stations (BS). Firstly, we consider the transmission power setting and design a prioritization mechanism to determine the link selection. Secondly, we construct energy consumption models for user devices (UDs), BS and UAVs separately. Finally, we propose UAV-aided MEC Genetic Algorithm (UMGA) based on genetic algorithm (GA) to minimize the overall energy consumption of the UAV-aided MEC system. Simulation results show that the system energy consumption of our proposed scheme is lower than the system energy consumption of other baseline schemes. Compared with the three baseline algorithms, the performance of our algorithm is improved by 660%, 300% and 41.2%, respectively.

Keywords: Agent-Based Modeling, AI and Applications, Optimization and Self-Organization Approaches
Abstract: The rapid advancement in blockchain technology has enabled its applications across wide spectrum of fields. The blockchain throughput, which is usually measured by Transactions Per Second (TPS), is one of the key metrics to reflect the performance of the blockchain systems. However, current blockchain systems have low TPS rates that makes them unsuitable for latency critical applications like Vehicle-to-vehicle (V2V) communication. To address the above issue, the sharding technology, which divides the network into multiple disjoint groups so that transactions can be processed in parallel, is applied to the blockchain systems as a promising solution to improve TPS. This paper considers the Optimal Blockchain Sharding (OBCS) problem which is formulated as a Markov Decision Process (MDP) where the decision variables are the number of shards, block size and block interval. Previous works solved the OBCS problem via Deep Reinforcement Learning (DRL) based methods where the action space has to be discretized such that it is not too large for tractability. However, the discretization degrades the solution quality since the optimal solution usually lies between discrete values. In this paper, we treat the block size and block interval as continuous decision variables and propose a sharding control algorithm based on Parametrized Deep QNetworks (P-DQN) to efficiently handle the discrete-continuous hybrid action space without the scalability issue. Experimental results show that our Parametrized Deep Q-Networks Blockchain Sharding (P-DQNBS) method can effectively improve the TPS by up to 20%.

Keywords: Cybernetics for Informatics, Evolutionary Computation
Abstract: With recent advances in technology, many-core systems have become increasingly common in high-performance computing applications, such as embedded systems and artificial intelligence. To fully utilize the processing power offered by this architecture, it is necessary to have good management and allocation of application tasks on Processing Elements (PEs), using Design Space Exploration (DSE) techniques. In this article, we develop an approach, called a Packet Simulator Tool for many-core Systems, implemented in a simulation environment using the Matlab, for analysis and calculation of the metric related to energy efficiency based on packet traffic using a Network-on-chip (NoC) architecture for a manycore processor. The approach uses high-level abstraction, is modular, and can be integrated with statistical analysis and Directed Acyclic Graph (DAG) generation tools. The experimental results showed that, although the NoCTweak simulator obtained better results, the approach proposed in this article proved to be promising for academic studies due to its ease of use and faster learning curve.

Keywords: Agent-Based Modeling, AI and Applications, Heuristic Algorithms
Abstract: The doubly resource constrained elementary shortest path problem (DRCESPP) has important applications in intelligent network scenarios. For solving this strong NP-hard problem, we introduce learning-based techniques and present a solution framework integrating preprocessing, graph neural networks (GNNs) and deep reinforcement learning (DRL). First, the preprocessing procedure reduces the network size and provides initial feasible paths. Then, the classifier, implemented by a graph attention network (GAT), filters out the reduced networks where better paths exist after preprocessing. Finally, a DRL-based heuristic attempts to construct the optimal path for the filtered reduced networks with an end-to-end solution paradigm. We devise a crafted reward function and a shared low-variance baseline for the reinforcement learning optimization algorithm. Our experiments suggest that the proposed framework achieves better performance compared with competitive heuristic algorithms in terms of solution quality and computational efficiency.

Keywords: Expert and Knowledge-Based Systems, Computational Intelligence in Information, Neural Networks and their Applications
Abstract: Text semantic matching is a core problem in Natural Language Processing (NLP), such as information retrieval and question answering, which is significant in intelligent human-computer interaction. However, most deep neural matching models are driven by external knowledge, which lacks fine-grained feature extraction from the sentences, leading to limited performance improvement. Therefore, this paper focuses on generating sentence semantic representations without external knowledge for sentence pair matching. We propose a Gated Attentive Convolutional Recurrent Neural Network (GACRNN), which incorporates a Gated Convolutional Neural Network (GCNN), Multi-scale Cross-Channel Attention Block (MC2AB), and bidirectional gate recurrent units (BiGRUs). First, a gate mechanism is introduced in the convolutional neural network to control the information interaction to extract multi-scale features from the sentence. Then, a multi-scale cross-channel attention mechanism is utilized to capture the feature dependencies at different scales in the channel dimension to generate expressive sentence representation. Finally, an extensive evaluation is conducted on two open-domain and two restricted-domain datasets. The experiment results show that the proposed model outperforms other baselines in terms of sentence pair semantic matching accuracy.

Keywords: Computational Intelligence, Machine Learning, Application of Artificial Intelligence
Abstract: For the past decade, the K Nearest Neighbor (K-NN) search in high dimensional space has been explored extensively. Considerable theoretical and practical algorithms to accelerate approximate K-NN search have been presented. The approximate methods can improve searching efficiency and achieve satisfactory performance. Nevertheless, they are inherently approximation approaches and are not guaranteed to yield exact solutions. To obtain the K-NN results over high dimensional datasets efficiently while guaranteeing the same results as the linear search is a challenging task, attracting a large number of scholars. To this end, in this paper, we focus on improving the exact K-NN search efficiency over high dimensional datasets, and present a framework named LBPC to solve this problem. The lower bound based method and parallel computing are combined in LBPC to accelerate the exact K-NN search. In LBPC, the whole K-NN search task is divided into some sub-tasks and these sub-tasks can be conducted concurrently using the lower bound based method. The LBPC scheme allows users to utilize any lower bound to accelerate the exact K-NN query. In this paper, we use the segment mean to construct the lower bound and provide the theoretical analysis to show its computational efficiency and lower bound property. Various experiments are conducted to analyze the efficiency of LBPC, and the experimental results validate its effectiveness.

Keywords: Deep Learning, Application of Artificial Intelligence
Abstract: 在本文中，我们提出了一种多层预训练 克服采掘业局限性的方法 机器阅读理解（EMRC）捕获全局 深入的语义信息并促进深度 文本和问题之间的交互。提议的 框架采用掩码语言模型校正BERT （MacBERT）模型和多层感知（MLP）进行预测 每个位置作为答案的概率。要解决 跨度提取，无法回答和是/否问题，我们 采用双向长短期记忆（BiLSTM）和 自我注意创建不同的目标层 神经网络模型。新模型表现出坚固耐用 泛化和交互式提取功能。跟 中国司法阅读理解（CJRC）数据集， 实验结果表明，所提算法 在民用领域带来 3.5% 的显著性能提升 案件和 4.9% 的刑事案件在 F1 分数方面。

Keywords: Communications, Fault Monitoring and Diagnosis, System Modeling and Control
Abstract: We present NadGPT, a transformer-based semi-supervised framework for network anomaly detection. It is known that transformer models are good at modeling long sequence data such as network traffic; however, without sufficient ground-truth labels, transformer models tend to suffer from over-fitting thus leading to inferior performance. Inspired by the recent success of GPT models in natural language processing (NLP), we propose a new auxiliary self-supervised task plugged to the backbone transformer, which enables GPT-like auto-regressive training on network traffic sequence without using ground-truth labels. Experiments demonstrate the proposed method greatly reduces the requirements of labels in network anomaly detection. For example, on ISCX 2012 dataset, given only 0.05% training labels our semi-supervised approach obtains nontrivial 81.7% (2-class) and 64.9% (5-class) F1-scores on the validation set, which is far better than the supervised counterparts using the same training data. We hope our research could inspire more label-efficient methods in network traffic analysis.

Keywords: Fault Monitoring and Diagnosis, Manufacturing Automation and Systems, System Modeling and Control
Abstract: Spot welding is one of the most frequently used material joining techniques in the Automotive Industry. Splashes are an anomalous condition of material expulsion that occurs randomly during the process and since it might result in welds with inadequate quality, it should be avoided. This study uses data of the Spot Welding process of a manufacturing unit that uses BOSCH technology and aims to apply Autoencoders considering a supervised learning approach to identify the occurrence of splashes. Additionally, its goal is to verify if the Autoencoder would outperform traditional techniques in this context when employed to identify rarer anomalies, as anticipated by the studies in the literature review. For this reason, the results for datasets with different anomaly rates are evaluated.