Keywords: Human-Machine Interface, Wearable Computing
Abstract: In recent years, deep learning (DL) has contributed significantly to the improvement of motor-imagery brain-machine interfaces (MI-BMIs) based on electroencephalography (EEG). While achieving high classification accuracy, DL models have also grown in size, requiring a vast amount of memory and computational resources. This poses a major challenge to an embedded BMI solution that guarantees user privacy, reduced latency, and low power consumption by processing the data locally. In this paper, we propose EET-TCNet, a novel temporal convolutional network (TCN) that achieves outstanding accuracy while requiring few trainable parameters. Its low memory footprint and low computational complexity for inference make it suitable for embedded classification on resource-limited devices at the edge. Experimental results on the BCI Competition IV-2a dataset show that EEG-TCNet achieves 77.35% classification accuracy in 4-class MI. By finding the optimal network hyperparameters per subject, we further improve the accuracy to 83.84%. Finally, we demonstrate the versatility of EEG-TCNet on the Mother of All BCI Benchmarks (MOABB), a large scale test benchmark containing 12 different EEG datasets with MI experiments. The results indicate that EEG-TCNet successfully generalizes beyond one single dataset, outperforming the current state-of-the-art (SoA) on MOABB by a meta-effect of 0.25.

Keywords: Human-Machine Interface, Human-Computer Interaction
Abstract: In this study, we adopted visual motion imagery, which is a more intuitive brain-computer interface (BCI) paradigm, for decoding the intuitive user intention. We developed a 3-dimensional BCI training platform and applied it to assist the user in performing more intuitive imagination in the visual motion imagery experiment. The experimental tasks were selected based on the movements that we commonly used in daily life, such as picking up a phone, opening a door, eating food, and pouring water. Nine subjects participated in our experiment. We presented statistical evidence that visual motion imagery has a high correlation from the prefrontal and occipital lobes. In addition, we selected the most appropriate electroencephalography channels using a functional connectivity approach for visual motion imagery decoding and proposed a convolutional neural network architecture for classification. As a result, the averaged classification performance of the proposed architecture for 4 classes from 16 channels was 67.50 (±1.52)% across all subjects. This result is encouraging, and it shows the possibility of developing a BCI-based device control system for practical applications such as neuroprosthesis and a robotic arm.

Keywords: Human-Machine Interface, Human-Machine Cooperation and Systems, Assistive Technology
Abstract: A Brain-Computer Interface (BCI) allows its user to control machines or other devices by translating its brain activity and using it as commands. This kind of technology has as potential users people with motor disabilities since it would allow them to interact with their environment without using their peripheral nerves, helping them to regain their lost autonomy. One of the most successful BCI applications is the P300-based Speller. Its operation depends entirely on its capacity to identify and discriminate the presence of the P300 potentials from electroencephalographic (EEG) signals. For the system to do this correctly, it is necessary to choose an adequate classifier and train it with a balanced data-set. However, due to the use of an oddball paradigm to elicit the P300 potential, only unbalanced data-sets can be obtained. This paper focuses on the training stage of two classifiers, a deep feedforward network (DFN) and a deep belief network (DBN), to be used in a P300-based BCI. The data-sets obtained from healthy subjects and post-stroke victims were pre-processed and then balanced using a Self-Organizing Maps-based under-sampling approach prior training looking to increase the accuracy of the classifiers. We compared the results with our previous works and observed an increase of 7% in classification accuracy for the most critical subject. The DFN achieved a maximum classification accuracy of 93.29% for a post-stroke subject and 93.60% for a healthy one.

Keywords: Brain-based Information Communications, Human-Computer Interaction, Human-Machine Interface
Abstract: Imagined speech is spotlighted as a new trend in the brain-machine interface due to its application as an intuitive communication tool. However, previous studies have shown low classification performance, therefore its use in real-life is not feasible. In addition, no suitable method to analyze it has been found. Recently, deep learning algorithms have been applied to this paradigm. However, due to the small amount of data, the increase in classification performance is limited. To tackle these issues, in this study, we proposed an end-to-end framework using Siamese neural network encoder, which learns the discriminant features by considering the distance between classes. The imagined words (e.g., arriba (up), abajo (down), derecha (right), izquierda (left), adelante (forward), and atrás (backward)) were classified using the raw electroencephalography (EEG) signals. We obtained a 6-class classification accuracy of 31.40 ± 2.73% for imagined speech, which significantly outperformed other methods. This was possible because the Siamese neural network, which increases the distance between dissimilar samples while decreasing the distance between similar samples, was used. In this regard, our method can learn discriminant features from a small dataset. The proposed framework would help to increase the classification performance of imagined speech for a small amount of data and implement an intuitive communication system.

Keywords: Brain-based Information Communications, Human-Computer Interaction, Human-Machine Interface
Abstract: The ability to perceive and recognize objects is fundamental for the interaction with the external environment. Studies that investigate them and their relationship with brain activity changes have been increasing due to the possible application in an intuitive brain-machine interface (BMI). In addition, the distinctive patterns when presenting different visual stimuli that make data differentiable enough to be classified have been studied. However, reported classification accuracy still low or employed techniques for obtaining brain signals are impractical to use in real environments. In this study, we aim to decode electroencephalography (EEG) signals depending on the provided visual stimulus. Subjects were presented with 72 photographs belonging to 6 different semantic categories. We classified 6 categories and 72 exemplars according to visual stimuli using EEG signals. In order to achieve a high classification accuracy, we proposed an attention driven convolutional neural network and compared our results with conventional methods used for classifying EEG signals. We reported an accuracy of 50.37 ± 6.56% and 26.75 ± 10.38% for 6-class and 72-class, respectively. These results statistically outperformed other conventional methods. This was possible because of the application of the attention network using human visual pathways. Our findings showed that EEG signals are possible to differentiate when subjects are presented with visual stimulus of different semantic categories and at an exemplar-level with a high classification accuracy; this demonstrates its viability to be applied it in a real-world BMI.

Keywords: Self-Organization, Swarm Intelligence, Optimization
Abstract: Research in multi-robot and swarm systems has seen significant interest in cooperation of agents in complex and dynamic environments. To effectively adapt to unknown environments and maximize the utility of the group, robots need to cooperate, share information, and make a suitable plan according to the specific scenario. Inspired by Maslow's hierarchy of human needs and systems theory, we introduce Robot's Need Hierarchy and propose a new solution called Self-Adaptive Swarm System (SASS). It combines multi-robot perception, communication, planning, and execution with the cooperative management of conflicts through a distributed Negotiation-Agreement Mechanism that prioritizes robot's needs. We also decompose the complex tasks into simple executable behaviors through several Atomic Operations, such as selection, formation, and routing. We evaluate SASS through simulating static and dynamic tasks and comparing them with the state-of-the-art collision-aware task assignment method integrated into our framework.

Keywords: Swarm Intelligence
Abstract: In the field of swarm intelligence, it is usually complicated to express the logic of swarm behaviors. Behavior tree has drawn a lot of attention to be a practical approach to solving this problem in recent years. However, how to automatically design the logic of swarm behaviors according to the target of a task is the focus of swarm intelligence. Hence, we propose an automatic optimizing framework named Benign which is capable of using gene expression programming (GEP) to optimize the logic of swarm behaviors. In Benign, the basic swarm behaviors and the relationships among those behaviors are mapped to nodes of behavior tree by the method named Matt firstly. With these nodes, we design an artificial behavior tree. After that, the artificial behavior tree is transformed into an expression tree in GEP according to the method named Meet. Finally, GEP is used for optimization to generate the expected logic of swarm behaviors. We conduct simulation experiments to validate the efficiency of Benign. The experimental results show the superiority of Benign. Compared with the logic of the artificial behavior tree before optimization, the conduction of the optimized logic of swarm behaviors increases efficiency by more than 50%.

Keywords: Swarm Intelligence, Agent-Based Modeling, Optimization
Abstract: A variant of multi-agent path finding in continuous space and time with geometric agents MAPF^R is addressed in this paper. The task is to navigate agents that move smoothly between predefined positions to their individual goals so that they do not collide. We introduce a novel solving approach for obtaining makespan optimal solutions called SMT-CBS^R based on satisfiability modulo theories (SMT). The new algorithm combines collision resolution known from conflict-based search (CBS) with previous generation of incomplete SAT encodings on top of a novel scheme for selecting decision variables in a potentially uncountable search space. We experimentally compare SMT-CBS^R and the previous CCBS algorithm for MAPF^R.

Keywords: Swarm Intelligence, Neural Networks and their Applications, Machine Learning
Abstract: Deep reinforcement learning has made significant progress in multi-agent tasks in recent years. However, most previous studies focus on solving full cooperative tasks, which do not perform well in mixed tasks. In mixed tasks, the agent needs to comprehensively consider the information provided by friends and enemies to learn its strategy, and its strategy is sensitive to the received information. There is a great necessity to efficiently learn information representation for mixed tasks. To this end, we present an approach that conducts information representation learning for multiple agents using hierarchical attention mechanism. Our approach adopts the framework of centralized training and decentralized execution. It applies hierarchical attention to centrally computed critics, so critics process the received information more accurately and assist actors to choose better actions. The hierarchical attention critic uses two different attention levels, the agent-level and the group-level, to assign different weights to information of friends and enemies respectively and then summarize them at each timestep. It can achieve more effective and scalable learning in mixed tasks. In addition, our approach uses recurrent neural networks that process sequence input information more efficiently. Experimental results show that our approach is not only applicable to cooperative environments but also better in mixed environments. Especially in the predator-prey task, our approach receives twice as much reward as baselines.

Keywords: Image Processing/Pattern Recognition, Neural Networks and their Applications
Abstract: An omni-directional image (ODI) is the image that has a field of view covering the entire sphere around the camera. The ODIs have begun to be used in a wide range of fields such as virtual reality (VR), robotics, and social network services. Although the contents using ODI have increased, the available images and videos are still limited, compared with widespread snapshot images. A large number of ODIs are desired not only for the VR contents, but also for training deep learning models for ODI. For these purposes, a novel computer vision task to generate ODI from a single snapshot image is proposed in this paper. To tackle this problem, the conditional generative adversarial network was applied in combination with class-conditioned convolution layers. With this novel task, VR　images and videos will be easily created even with a　smartphone camera.

Keywords: Image Processing/Pattern Recognition, Neural Networks and their Applications, Machine Learning
Abstract: Defects on the surface of fabrics seriously affect the production speed and quality of textile products. There are many difficulties in the detection of surface defects on fabrics, such as substantial differences in length-width ratio, uneven distribution, and few features. However, existing methods have the disadvantages of slow detection speed and high misdetection rate. This present study proposes a method of integrating deformable convolution and pyramid network in Cascade R-CNN (IDPNet) for fabric defect detection. First, image data are labeled according to the type and distribution of defects. Then we design a novel multi-stage object detection architecture named IDPNet to detect defects on the surface of fabrics. In the first stage, Resnet50, in combination with feature pyramid network and deformable convolution is used to improve the detection performance of small defects. Besides, we trained a sequence of detectors with increasing IoUs stage by stage based on Cascade R-CNN in the second stage. Finally, experimental results demonstrate that the proposed neural network equip an outstanding performance against other approaches and achieve the accuracy of 91.57% in fabric defect detection, which proves its utility in practice.

Keywords: Image Processing/Pattern Recognition, Neural Networks and their Applications, Machine Learning
Abstract: Detecting an anomalous human behavior can be a challenging task. In this paper, we present a novel objective function for autoencoders which include a temporal component. Our method is a fully end-to-end semi-supervised approach for video anomaly detection. The autoencoder is trained to reconstruct a sample from a partial input, by interpolating latent codes obtained from this partial input. We show this approach improves over using usual autoencoder objective functions for video anomaly detection and achieves results close to the state of the art on a broad range of datasets. Our code will be made publicly available.

Keywords: Image Processing/Pattern Recognition, Optimization
Abstract: Colour quantisation is a common image processing technique to reduce the number of distinct colours in an image which are then represented by a colour palette. Selection of appropriate entries in this palette is challenging since the quality of the quantised image is directly dictated by the palette colours. In this paper, we propose a novel colour quantisation algorithm based on the human mental search (HMS) algorithm and subsequent refinement of the colour palette using k-means. HMS is a recent population-based metaheuristic algorithm that has been shown to yield good performance on a variety of optimisation problems. In the first stage, we use HMS to find a high-quality initial colour palette. In the second stage, this palette is refined using k-means to converge towards a local optimum and thus to further improve the quality of the quantised image. We evaluate our algorithm on a set of benchmark images and compare it to several conventional and soft computing-based colour quantisation algorithms to demonstrate excellent image quality, outperforming the other methods.

Keywords: Machine Learning, Expert and Knowledge-based Systems, Image Processing/Pattern Recognition
Abstract: We propose a new method, named isolation Mondrian forest (iMondrian forest), for batch and online anomaly detection. The proposed method is a novel hybrid of isolation forest and Mondrian forest which are existing methods for batch anomaly detection and online random forest, respectively. iMondrian forest takes the idea of isolation, using the depth of a node in a tree, and implements it in the Mondrian forest structure. The result is a new data structure which can accept streaming data in an online manner while being used for anomaly detection. Our experiments show that iMondrian forest mostly performs better than isolation forest in batch settings and has better or comparable performance against other batch and online anomaly detection methods.

Keywords: Knowledge Acquisition in Intelligent, Computational Intelligence, Intelligent Internet Systems
Abstract: One of the most important sources of revenue for social media platforms, like, Twitter, Facebook, Reddit, etc., is advertising. An effective social media advertising plan moves people from awareness and interest in desire and action. Despite the potentiality, campaigns and marketing strategies should be improved. One of the challenges is to identify the right target audience at the right time, considering both communities of interests and locations and the development of these conditions along the timeline. This is crucial to create the right communication strategy and the right advertising message. This paper proposes a context-aware ad-targeting methodology using time, locations, and inferring users' interests by analyzing published content. The method relies on a fuzzy extension of Triadic Formal Concept Analysis for identifying Location-based and Content-based communities of users. Then, a task of community fusion takes place, named Join, for matching a target audience. The matching may be tuned for identifying a wide or narrow community and implementing a fine-grained ad targeting. Experimental results are given.

Keywords: Computational Intelligence, Expert and Knowledge-based Systems, Fuzzy Systems and their applications
Abstract: It is widely known that there is a lot of useful information hidden in big data, and it is prevalent for individuals to mine crucial information for utilization in many real-world applications. To find patterns that can represent the supporting transaction, a recent study was conducted to mine high-utility occupancy patterns whose contribution to the utility of the entire transaction is greater than a certain value. Moreover, in realistic applications, patterns may not exist in transactions but be connected to an existence probability. In this paper, a novel algorithm, called High Utility-Occupancy Pattern Mining in Uncertain databases (UHUOPM), is proposed. The patterns found by the algorithm are called Potential High Utility Occupancy Patterns (PHUOPs). To reduce memory cost and time consumption and to prune the search space in the algorithm as mentioned above, probability-utility-occupancy list (PUO-list) and probability-frequency-utility table (PFU-table) are used. Finally, substantial experiments were conducted to evaluate the performance of proposed UHUOPM algorithm on both real-life and synthetic datasets.

Keywords: Multimedia Computation, Machine Vision, Computational Intelligence
Abstract: Music has always been a popular media because it can relax our pressure of life. However, the music appealing to an individual could shift under his/her different emotions. For example, the preferred music in a sad mode is very possibly different from that in a happy manner. Therefore, effectively representing the human sense hidden in music can link the user emotion to music. To aim at this issue, Music Retrieval Information (MIR) were proposed for recognizing musical emotion. In the past, although some studies have been made on music emotion recognition, their effectiveness is not satisfactory. A potential reason is that the audio features extracted are not robust enough to discriminate the diversity between music and emotion. Hence, in this paper, we propose an effective music recognition method, which fuses Deep Learning (DL) and Support Vector Machine (SVM). The major difference between the proposed method and traditional audio-based studies is that the proposed method aggregates the partial recognition results of music to achieve better recognition precision. The experimental results on a real dataset of CAL500 show that the proposed method performs better than some other audio-based music emotion labeling methods.

Keywords: Machine Learning, Neural Networks and their Applications, Computational Intelligence
Abstract: Time series prediction has been challenging topic in several application domains. In this paper, an ensemble of two top performing deep learning architectures across different applications such as fresh produce (FP) yield prediction, FP price prediction and crude oil price prediction is proposed. First, the input data is trained on an array of different machine learning architectures, the top two performers are then combined using a stacking ensemble. The top two performers across the three tested applications are found to be Attention CNN-LSTM (AC-LSTM) and Attention ConvLSTM (ACV-LSTM). Different ensemble techniques, mean prediction, Linear Regression (LR) and Support vector Regression (SVR), are then utilized to come up with the best prediction. An aggregated measure that combines the results of mean absolute error (MAE), mean squared error (MSE) and R2 coefficient of determination (R2) is used to evaluate model performance. The experiment results show that across the various examined applications, the proposed model which is a stacking ensemble of the AC-LSTM and ACV-LSTM using a linear SVR is the best performing based on the aggregated measure.

Keywords: Neural Networks and their Applications, Machine Learning, Industry 4.0
Abstract: Missing data are commonly found in time series datasets. These missing elements are usually a hurdle in utilizing the datasets in prediction or forecasting, making imputation of those missing values imperative. Due to the non-linear dependencies between the current and previous values, imputation remains a challenging task. Conventional methods such as averaging, deletion or filling with the last observed value add bias to the data and are therefore inefficient. Since different time series showcase varying characteristics, figuring out which imputation method works best for the respective time series is essential. In this work, seven different deep learning (DL) imputation methods are examined along with three machine learning (ML) ensembles. To enable a recommendation of the best imputation method for each time series type, the imputation models are tested using the four main types of time series: trend (T), seasonal (S), combined trend and seasonal time series (T&S) and random (R) time series. Results indicate that the Gated Recurrent Unit (GRU) neural networks are, in general, the best for missing values imputation with varying complexity based on the time series type. For example, it is found that the residual GRU is recommended for the trend and seasonal time series while the GRU is recommended for the combined type. Conversely, all tested DL imputation models can be used with the random time series type. In addition, the considered ML ensembles do not perform as high as the DL models with all tested types of times series.

Keywords: Machine Learning, Neural Networks and their Applications
Abstract: Abstract—Adequately priced orders and time for fresh produce (FP) are two factors that bring financial benefits to vendors and minimizes waste. However, many factors such as income, labor, and other trade issues affect the price that include uncertainties due to climate change, making decisions on FP procurement prices and quantities extremely challenging. Two artificial intelligence-based forecasting tools, i.e. a single variate and a multivariate model, are trained, tested, and compared in this study to predict future daily offer prices up to 7 days ahead for strawberries using bilateral transactions for the distribution centers of Loblaws Companies Limited (LCL) in Canada. Results reveal that the developed multivariate model, utilizing both prices of the LCL dataset and California’s strawberries yield dataset as predictors, outperforms the best single variate model.

Keywords: Intelligent Internet Systems
Abstract: With the continuous growth of the number of mobile devices connected to the Internet, cellular network infrastructure owners are facing several new challenges. The fifth generation of mobile technology (5G) is planned to enable a fully mobile and connected society and to empower socio-economic transformations. In 5G, different scenarios with high and strict requirements are being deployed. To deal with such heterogeneity, more advanced communication services are required. Network Function Virtualization (NFV) combined with distributed micro data centers (MDCs) can improve the management of the data flows through several 5G base stations. On the one hand, NFV provides high scalability without requiring significant architectural changes; and on the other hand, distributed MDCs decentralize architecture management, bringing the computational capabilities nearer to the base stations, hence reducing transmission delay. Nonetheless, the introduction of MDCs raises new points of failures in the architecture, impacting user experience. In this paper, we evaluate the impact of allocating MDCs to host base station functionalities; similarly, network functions run as virtual machines in the MDC, and their failures cannot be ignored. We consider a real footrace scenario as a case study. As packets are lost due to the high mobility of the runners and spectators we examine this metric. From simulations results, we discover that as the number of MDCs increases, the number of failures increases as well. However, the impact of each failure in terms of lost packets decreases considerably when considering scenarios with more MDCs. For example, by increasing the number of MDCs from one to seven, the failure impact in terms of lost packet drops by around 87.62%.

Keywords: Neural Networks and their Applications, Machine Learning, Optimization
Abstract: Mislabeled samples are ubiquitous in real-world datasets as rule-based or expert labeling is usually based on incorrect assumptions or subject to biased opinions. Neural networks can "memorize" these mislabeled samples and, as a result, exhibit poor generalization. This poses a critical issue in fault detection applications, where not only the training but also the validation datasets are prone to contain mislabeled samples. In this work, we propose a novel two-step framework for robust training with label noise. In the first step, we identify outliers (including the mislabeled samples) based on the update in the hypothesis space. In the second step, we propose different approaches to modifying the training data based on the identified outliers and a data augmentation technique. Contrary to previous approaches, we aim at finding a robust solution that is suitable for real-world applications, such as fault detection, where no clean, "noise-free" validation dataset is available. Under an approximate assumption about the upper limit of the label noise, we significantly improve the generalization ability of the model trained under massive label noise.

Keywords: Machine Learning, Expert and Knowledge-based Systems, Optimization
Abstract: Control moment gyroscope is used as an attitude control system in satellites and its failure may results in mission failure. Fault diagnosis can prevent this if accompanied by an in-time remedial action. In this paper, a data-driven fault diagnosis model is developed using an optimized support vector machine to diagnose multiple in-phase faults of the satellite control moment gyroscope. The wavelet packet transform is used for feature extraction. The principal component analysis is used to reduce the number of features. Grid search is used to find the optimum values for the model hyperparameters and cross-validation with five-folds is used to validate the final model. The results show that the proposed model can predict faults with 62% accuracy.

Keywords: Optimization
Abstract: This paper exploits the analogy between the stochastic and zonotopic Kalman filters with the aim of formulating a metric that allows to assess the impact of an external zonotopically bounded signal in an state estimate optimal control scheme. To that end, the concept of linear quadratic zonotopic (LQZ) controller is introduced. Besides, the design of a zonotopically bounded watermarking signal such that enforces replay attack detectability is also addressed. The proposed detection scheme takes advantage of the observability loss of an textit{a priori} known exogenous signal during the replay phase of the attack. Consequently, by injecting the corresponding estimation error generated by a set-based observer, a zonotopically bounded signal such that imposes replay attack detectability is achieved. A numerical example is provided.

Keywords: Machine Learning, Cybernetics for Informatics, Neural Networks and their Applications
Abstract: Smart Cyber Physical Grids are the new wave of power system technology that integrates networks of sensors with power stations for more efficient power generation and distribution. While utilizing communication networks is accompanied with tremendous advantages, it also increases the vulnerability of power systems to cyber attacks. Many methods for security and attack detection have been proposed in literature; however, most papers do not consider the imbalance of data in real power systems. In this paper, we propose a deep learning based method, referred to as Ensemble Stacked AutoEncoder (ESAE), aimed at tackling the problem of data imbalance. This method achieves superior performance on imbalanced data by developing a deep representation learning model to construct new balanced representations. The detection accuracy and model performance is improved by utilizing an ensemble architecture based on Stacked Autoencoders and Random Forest classifiers to detect attacks from the new representations. The proposed method is tested on all degrees of data imbalance using test cases of IEEE 14-bus, 30-bus, and 57-bus systems. Comparisons are made to several classifiers to demonstrate the effectiveness of the proposed algorithm

Keywords: Machine Vision, Machine Learning
Abstract: This paper proposes a person re-identification method to solve the misalignment problem caused by pose/viewpoint variations, occlusion, etc. By applying the densely aligned part key points, we design a multi-branch part reconstructing network for pixel-level alignment to map the pixels on the pedestrian image to a unified densely aligned space. Concretely, we utilize the scheme of multi-task learning. In our reconstructing subnet, we assign the task of reconstructing 24-part maps, while we conduct the feature representation learning task in the backbone network to learn the features with densely aligned information. Our method achieves high performance, outperforming most state-of-the-art methods.

Keywords: Neural Networks and their Applications, Image Processing/Pattern Recognition, Machine Learning
Abstract: The giant panda (panda) has lived on the earth for at least eight million years, and as an endangered species, it has received extensive attention from scholars from all walks of life. As an important part of the panda population investigation, the individual identification of pandas can not only provide useful indications but also verify the effectiveness of protection measures. Some work has introduced image processing techniques and deep learning techniques to help researchers identify pandas using face images of a panda. In this paper, we proposed a grouping feature learning method for panda face recognition. In particular, we designed a feature mapping module which can be easily embedded into the existing feature extractors, and a grouping loss function is adopted to constrain the feature mapping, allows the learned similar features to be aggregated together and increase generalization. We use the open captive panda dataset to verify our method, and the results show that our method is effective.

Keywords: Biometric Systems and Bioinformatics, Neural Networks and their Applications
Abstract: Optical coherence tomography (OCT) imaging can obtain high-resolution cross-sectional scans of the retina, which can be used in clinical diagnosis. Changes in the thickness of layers indicate the onset of retinal diseases, motivating an accurate measurement of the thickness of retinal layers. Thus, an automatic and robust layer segmentation method is necessary. In this paper, we propose a deep learning-based multi-task framework to obtain the topologically consistent layer segmentation in OCT B-scans. By integrating the distance maps of retinal layer surfaces, the segmentation task is regarded as a multi-task problem of regression and classification. Besides, considering the multi-task learning problem, we propose a task-specific attention module to learn the task-tailored features. Experiment results on a public OCT dataset with multiple sclerosis (MS) demonstrate the effectiveness of the proposed method.

Keywords: Machine Vision, Machine Learning
Abstract: The high-frequencies of images is very important both in keeping the edges and suppressing artifacts. To improve the performance of single image super-resolution (SISR) based on the SRGAN framework, we propose Super-Resolution Generative Adversarial Networks with high-frequency discriminator (HFDSRGAN) by designing an additional discriminator for image’s high-frequencies extracted by wavelets. Based on SRGAN, the image’s high frequencies extracted by discrete wavelet transformations (DWT) were then introduced into GAN. Moreover, an additional discriminator for these high frequencies was built. Since the proposed model provides a direct and efficient way to locates and estimates the high frequencies of the reconstruction image, the visual effects of reconstructed the images can be improved with fewer computation costs. Experiments show that HFD-SRGAN has improved the visual effects of SRGAN when using the same generator network as SRGAN. The evaluation results show the performance of our method is equal to the state-of-the-art methods.

Keywords: Image Processing/Pattern Recognition, Machine Learning
Abstract: Person re-identification (Re-ID) with deep neural networks has made tremendous improvements recently. The existing person Re-ID methods can do well in viewpoint, occlusion, resolution, etc. However, they are short of robustness under varying lighting conditions. The variable illumination will result in the inconsistency of color, contrast, and SNR (signal-noise ratio), which will cause lots of difficulties to identify the right person. This paper presents a multi-granularity and multi-semantic Re-ID model combined with image enhancement method to minimize the impact of illumination variations and optimize the feature extraction. The Retinex-based image enhancement method is used to balance the variable illumination and enhance the contour information of images. Furthermore, we add multi-granularity and multi-semantic layers in the network to extract powerful feature representation. The proposed model is evaluated on the Market-1501, DukeMTMC-reID and CUHK03 datasets. Extensive experiments show that the new deep neural network model can extract more robustness features from the enhanced images, and verify the effectiveness of our method under changing illumination conditions.

Keywords: Human-Machine Interface, Assistive Technology, Wearable Computing
Abstract: Surface electromyography (sEMG) has been proven competent and reliable to recognize speech musculature movement patterns. In other words, we can understand what a person prepares to say by collecting sEMG signals around the mouth. Therefore, sEMG-based Mime Speech Recognition (MSR) is a potential technique for human-machine interaction within noisy surroundings as well as the application of helping dysarthric patients. In this paper, we introduce multi-layer Bidirectional Long Short-Term Memory (BLSTM) networks with attention mechanism as a classifier for MSR, and verify it in the data set collected by ourselves. Six-channel sEMG signals are firstly acquired from elaborately selected facial muscles. Short-time Fourier Transform (STFT) and Convolutional Neural Networks (CNN) are utilized to extract time-frequency domain feature maps, replacing the handcrafted features in classic methods. The second phase of recognition process lies in the designed classifier. This classification system achieves over 97% accuracy in the four-class MSR task, significantly surpassing simple CNN and LSTM methods. Such result also indicates that excellent MSR results can be achieved without relying on handcrafted signal features.

Keywords: Human Performance Modeling, Affective Computing, Mental Models
Abstract: Handwriting and hand drawings have historically been used as a proxy metric to evaluate psychological and cognitive traits, in addition to fine motor skills. Detecting onset of cognitive diseases early is a very challenging task due to the expertise required to evaluate each individual subject, as well as the time that the process entails. In this work, we evaluated the application of state-of-the-art deep learning and transfer learning models to determine if the author of a copied hand drawing of a template displays signs of cognitive disease. Compared to expert cognitive evaluation, our best performing method yielded a mean accuracy of 67.60% and area under ROC of 0.595 in determining if an undiagnosed subject displays signs of cognitive disease. Our results suggest that state of the art techniques in deep learning have the potential to help alleviate the difficulty of screening for early signs of cognitive disease. Results also suggest that transfer learning did not perform as well as a purpose-built network architecture trained from scratch. Lastly, our results indicate that models which consider all drawings made by a subject outperform models that look at drawing-evaluation pairs independently.

Keywords: Assistive Technology, Affective Computing, Human-Computer Interaction
Abstract: We live in a world surrounded with ‘fake news’ and manipulated information, so a system assisting people with knowing what information to trust would be beneficial. Our research investigates situations where the presenters themselves have doubts about the information they are delivering, and we detect this via advanced affective computing techniques. To this end we examine the physiological foundations for observer recognition of the doubt effect: the subjective belief or disbelief of a presenter in some information he or she is presenting. Firstly, we construct stimulus videos that display presenters delivering information about which we manipulate their degree of doubt. We then show these stimuli to observers, and record four of their physiological signals. We find that a generalised neural network trained with physiological features is more accurate in differentiating the presenters’ doubt/manipulated belief when compared with the same observers’ own conscious judgments. The affective recognition performance improves when we analyse the physiological signals using multi-task learning techniques to train personalised and group personalised neural networks. The ability to recognise this doubt effect derives from observers’ fundamental emotional reactions to the viewed stimuli, reflected in their physiological responses, and learnt by our neural networks. We believe this system using observer physiological signals collected in real life could reveal accurate and hidden audience distrust, which could in turn lead to enhanced truthfulness in future public-presented statements.

Keywords: Human-Machine Interface
Abstract: Neural interface using motor units (MUs) decomposed from surface electromyography (sEMG) has provided a novel approach for the intuitive human-robot interaction. However, existing feature extraction methods from decomposed MUs are simplex, ignoring the inherent spatial information and the subtle interactions between different MUs. In this study, we proposed a MU-specific images based scheme for extracting features from decomposed MUs and further estimating wrist torques continuously. Specifically, MU-specific images were reconstructed from decomposed MUs using sEMG and fed into a convolutional neural network for feature extraction and estimating wrist torques. The results demonstrated that the proposed scheme significantly outperformed three conventional regression methods using decomposed spike count features, with R² equal to 0.86 ± 0.05 in pronation/supination and 0.90 ± 0.05 in flexion/extension. This study provides a novel scheme for estimation of continuous movement using decomposed MUs and potentially paves the way of neural interface.

Keywords: Human-Computer Interaction, Human Factors, Human-Machine Interface
Abstract: A study was conducted to examine the operational use of multi-sensor data fusion for military airborne intelligence, surveillance and reconnaissance. Participants performed a simulated picture compilation task wherein they had to identify all ships and planes in their area of operation using various sensors. One group performed the task using only native sensor data. The second group had imperfect data fusion to help them resolve the kinematic information, but they still had to identify each contact. The results indicated that data fusion automation improved the identification of ships and planes over the native sensor group. However, over time, map clutter continually increased for the group with fusion automation, surpassing the clutter for the native sensor group. The results suggest that while multi-sensor data fusion has benefits for picture compilation, dealing with plot clutter from false and spurious tracks is a key concern. Interface suggestions are provided to mitigate the effects.

Keywords: Human-Computer Interaction, User Interface Design, Information Visualization
Abstract: In Tactical Combat Casualty Care (TCCC), medics perform Role 1 care for battlefield casualties at point of injury by stabilizing them and transporting them to field care facilities such as a Battalion Aid Station (Role 2) or Field Hospital (Role 3) where clinicians provide critical care. Care provider experience and ability vary, and training in the field can help to improve recall and performance of infrequently used critical care skills. This becomes more necessary during Prolonged Field Care (PFC) when evacuation is not immediately available and more complex treatment may be required. Our Trauma Triage Treatment and Training Decision Support (4TDS) project has developed a decision support system (DSS) for Roles 1 and 2. As an application on a Android smart phone and tablet, 4TDS includes training scenarios in skills such as shock identification and management. 4TDS pairs with various vital signs sensors that can stream data for a machine learning algorithm that can detect the probability of shock in a casualty. A “silent test” is comparing algorithm performance with actual clinical diagnoses at Mayo Clinic, Rochester, MN. Usability assessment in an austere field setting will enable us to determine medic and clinician acceptance of 4TDS and how well it supports their decision making. Faster, more accurate decisions can improve TCCC patient care under conditions in which delays can increase morbidity and mortality.

Keywords: Human-Machine Cooperation and Systems, Human-Computer Interaction
Abstract: Military sign language is an important form of tactical communication, especially in restrict situations where either distance or a requirement for silence precludes oral means. Unfortunately, when soldiers cannot see each other, the communication mode of tactical gestures is no longer effective, which may hinder military operations. Vision-based approaches have been at the forefront in the field of hand gesture recognition. However, there still lacks of specific datasets and models for the task of military sign language recognition. In this paper, we collected a new first-person dataset named MSL, which contains 16 classes of 3,840 tactical gesture samples on battle scenario with more than 11,0000 video frames performed by 10 subjects. Moreover, we present a novel deep network, called ST-Xception architecture, in light of the depthwise separable convolutions to recognize such military sign language. By expanding the convolution filters and pooling kernels into 3D, our network can characterize the inherent spatio-temporal relationship of a certain tactical hand gesture. In particular, we further reduce computational cost and relieve overfitting by replacing the fully connected layers with adaptive average pooling. Experimental results show that our model outperforms existing models both on our in-house MSL dataset and two other benchmark datasets.

Keywords: Human-Machine Cooperation and Systems, Human-Computer Interaction
Abstract: By nature, military Manned-Unmanned Teaming (MUM-T) missions are characterized by complex state spaces, uncertainty and multiple types/sources of information that the pilot needs to assess in a time critical manner to derive tactical decisions. In order to support the helicopter crew in such missions, we investigate a software agent that interprets available situational and mission related information to create machine situational awareness. In this article, we present a work in progress on the concept and implementation of a tactical situation assessment system using an Influence Map evaluation polytree for knowledge representation. Furthermore, we look at the problem from a human autonomy teaming perspective. The agent provides tactical recommendations to the pilot and shares its inferred knowledge about possible future critical states. We present the current state of integration in our helicopter research flight simulator and outline challenges concerning effective human-machine cooperation in tactical decision-making.

Keywords: Human-Machine Cooperation and Systems, Mental Models, Augmented Cognition
Abstract: Effective teams coordinate their actions to achieve shared goals. In Human-Agent teams, the Artificial Intelligent Agents (AIAs) struggle to coordinate effectively due to a lack of understanding of their human teammate’s intent. This places a burden on the human teammate to extensively communicate explicitly what goals they are pursuing and how they are pursuing them. To improve the AIAs ability to coordinate, we have proposed Operationalized Intent as a means to explicitly model how an operator qualitatively desires a task to be performed. In this paper, we report the results of a study to track operator intent through a tactical scenario. The focus of this paper is on the dynamics of intent and it’s cohesiveness across operators. The study employed an immersive, advanced research, remotely piloted aircraft (RPA) simulator to study intent in a synthetic task environment. Using operational pilots and sensor operators in realistic scenarios we were able to elicit their intent under naturalistic conditions in the midst of challenging tactical situations to study the real-time dynamics. Analysis indicates that the method models intent which is dynamically responsive to changes in the situation and the data are suitably cohesive across operators to generalize to an operator role. When the intent data is coupled to situated data from the simulator it provides a labeled data source for future AIAs to estimate intent in real-time.

Keywords: Companion Technologies, Human-Machine Interface, Mental Models
Abstract: Conversational agents (CAs), often referred to as chatbots, are being widely deployed within existing commercial frameworks and online service websites. As society moves further into incorporating data rich systems, like the internet of things (IoT), into daily life, it is expected that conversational agents will take on an increasingly important role to help users manage these complex systems. In this, the concept of personality is becoming increasingly important, as we seek for more human-friendly ways to interact with these CAs. In this work a conceptual framework is proposed that considers how existing standard psychological and persona models could be mapped to different kinds of CA functionality outside of strictly dialogue. As CAs become more diverse in their abilities, and more integrated with different kinds of systems, it is important to consider how function can be impacted by the design of agent personality, whether intentionally designed or not. Based on this framework, derived archetype classes of CAs are presented as starting points that can hopefully aid designers, developers, and the curious, into thinking about how to work toward better CA personality development.

Keywords: Human Factors, Mental Models, Human-Machine Interface
Abstract: The vestibulo-ocular reflex (VOR) is a reflexive eye movement that is generated in a direction opposite to that of the head movement and is affected by mental workload (MWL). In a previous study, VOR was modulated by providing a mental task when participants gazed at a fixation point and were exposed to unpredictable body motion induced by a motion platform; the modulation was described by changes in the parameter of the VOR computational model. However, further studies are required to determine whether these results are valid when participants actively operate a vehicle in an outdoor environment, such as driving. Therefore, in the present study, an experiment was conducted to investigate the influence of a mental task for car drivers based on a parameter of the VOR model. The MWL during the experimental trials was evaluated via the weighted workload (WWL) score calculated from the NASA-Task Load Index. Specifically, the eye and head movements were measured when participants drove a car in situations with and without a mental task. The experimental results indicated that the self-motion estimation parameter of the model decreased, and the WWL score increased with the addition of a mental task. The results suggest that the influence of MWL on VOR during driving can be described by the self-motion estimation parameter of the model.

Keywords: Human Factors, Mental Models, Wearable Computing
Abstract: When drivers encounter a road narrowing two potential adaptation strategies come into play that may increase safety margins: decreasing speed and increasing neuromuscular stiffness of the arms. These two adaption strategies have so far been studied in isolation. We expect that there is a trade-off between these two strategies, and that risk duration would impact a driver’s selection of the trade-off. Specifically, we hypothesized that for a short risk duration, drivers will favour increased neuromuscular stiffness over speed reduction; and vice versa for longer risk durations. Twenty-six participants drove in a driving simulator and encountered different risk durations; realized by road narrowings (from 3.6 m to 2.2 m) of varying lengths (10 m, 100 m, 250 m, and 500 m). The neuromuscular stiffness was quantified by measuring the grip force exerted by both hands. The results show that all road narrowing conditions successfully induced driver adaptations, as a significant reduction in speed and increase in grip force was observed. However, the tested drivers did not consistently select the hypothesized different trade-offs for increasing duration of road narrowing: a low correlation was found between speed and grip force adaptations. Interestingly, individual trade-off were consistent: the within-subject variability in speed-grip force adaptations was low across the tested risk durations. Future research should further elucidate the underlying motivations for these individual adaptation strategies.

Keywords: Human-Machine Cooperation and Systems, Human Performance Modeling, Mental Models
Abstract: In mixed-initiative systems where teams of humans and automated agents collaborate to perform decision-making tasks, determining factors of joint performance include human cognitive workload and the level of trust placed by the operators in the automation. Both workload and trust are dynamic variables that change over time based on current task allocation and on the result of past interactions. In this paper, we propose a methodology leveraging quantitative models of trust and workload to automatically and dynamically suggest efficient task allocations in mixed human-machine systems. Our approach is based on a Markov decision process framework and is presented for concreteness in the context of a human-machine team performing repeated binary decision-making tasks. Simulation results show the emergence of interesting automation behaviors such as seeking trust, attempting to repair trust after an error and adjusting human workload for optimal performance. Overall, the human-aware dynamic task allocation strategy shows the potential of significant team performance improvement compared to a static task distribution, even in the presence of significant errors in the trust and workload models used.

Keywords: Human-Machine Cooperation and Systems, Mental Models, Human Factors
Abstract: Latest generations of automobiles are gradually being equipped with technologies that have increasing automation, a trend which had led to increase in the system complexity as well as increased human-automation interactions. Failures in such complex human-automation interactions increasingly occur due to the mismatch between what operators know about the system and what the designers expect operators to know. Causes of road accidents also change due to role shift of drivers from controlling the vehicle to monitoring the in-vehicle controllers. Failures in such complex systems involving human-automation interactions increasingly occur due to the emergent behaviours from the interactions, and are less likely due to reliability of individual components. Traditional safety analysis methods fall short in identifying such emergent failures. This paper focuses on using a systems thinking inspired safety analysis method called System Theoretic Process Analysis (STPA) to identify potential failures. The analysis focuses on a SAE Level-4 Vehicle that is in the development phase, and is controlled partially by a safety driver and its built-in Autonomous Driving System (ADS). The analysis yields that while increase in complexity does increase system functionality, it also brings a challenge to evaluate the safety of the system and potentially causes incorrect human-automation interactions, leading to an accident. After the possible inadequate driver-vehicle interactions are identified by STPA, corresponding requirements were then proposed in order to avoid the unsafe behaviour and thus preventing the hazards.

Keywords: Assistive Technology, User Interface Design, Entertainment Engineering
Abstract: Recently, various types of robot partners aiming at social implementation have been developed and researched, but they have not yet been popularized. In the case of communication robots, it has been reported that it is easier and more comfortable for humans to communicate with multiple robots. However, multi-robots require large installation space; the role or character of each robot is not clear, and it is difficult for users to design utterance and action contents. Therefore, we propose a new design methodology of the body-sharing multi-robot system and develop a kangaroo-based robot that contains two physically interconnected objects as one of the models of multiple communication robots. Next, we show the evaluation of the proposed robot in terms of impression and discuss the possibility of social implementation of the developed body sharing multi-robot system.

Keywords: Assistive Technology, User Interface Design, Human-Computer Interaction
Abstract: This paper ponders the effects of introducing an assistive force visual display on a subject's force perception. Correct and incorrect displays were tested and were found to significantly affect the subject's force perception as compared to no visual feedback case. The elbow joint was considered for this study and assisted with 4 wearable artificial muscles called pneumatic gel muscles (PGMs). It was observed that subjects tend to underestimate the assistive force. This force underestimation could be significantly reduced by the introduction of a visual display. Moreover, the level of force underestimation was higher in case of higher force values. In other words, the higher the actual force provided, the higher the level of force underestimation in all 3 conditions: correct, incorrect and no visual feedback. It was also observed that the perceived error (perceived force - actual force) was directly in proportion with the visual error (visual force - actual force).

Keywords: Human-Machine Cooperation and Systems, Human-Machine Interface, User Interface Design
Abstract: Self-driving cars will relief the human from the driving task. Nevertheless, the human might want to intervene in the driving process and thus needs the possibility to control the car. Switching back to fully manual controls is uncomfortable once being passive and engaging in non-driving-related activities. A more comfortable way is controlling the car with elemental maneuvers (e.g., "turn left" or "stop"). Whereas touch interaction concepts exist, contactless interaction through voice and mid-air gestures has not yet been explored for maneuver-based car control. In this paper, we, therefore, compare the general eligibility of voice and mid-air gesture with touch interaction as the primary maneuver selection mechanism in a driving simulator study. Our results show high usability for all modalities. Contactless interaction leads to a more positive emotional perception of the interaction, yet mid-air gestures lead to higher task load. Overall, voice and touch control are preferred over mid-air gestures by most users.

Keywords: Human-Machine Interface, User Interface Design, Human Factors
Abstract: Remote crane operation has been a subject of research for several years. So far, there is a lack of intuitive human-machine interfaces that resolve present incompatibilities between operator orientation, the crane's joint movements, and the resulting load movement. This results in poor interface quality and thus in unnecessary time losses, errors, and mental stress. To overcome this gap, an intuitive app interface is developed and applied on a loader crane with boom tip control. Intuitiveness is assured by using user-centered design guidelines and multiple feedback loops. An evaluation study with 56 test persons in two experience groups is used to assess the interface's usability in terms of effectiveness, efficiency, and satisfaction. Results show that the human-machine interface presented here reduces errors and increases the operator's satisfaction in both user groups. Furthermore, unexperienced crane operators in particular profit from better load placement accuracy and greater time efficiency. The intuitive app-based human-machine interface can therefore improve remote crane operation significantly.

Keywords: Virtual and Augmented Reality Systems, Human Factors, User Interface Design
Abstract: Virtual environments can replicate the visual appearance of terrain conditions, but movements involved in using the interfaces confer their own bodily sensations, which can be incongruent with the visual presentation. Assuming that more natural interfaces produce more natural locomotor behaviors, we propose a framework for assessing the quality of a locomotion interface. Using this framework, we studied the interaction of different locomotion interfaces with visual information on wayfinding decisions in a virtual environment. We compared decisions made using a dual joystick gamepad with a walking-in-place metaphor. Paths presented on a given trial differed visually in one of the following aspects: (a) incline, (b) friction, (c) texture, and (d) width. In this experiment, choices made with the walking-in-place interface more closely matched visual conditions which would minimize energy expenditure or physical risk in the natural world. We provide some observations that would further validate this approach and improve this method in future implementations. This approach provides a way of both studying factors in perceptual decision making and demonstrates the effect of interface on natural behavior.

Keywords: Human-Computer Interaction, Virtual and Augmented Reality Systems, Web Intelligence and Interaction
Abstract: The use of cybernetic technologies in data-driven surgery simulation is a trendy topic. Although there exist promising studies in a virtual simulation of surgery operations there exists relatively little effort in the plastic surgery domain. This paper aims to present a data-driven interactive virtual reality simulation environment that is built on a novel surgical operational context taxonomy. The proposed taxonomy is used to develop a data model covering pre-surgery, peri-surgery, and post-surgery operations enabling effective monitoring of the surgery steps. The data model helps users to filter useful data about surgery steps and health data. Such a data-driven filtering mechanism will also assist the training of surgeon candidates to get more prepared for real-life surgery operations. The suggested solution also enables health data flow over an IoT backbone which augments the ambient-assisted reality of the surgery operation environment.

Keywords: Human-Computer Interaction, Virtual and Augmented Reality Systems
Abstract: To facilitate the development of a new generation of Virtual Reality systems and their introduction in everyday life applications, new intuitive, immersive methods of interfacing have to be developed. Over the years, Electromyography (EMG) based interfaces have been utilized for unobtrusive interaction with computer systems. However, previous EMG studies have not explored the continuous decoding of the effects of human motion (e.g., manipulated object behavior) in simulated and virtual environments. In this work, we present an EMG based learning framework that can allow for an immersive interaction with Virtual Reality environments. To do that, EMG activations from the muscles of the forearm and the hand were acquired during the execution of object manipulation tasks in a virtual world along with the motion of the object. The virtual world was visualized using an HTC Vive VR headset, while the hand motions were tracked with a dataglove equipped with magnetic motion capture sensors. The object motion decoding was formulated as a regression problem using the Random Forests methodology. The study shows that the object motion can be successfully decoded using the EMG activations, despite the lack of haptic feedback.

Keywords: Virtual and Augmented Reality Systems
Abstract: In recent years, visual-inertial odometry for state estimation has achieved a huge success in terms of robustness and accuracy. However, current visual-inertial odometry algorithms still struggle with high computational cost, which severely hinders its application on computational resource limited platforms. In this paper, a novel tightly coupled visual-inertial odometry algorithm that can maximize utilize the information provided by 3D map points and IMU pre-integration to reduce the algorithm’s computational cost is proposed. In the front-end, an accelerated optical flow tracking algorithm using the depth known 3D map points and IMU pre-integration between two consecutive frames is proposed to reduce the tracking time. In the back-end, a joint optimization model based on hybrid residuals is proposed, which consists the geometric, the photometric and the IMU residuals. It’s worth noting that partial feature points with good tracking quality from all the feature points that the algorithm is tracking on are selected to construct photometric residuals. Therefore, those selected feature points can provide double constraints for the motion. The one feature point provide double motion constraints strategy allows our odometry algorithm use fewer feature points under the condition of same number of motion constraints which really reduced the amount of feature points also the computational resource needed by our algorithm to achieve a comparable pose estimation accuracy. Experiment results on public available dataset demonstrate the effectiveness of our proposed approach

Keywords: Virtual and Augmented Reality Systems, Human-Machine Cooperation and Systems
Abstract: Whether for localization, path planning, or scene manipulation, complete and accurate scene reconstruction is an essential component of robotic operation. Due to their low cost and versatility, vision-based scene reconstruction methods have been the subject of research for decades. However, a major disadvantage of vision-based methods is that they require the scene to be populated with distinctive features that can be unambiguously matched across different images. In the absence of these features, such as in planar homogeneously painted surfaces, the scene reconstruction fails. This paper proposes a novel idea, where the user can virtually texturize planar surfaces at run-time to be used for the scene reconstruction. To do so, the corners of planes are tracked across the images and used to warp virtual texture patches to the correct perspective. Two methods are then proposed, one that actively tracks the corners as long as they are in view, and one that requires the camera poses to augment planes once their corners are no longer visible. The conducted experiments demonstrate the effectiveness of our approach as it increases the number of points in scene reconstructions. The end result is a denser scene reconstruction where textureless planes, typically not recovered in traditional methods, are reconstructed.

Keywords: Virtual and Augmented Reality Systems, Wearable Computing, Human-Computer Interaction
Abstract: This paper presents two Inverse Kinematics (IK) algorithms that are used for digital twin Augmented Reality (AR) applications. The first algorithm is a simple Inverse Kinematics (IK) Unity code that considers up to 9 points on the human body to model the motion. The second algorithm is the BIO IK that can consider up to 38 points. The performance of the algorithms is compared with data obtained by a Motion Capture (MoCap) measurement system. The metric of accuracy was used to quantify the performance evaluation and was modeled as the error of the position of the modeled joints of a human avatar with those measured by the MoCap system. It is observed that the obtained accuracy of the position increases with the number of points that is considered by the IK algorithm. For the purpose of this investigation, a MoCap system based on 13 cameras and 38 markers on the human body was used to measure the location of the joints of a human operator performing specific motions. The motion of lifting was the epicenter of the investigation that causes the larger amount of accidents in typical manufacturing facilities. The application of this research falls within the concepts of Digital Twin (DT) in Industry 4.0 scenarios.

Keywords: Human-Machine Interface, Wearable Computing, Assistive Technology
Abstract: Surface electromyography (sEMG) is a popular bio-signal used for controlling prostheses and finger gesture recognition mechanisms. Myoelectric prostheses are costly, and most commercially available sEMG acquisition systems are not suitable for real-time gesture recognition. In this paper, a method of acquiring sEMG signals using novel low-cost, active, dry-contact, flexible sensors has been proposed. Since the active sEMG sensor was developed to be used along with a bionic arm, the sensor was tested for its ability to acquire sEMG signals that could be used for real-time classification of five selected gestures. In a study of 4 subjects, the average classification accuracy for real-time gesture classification using the active sEMG sensor system was 85%. The common-mode rejection ratio of the sensor was measured to 59 dB, and thus the sensor's performance was not substantially limited by its active circuitry. The proposed sensors can be interfaced with a variety of amplifiers to perform fully wearable sEMG acquisition. This satisfies the need for a low-cost sEMG acquisition system for prostheses.

Keywords: Wearable Computing, Entertainment Engineering, Assistive Technology
Abstract: We introduce a novel use of wearable technology to reconstruct and output high dynamic range (HDR) audio signals. The purpose of designing an end-to-end HDR audio wearable device is to create an audio device capable of replicating and even extending the dynamic range of human hearing. We also introduce a different method of visualizing audio by way of a rotary Sequential Wave Imprinting Machine (SWIM). The results show that real-time HDR audio processing and visualization are possible at the standard audio sampling rate of 44.1 kHz. Additionally, the quality of the recorded audio improved from the originally captured data after HDR processing. We further analyze the dependence of different audio gain combinations as well as the influence of the number of audio input channels on the quality of the HDR signal reconstruction in various acoustic environments.

Keywords: Wearable Computing, Human-Computer Interaction, Human-Machine Cooperation and Systems
Abstract: Hand gesture recognition with wearables typically focuses on the characteristics of a single point on hand, but ignores the diversity of motion information over hand skeleton. As a result, current methods suffer from two key challenges to manage multiple hand joints: displacement detection and motion representation. This leads us to define a spatio-temporal framework, named STGauntlet, that explicitly characterizes the hand motion context of spatio-temporal relations among multiple joints and detects hand gestures in real-time. The framework introduces the Lie algebra to capture the inherent structural varieties of hand motions with spatio-temporal dependencies among multiple joints. In addition, we developed a hand-worn prototype with multiple motion sensors respectively attached to various joints on hand and collected 7000 samples of seven gestures from nine subjects. Our in-lab study shows that STGauntlet is capable of detecting gesture types together with their 3D tracking trajectory with 97.35% and 95.17% accuracies for subject dependent and independent recognition, respectively.

Keywords: Wearable Computing, Human-Machine Interface
Abstract: This paper evaluates the possibility of performing fine gesture recognition including finger movements on a low-tech device. In particular, we present a solution with a recognition model that is small enough to fit in the memory of a low-tech device and describe related difficulties associated with this approach. Several different Machine Learning techniques are employed and their individual advantages and drawbacks are explored for the task at hand. Our results indicate an average of 95% accuracy during real-time testing for an eight class decoding task with a custom Recurrent Neural Network approach, that runs on the low-tech device, namely an Arduino Nano 33 BLE. The novelty and strength of this research lies in the fact that we are able to recognize fine hand gestures including finger movements rather than recognizing only coarse hand gestures. The recognition process is conducted on the low-tech device and as a result this solution has all advantages that are typically associated with embedded systems, namely cost-efficiency, battery life efficiency, and a high degree of independence from other devices as well as compatibility with them.

Keywords: Human-Machine Interface, Assistive Technology, Human Factors
Abstract: Contracture is assessed by the palpation performed by a physician or physical therapist based on the range of motion and slidability of a joint. However, the diagnosis of contracture depends on the experience and skills of the therapist and is subjective. Quantification could improve the efficiency and accuracy of the diagnosis. Therefore, we aim to quantify contracture palpation using a wearable tactile sensor by focusing on the frictional vibrations that are generated by the disturbance in the slidability during palpation. In a previous study, the pulse density that was measured by detecting a pulse from the sensor output at a threshold exhibited a good relationship with the subjective rating of the frictional vibration. However, as this sensor measures the propagational vibrations of the skin, it is influenced by the characteristics of individual therapist's fingers. Thus, in this paper, three types of pulse detection threshold setting methods (constant, adjustment with a sine wave, and adjustment with white noise) are proposed. While conducting experiments, the same frictional vibration was applied on several participants by reproducing the frictional vibrations with a vibrator as a tactile display. The results demonstrated that the threshold adjusting method with white noise, which overlapped with the frequency band of the frictional vibrations, significantly reduced the variance in the finger characteristics.

Keywords: Wearable Computing
Abstract: Robots are widely used as unmanned vehicles in smart factories. In order to operate the robot in a known environment, the robot must be able to plan the path. The planned path must enable the robot to reach a destination from the starting point and the robot must avoid all obstacles during movement. This study uses a four-parameter logic curve path planning method that uses a closed formula solution and a curve with minimal design parameters to quickly generate an ideal path. The characteristics of the curve are used to derive the S and half-S curves as the solution path. The shortest path is used as the selection reference target to select parameters B and C for the four-parameter logic curve. For the S path, the best solution is chosen. The half -S curve is limited by the elastic end-point heading angle, which gives a unique set of parameter solutions. The generated path does not allow the robot to completely avoid obstacles so a hierarchical half-S curve path planning mechanism is used. A via point is generated at the collision point using the gradient vector for the edge of the obstacle. The planner continues to perform half-S path planning until the path planning ends.

Keywords: Wearable Computing
Abstract: A mobile node localization algorithm based on the Extended Kalman Filter (EKF) along with Radio Signal Strength Index (RSSI) information is proposed in this paper for mobile sensor networks. The localization process is two-fold: the initialization phase and subsequent localization phase. If a node receives the information broadcast from three anchor (or beacon) nodes, it will be localized initially and then enter the subsequent phase. Different from the localization methods using EKF with RSSI requiring three anchor nodes or more, the proposed method estimates the position of a localized node whether or not any anchor node is received. The simulation results indicate that the node localization rate of the proposed method outperforms. Also, more anchor nodes received, more accuracy localization results observed.

Keywords: Wearable Computing, User Interface Design, Human-Machine Interface
Abstract: This paper presents the packet processing and transmission of a field programmable gate array (FPGA) development board by using an RS-485 interface module and Verilog HDL. The proposed communication protocol was established between the sensors in the sensing layer and web server. In the sensing layer, the sensor system, which comprises a temperature sensor, warning light, and fan, controls the environment temperature. An attractive Internet of Things application system was proposed to simultaneously monitor real-time temperature information through wireless communication and the webpage. Node-RED software was used to develop the home care system because of its advantages in facilitating management and maintenance. The linkage function was written in the JavaScript language on Node-RED software. In the designed system, when the temperature reaches the preset value, the fan and warning light automatically turn on and a notification email is sent to the user. The measurement results showed that the throughput and conversion time were 0.00958 Mbps and 283.83 ms, respectively, at a clock frequency of 1 MHz and Baud rate of 9600 bps.

Keywords: Human-Machine Cooperation and Systems, Human Performance Modeling, Human-Machine Interface
Abstract: It is recognized that system trustworthiness is a hyperstructure embodied by the structural, behavioral, and system dimensions with a set of coherent attributes. We explore a theoretical framework of tripartite trustworthiness that can be applied to real-world autonomous systems. We present a formal study of the essences and mathematical models of system trustworthiness and their quantitative measurements in the contexts of autonomous and mission-critical intelligent systems where humans and machines interact in a hybrid environment.

Keywords: Virtual and Augmented Reality Systems, Human-Computer Interaction, Human-Machine Interface
Abstract: A model of integration for vibrotactile and force cues is important for facilitating human users’ task performance in human-machine systems. One of such human-machine systems is an interactive three-dimensional (3D) virtual environment (VE). In this paper, we proposed proportional likelihood estimation (PLE) as a model of integration for vibrotactile and force cues. Assuming human responses to cues as Gaussian distributions, PLE integrates these cues proportionally according to certain weighted contributions. We conducted an experiment to verify the suitability of PLE. For the experiment, we created a VE in which a human user executed interactively an identification task. The task required the user to identify visually indiscernible defects on a transmission line with a flying drone. The defects were indicated to the user through vibrotactile and/or force cues. These cues were in a co-located or dis-located setting, respectively, on the user’s right hand and/or forearm. The PLE predictions of integrating the vibrotactile and force cues were able to match the empirical observation of these combined cues. PLE also elucidated this cue integration successfully when applying to an existing dataset acquired under a different experimental condition. Further analyses revealed that the cue integration may not be entirely additive. Hence, PLE could shed a light on the cue integration for facilitating user interaction in human-machine systems, like VEs.

Keywords: Assistive Technology, Augmented Cognition
Abstract: This article proposes a Bayesian surprise minimization scheme to perform adaptive estimation for a family of linear Gaussian dynamic models. It is shown that the re-defined Bayesian surprise in linear Gaussian dynamic systems is a function of the Kalman filter parameters and plays a key role in the state-estimation process. The proposed representation of the Kalman filter illustrates that the information from the Bayesian surprise and the innovation process contributes to the estimation of the state vector and its covariance matrix. This unique approach yields a new set of linear estimation algorithms, where filtering is purely performed with respect to the Bayesian surprise. Simulation results confirm that the information in Bayesian surprise can be sufficient to achieve optimal estimation. In addition, an alternative approach is proposed to test filter consistency based on Bayesian surprise.

Keywords: Human-Machine Cooperation and Systems, Multi-User Interaction
Abstract: Angle of Arrival (AoA) approach with applications to Bluetooth Low Energy (BLE) has been recognized as an effective indoor localization method because of its ability for position determination with low estimation error. However, there are several issues including Carrier Frequency Offset (CFO), multipath effect, Inter-Symbol Interference (ISI), noise, and phase shifting faced by the AoA. To tackle these issues, we first highlight the wireless signal model in BLE standard and formulate the transmitted signal, wireless channel model, and the signal received by Linear Antenna Array (LAA). In addition, the paper introduces a novel fusion processing technique to eliminate the destructive impact of the wireless channel on the received signal, which leads to accurate angle detection following precise position estimation. The effectiveness of the proposed fusion processing method is evaluated through an experimental testbed in the presence of noise and Rayleigh fading channel. Based on the simulation results, the proposed processing approach illustrates significant improvements in the angle detection and path tracking in companion to its counterparts.

Keywords: Assistive Technology, Affective Computing, Human-Computer Interaction
Abstract: This paper addresses the evaluation of the performance of face and facial expression biometrics through a decision support system. The evaluation criteria include capturing the risk of the system, estimating the reliability of decision, and predicting the change in the perceived operator's trust in the decision. The relevant applications include human behavior monitoring and stress detection in individuals and teams, and in situational awareness system. Using an available database of cross-spectral videos of faces and facial expressions, we conducted a series of experiments to 1) demonstrate the phenomenon of biases in biometrics that affect the evaluated measures of the performance in human-machine systems, 2) explore the overall risk of the system caused by error rates such as false match and false non-match rates, and 3) calculate the reliability of cross-spectral and emotion-varying face identification.

Keywords: Model-based Systems Engineering, Decision Support Systems, Intelligent Assistants and Advisory Systems
Abstract: This paper attempts to conduct analysis on the WHO dataset to produce predictive analysis applying different machine learning regression approaches such as decision trees, LSTM, and CNN regressor. The primary data has 91 entries, which consists of data of various countries with respect to dates along with confirmed cases, confirmed deaths, and recovered cases. The dataset has been divided into 70:30 in which 70 percent is used for training and validation, and 30 percent is used for testing. The coronavirus disease outbreak started in 2019, arising in Wuhan, China. The key objective is to exercise different artificial intelligence approaches, we ought to predict the confirmed cases, confirmed deaths, and recovered cases, and further, various visualization techniques have been used to deduce the meaningful inferences from the model’s prediction and perform specific analytics on the results concluded. The prediction models such as LSTM and CNN are evaluated on the basis of several loss functions such as R2 score and Mean Squared Error.

Keywords: Social Network Systems
Abstract: The coronavirus disease, also known as the COVID-19, is an ongoing pandemic of a severe acute respiratory syndrome. The pandemic has led to the cancellation of many religious, political, and cultural events around the world. A huge number of people have been stuck within their homes because of unprecedented lockdown measures taken globally. This paper examines the reaction of individuals to the virus outbreak-through the analytical lens of specific hashtags on the Instagram platform. The Instagram posts are analyzed in an attempt to surface commonalities in the way that individuals use visual social media when reacting to this crisis. After collecting the data, the posts containing the location data are selected. A portion of these data are chosen randomly and are categorized into five different categories. We perform several manual analyses to get insights into our collected dataset. Afterward, we use the ResNet-50 convolutional neural network for classifying the images associated with the posts, and attention-based LSTM networks for performing the caption classification. This paper discovers a range of emerging norms on social media in global crisis moments. The obtained results indicate that our proposed methodology can be used to automate the sentiment analysis of mass people using Instagram data.

Keywords: Social Network Systems, Model-based Systems Engineering
Abstract: The existing opinion dynamics models based on the impact of opinion leaders tend to only consider the impact of opinion leaders on normal individuals, but ignores the impact of normal individuals on opinion leaders. Normal individuals in social network can also change the opinions of opinion leaders by independent affection mobilization. In this paper, take into account affection mobilization, an affection mobilization leadership index (AMLI) is used to identify opinion leaders and influential agents who can mobilize affection independently. A novel opinion dynamics affection mobilization-based Hegselmann–Krause model (AMHK) is then proposed. Extensive experiments on both artificially generated and real-trace network datasets verify the effectiveness and efficiency of the proposed model. Appropriate proportion of influential agents can promote the reach of consensus, while excess influential agents could lead a consensus with fragmented opinions, which provides a significant train of thought in guiding public opinion.

Keywords: Cooperative Systems, Systems Biology, Decision Support Systems
Abstract: In this paper, a robust Pareto suboptimal strategy for an uncertain susceptible-infected-recovered (SIR) model with state delay is investigated using the static output feedback (SOF). After linearizing the original nonlinear SIR model, a sufficient condition for the existence of a proposed strategy set is derived in terms of higher-order cross-coupled matrix equations (HCMEs). Because the concept is based on the guaranteed cost control technique, robust stability and the existence of the cost bound are both attained. To avoid the treatment of HCMEs, a recursive algorithm based on the linear matrix inequality (LMI) is discussed. Finally, a practical SIR time-delay model is performed to demonstrate the effectiveness and reliability of the proposed strategy.

Keywords: Systems Medicine, Systems Biology, System of Systems
Abstract: Deep brain stimulation (DBS) for Parkinson's disease, essential tremor and epilepsy is an established palliative treatment. DBS uses electrical neuromodulation to suppress symptoms. Most current systems provide a continuous pattern of fixed stimulation, with clinical follow-ups to refine settings constrained to normal office hours. An issue with this management strategy is that the impact of stimulation on circadian, i.e. sleep-wake, rhythms is not fully considered; either in the device design or in the clinical follow-up. Since devices can be implanted in brain targets that couple into the reticular activating network, impact on wakefulness and sleep can be significant. This issue will likely grow as new targets are explored, with the potential to create entraining signals that are uncoupled from environmental influences. To address this issue, we have designed a new brain-machine-interface for DBS that combines a slow-adaptive circadian-based stimulation pattern with a fast-acting pathway for responsive stimulation, demonstrated here for seizure management. In preparation for first-in-human research trials to explore the utility of multi-timescale automated adaptive algorithms, design and prototyping was carried out in line with ISO risk management standards, ensuring patient safety. The ultimate aim is to account for chronobiology within the algorithms embedded in brain-machine-interfaces and in neuromodulation technology more broadly.

Keywords: Assistive Technology, Human-Machine Interface, Wearable Computing
Abstract: Multiarticulate bionic arms are now capable of mimicking the endogenous movements of the human hand. 3D-printing has reduced the cost of prosthetic hands themselves, but there is currently no low-cost alternative to dexterous electromyographic (EMG) control systems. To address this need, we developed an inexpensive (~675) and portable EMG control system by integrating low-cost microcontrollers with a six-channel surface EMG (sEMG) acquisition device. Using this low-cost control system, we quantify, in a pilot study, the performance of a common EMG-based control algorithm–the modified Kalman filter (MKF)–when computational resources and electrode count are limited. We also demonstrate the ability to provide proportional and independent control of various six-degree-of-freedom prosthetic hands in real-time using the MKF. We found no significant differences in the signal-to-noise ratio (SNR) of the low-cost control system and that of a high-end research-grade system (paired t-tests). We also found no significant difference in the Root Mean Squared Errors (RMSEs) of predicted hand movements for the low-cost control system and that of the research-grade system when using only six sEMG electrodes. We then demonstrate that the SNR of the low-cost control system is statistically no worse than 44% of the SNR of the research-grade system (equivalence tests). Likewise, we demonstrate that RMSEs were typically a few percent better than, and statistically not more than 6% worse than, RMSEs of a research-grade system. This held true even when controlling up to six degrees of freedom on a prosthetic hand. Despite minimal computational resources and only six sEMG electrodes, the system performs satisfactorily and highlights the practicality and efficiency of the modified Kalman filter for dexterous EMG-based control. Successful deployment of this low-cost control system constitutes an important step towards the commercialization and wide-spread availability of dexterous bionic hands.

Keywords: Human-Machine Interface, Brain-based Information Communications, Human Factors
Abstract: To quantify mental effort and track neurophysiological changes due to powered-robotic-exoskeleton based gait training objectively, we investigated the feasibility of using functional connectivity and neural efficiency metrics obtained from functional near infrared spectroscopy to monitor neurophysiological changes during powered robotic exoskeleton-based gait training in two stroke and two spinal cord injury (SCI) patients. Increased functional connectivity between different brain regions were associated with improved gait performance in stroke patients but indicated increased mental workload with no gait changes in SCI patients. Neural efficiency provided cost of maintaining motor performance in all four patients. Both metrics show potential in tracking mental effort and gait training progress and may serve as valuable inputs to rehab exoskeleton brain computer interfaces and contribute to the development of personalized rehabilitation programs.

Keywords: Human-Machine Cooperation and Systems, Human Performance Modeling, Human-Machine Interface
Abstract: As Intelligent Transport Systems (ITS) advances, more and more people will have the opportunities to drive vehicles with autonomous capabilities. This rise in number of semi-autonomous vehicles also gives rise to several challenges with regards on how human factors come into play in interacting with the vehicle's Automated Driving System (ADS). One important interaction of an ADS with Level 3 Conditional Automated Driving capabilities is Request-to-Intervene (RTI), which alerts drivers to takeover the vehicle during an automated driving session, however, the driver is not necessarily ready to receive the authority. To see whether an ADS can detect the readiness of the user for RTI, in this preliminary study we evaluated the mental states of ADS users in naturalistic driving conditions by comparing them with those of drivers and passengers. The mental states were evaluated by measuring their heart rate and by calculating specific features of Heart Rate Variability (HRV), specifically NN50 and pNN50 indices, during driving events (turning, lane changing, and stopping) and no-events. The results showed the NN50 and pNN50 values of manual driving were significantly different from those of ADS driving and passenger, suggesting that ADS driving has a higher level of relaxed state. In addition, events such as lane-changing in the ADS driving did not induce significantly different NN50 and pNN50 from non-event situation, which may imply the participants did not pay attention to such events.

Keywords: Human-Computer Interaction, Virtual and Augmented Reality Systems
Abstract: Recent work in prosthetic devices suggests that forearm surface electromyography (sEMG) is a promising technology for human-computer interactions. Specifically, a system able to detect individual finger movement can have many clinical and non-clinical applications. Popular consumer-grade sEMG armbands are limited by their fixed electrode arrangement, which can negatively affect the classification of subtle finger gestures. We propose a low-cost, 3D-printed armband with fully adjustable electrode placement for the detection of single-finger tapping motions. We trained machine learning classifiers on features extracted from eight-channel sEMG signals to detect movement from nine fingers. We obtained a classification accuracy of 71.5 ± 1.1% for a K-Nearest Neighbours (KNN) classifier using features extracted from 500 ms windows of sEMG data. Moreover, a KNN model trained on 200 ms windows from a subset of particularly clean data obtained an accuracy of 93.0 ± 0.5%. We also introduce a novel haptic feedback mechanism to improve user experience when using the armband, and propose an augmented reality typing interface as a potential application of our armband.

Keywords: Human-Computer Interaction, Human-Machine Interface
Abstract: In this work, we focused on end-to-end speech recognition for less-resourced language, Amharic. The result can be integrated with other tasks such as spoken content retrieval. We explored three models, which consist of Convolutional Neural Networks, Recurrent Neural Networks, and Connectionist Temporal Classification, towards end-to-end speech recognition on less-resourced language. Further, we studied the possibility of having an end-to-end system with 1-best output keeping the network parameters and computational resource minimal. The paper gives attention to finding a more suitable sub-lexical unit for the Amharic end-to-end speech recognition system which can be used as an audio indexing unit. We present the first result comparing grapheme, phoneme, and syllable-based end-to-end speech recognition systems for our target language. The models are evaluated on approximately 52 hours of Amharic speech corpus containing read-speech, audiobooks, and multi-genre radio programs. On the test set, we report a character error rate (CER) of 19.21% and a syllable error rate (SER) of 39.98% for a syllable-based end-to-end model without lexicons and language model integrated.

Keywords: Human Performance Modeling, Mental Models, Wearable Computing
Abstract: Mental workload assessment is of great importance for safety critical applications, especially in situations that involve physical demands, such as with first responders (e.g., paramedics, firefighters, or police officers). Advancements in physiological signal monitoring with wearable sensors have made way for real-time mental workload assessment using physiological signals. However, these models have typically been conducted in controlled laboratory settings and rely on a single physiological modality. As a result, such models often experience a drop in performance due to movement artifacts introduced in real-life conditions. In this paper, we demonstrate that a multi-modal mental workload model not only improves measurement accuracy, but can also increase robustness against physical activity artifacts. To this end, an experiment was conducted where mental workload and physical activity levels were modulated simultaneously while physiological data was collected from 48 participants using off-the-shelf wearable devices. Results show improved mental workload assessment with multi-modal fusion under varying physical activity conditions.

Keywords: Human-Computer Interaction, Human Factors, Human Performance Modeling
Abstract: Argentine tango has been shown to help psychological and physical health by reducing perceived levels of depression and stress, similar and at times better than meditation. An often reported experience in Argentine tango is "flow", which is described as total involvement from the dancers. While this state has been self-reported by experienced dancers while dancing, it has yet to be quantified in real-time. However, with the emergence of portable and wearable devices for the acquisition of physiological signals such as electroencephalography (EEG) and electrocardiography (ECG), and recent innovations in EEG artifact removal algorithms, this quantification may now be possible. In this work presents a case study where we aim to first validate the potential of recording usable EEG and ECG data from dancers while dancing, in an unobtrusive manner, as well as investigate the existence of neurophysiological correlates of Argentine tango flow.

Keywords: Brain-based Information Communications, Human-Computer Interaction, Human-Machine Interface
Abstract: Electroencephalography (EEG) signals do not show electrical neuronal activity at source level. Thus, in EEG-based brain machine interface, information relatively easy to occur, such as motor imagery, has been used. Independent component analysis is a method to estimate multiple sources from EEG signals. If independent components can take some parts encoded by movement direction, it could improve classification performance for brain-machine interface. Therefore, this study aimed to find features contributing to specific direction in planning phase from independent components. Subjects performed a reaching task and were instructed to move their hand to one of five directions. Independent components were calculated from EEG signals during planning phase. We constructed feature vectors of each independent component with analysis of variance (ANOVA) and performed direction classification. Our results showed that using event-related potentials of independent components, classification accuracy were higher than chance level. Independent components that achieved high performance were associated with local region in the brain. Times and duration contributing to high performance were not identical to one another. We confirmed that independent components related to intended direction can be extracted and contributed to high accuracy.

Keywords: Affective Computing, Wearable Computing, Brain-based Information Communications
Abstract: For thousands of years in the history of our human society, people are inevitably segregated by long-distances. No matter whatever the reasons are, due to working, studying, visiting, traveling, or even the self-isolations as a result enforced by pandemic diseases, we are always separated with our closest friends, families and/or loved ones all the times in our life. There are no effective ways to bond us all together while we are away from each other. Jean Joseph 2.0 is an ergonomic, sleek, and human-centered non-invasive neurohaptic interface that turns human emotion brain signals into physical touch stimulations and synthetic images. It allows “speech-free”, "typing-free” remote communications with your friends by direct brain-to-brain telepathy. While wearing our earbud-like simple brain-computer interface and the robotic haptic suit/armband, the developed BCI biosensing algorithms can interpret EEG signals, and the deep Generative Adversarial Network (GAN) will translate the “feelings of missing someone” into perceivable images and haptic sensations conveyed remotely to your friends. This method provides on-the-fly telepathy and peaceful feelings when people are segregated from their families, close friends, or loved ones. The only thing they have to do is by just thinking “I miss you”, without additional fumbling of speaking on the cellphones, using hands to open the Apps or typing text messages in order to communicate with each other.

Keywords: Human-Machine Interface, Virtual and Augmented Reality Systems, Brain-based Information Communications
Abstract: Measuring saccadic eye movements when wearing a virtual reality (VR) head-mounted display (HMD) has recently gained a lot of attention, as it allows for enriched user experiences. This has led to an increase in devices showcasing camera-based eye tracking capabilities. Such devices, however, can be orders of magnitude more expensive than conventional off-the-shelf HMDs. In this study, we explore the use of low-cost sensors embedded directly into the faceplate of the HMD to measure electroencephalography (EEG) and electrooculography (EOG) signals. In a ''do-it-yourself'' manner, we rely on the openBCI biosignal amplifier for data acquisition. A 7-channel system was tested on four participants who attended visually to a moving target in their field-of-view that moved every 10 degrees over a circumference. Time series and handcrafted features were extracted from the measured ExG signals and served as input to two different classifiers: support vector machine (SVM) and a multilayer perceptron (MLP). A hierarchical classification approach was proposed and found to achieve the best results with the fusion of both features sets, resulting in an average accuracy of 76.51% with an SVM. The results are encouraging and suggest that accurate, low-cost classification of saccadic eye movements may be possible.

Keywords: Heuristic Algorithms, Optimization, Swarm Intelligence
Abstract: Nowadays, Unmanned Aerial Vehicles (UAVs) are playing increasingly important roles in agriculture, rescuing and surveillance due to their small size, low cost and strong adaptability. Coverage Path Planning (CPP) is a fundamental problem for UAV applications, which means to find a path covering all the targets or regions of interest. Researches on CPP in a single region have been studied for decades, but rare to be devoted to covering multiple scattered regions of multiple UAVs. This paper proposes an attempt to solve this problem in a short time and take the task balance of multiple UAVs into account meanwhile. This work constructs a model for multiple UAVs to cover scattered regions firstly. In view of the high computational complexity of the precise solution, we keep innovating a heuristic measure based on the model to make the solving process feasible. To settle the problem of imbalanced time consumption caused by super regions, we improve the previously built model furthermore. A series of experiments validated that the presented approaches exhibit the effectiveness and balance of time consumption in diverse scenarios.

Keywords: Swarm Intelligence, Neural Networks and their Applications, Knowledge Acquisition in Intelligent
Abstract: In large-scale multi-agent systems, the existence of a large number of agents with different target tasks and connected by complex game relationships causes great difficulty for policy learning. Therefore, simplifying the learning process is an important issue. In multi-agent systems, agents with the same target tasks or attributes often interact more with each other and exhibit behaviors more similar. That means there are stronger collaborations between these agents. Most existing multi-agent reinforcement learning (MARL) algorithms expect to learn the collaborative strategies of all agents directly in order to maximize the common rewards. This causes the difficulty of policy learning to increase exponentially as the number and types of agents increase. To address this problem, we propose a goal-based hierarchical group communication (GHGC) algorithm. This algorithm divides the agents into different groups, and maintains the group’s cognitive consistency through knowledge sharing. Subsequently, we introduce a group communication and value decomposition method to ensure cooperation between the various groups. Experiments demonstrate that our model outperforms state-of-the-art MARL methods on the widely adopted StarCraft II benchmarks across different scenarios, and also possesses potential value for large-scale real-world applications.

Keywords: Swarm Intelligence, Optimization, Heuristic Algorithms
Abstract: The virtual machine (VM) placement problem with the objective to save energy consumption and improve machine utility has been studied extensively in Cloud computing. However, the connection information among VMs during the execution of scientific workflows is seldom considered in existing studies. Therefore, this paper intends to build a novel connection-aware model for VM placement in scientific workflows. Different from existing studies, as the connection information of VMs is considered following the topology of workflows, not only the CPU capacity and memory capacity but also the transmission bandwidth among machines should be considered. An energy-aware, traffic-aware, connection-aware ant colony optimization (ETCACO) approach is developed. The proposed ETCACO combines Ant Colony Optimization (ACO) with a scheduler, namely greedy placeman. Experiments are performed to compare the proposed model with the traditional approach. It is discovered that by taking the connection information into consideration, the proposed approach can reduce energy consumption by 7%.

Keywords: Swarm Intelligence, Neural Networks and their Applications, Machine Learning
Abstract: One of the toughest challenges in the multi-agent deep reinforcement learning (MADRL) is that when the opponents' policies change rapidly, the collaborative agents can't learn well to respond to the opponents' policies effectively. This may lead to a local optimum w.r.t. the learned policy of the collaborative agents may be only locally optimal to the opponents' current policies. To address this problem, we propose a novel algorithm termed Friend-or-Foe Deep Deterministic Policy Gradient (FD2PG), in which the cooperative agents can be trained more robust and have stronger cooperation ability in continuous action space. These collaborative agents can generalize easily and respond correctly, even if their opponents’ policies alter. Inspired by the classic Friend-or-Foe Q-learning algorithm (FFQ), we introduce the idea of minimizing the foes and maximizing the friends into the centralized training distributed execution framework, multi-agent deep deterministic policy gradient algorithm (MADDPG), to enhance collaborative agents' robustness and cooperativity. Besides, we introduce a Minimax Multi-Agent Learning (MMAL) method to explore two special equilibriums (the adversarial equilibrium and the coordination equilibrium), which can guarantee the convergence of FD2PG and improve optimization. Extensive fine-grained experiments, including four representative scenario experiments and two scale-performance correlation experiments, were conducted to demonstrate the superior performance of FD2PG comparing with existing baselines.

Keywords: Machine Learning, Neural Networks and their Applications, Machine Vision
Abstract: Due to the particularity of SAR image, existing SAR image classification models often lack strong robustness against noise. Moreover, SAR images are naturally prone to speckle noise and sensitive to observed azimuth. To solve these problems, in this paper, we propose a novel fusion framework in which the convolutional layer with increased stride is used to replace the max pooling layer. Unlike max pooling layer roughly extracts the maximum pixel value in one region as its main feature, which is easy to introduce noise, convolution operation can update the weights and learn features more rationally by back-propagation. It also can achieve the same purpose of down sampling as pooling layers. In order to make full use of feature maps from different layers, our framework fuses the feature vectors extracted from different layers, which helps improve the performance of our classification model. For the problem of overfitting caused by the small MSTAR dataset of SAR images, we replace fully connected layers with convolution layers to relieve the overfitting of the convolution layers by reducing the number of parameters. In order to improve the robustness against observed azimuth angles of the dataset, we adopt the multi-channel calibration and superposition as model’s input, which can be used in real flight platform. The extensive experiments conducted on the MSTAR dataset have clearly demonstrated that our framework achieves higher classification accuracy, stronger robustness against noise than other existing methods, as well as its excellent classification performance for the targets of the same category and different subcategories, which is more difficult to be classified.

Keywords: Machine Learning, Neural Networks and their Applications, Optimization
Abstract: Activation function is a crucial component in artificial neural networks for its capability of converting linear function of input to complex nonlinear expression. It also plays an important role generating enhancement nodes in broad learning system(BLS). In this paper, we perform the comparison of 20 popular activation functions on different datasets in classification and regression. Among all selected activation functions, sigmoid leads to faster training process and greater approximation capability than others in general tasks. Meanwhile, the statistical analysis demonstrates that the type of activation function does not affect the performance of BLS too much. Afterwards, we assemble some best-performing activation functions to form a combination within convex restriction, which achieves better performance than corresponding base activation functions in standard BLS.

Keywords: Machine Learning, Neural Networks and their Applications, Optimization
Abstract: System verification is a critical process in the development of engineered systems. Engineers gain confidence in the correct functionality of the system before it is deployed into operation by executing verification activities. Choosing the right set of verification activities at the right system development stage, that is, designing a verification strategy (VS), is essential to balancing information discovery and verification cost. Only recently, quantitative methods have been proposed to support the design of verification strategies. However, their applicability in real-life scenarios is impractical due to their limited computational efficiency in the high dimensional solution space of the VS selection problem. This paper presents a reinforcement learning (RL) approach to search for a near-optimal VS. Specifically, the VS design problem is formulated as a Markov decision process (MDP) in which a value function is required. Then we combine tree search and a neural network (NN) to design a RL algorithm. In the RL algorithm, the value function is approximated as a NN that is trained in an iterative way. The near-optimal VS can be generated from the trained NN. A case study is presented to show the superiority of the proposed method.

Keywords: Machine Learning, Neural Networks and their Applications, Optimization
Abstract: Fraud detection systems aim to process a massive amount of data at high speed. To address the issues of data scalability, we introduce a chunk-based incremental classification approach based on a neural network (MLP) and a memory model to tackle the stability-plasticity dilemma. The incremental approach adapts the fraud model sequentially with incoming data chunks and retains past chunks a little more. We employ a large-scale credit-card fraud dataset that we organize into initial and incremental chunks for training and testing. Using data sampling, we solve the data skew problem, a critical issue in fraud detection. After each incremental phase, we evaluate the performance of the adjusted MLP classifier using the testing chunk. The experimental results demonstrate the effectiveness and efficiency of our incremental method and its superiority to the non-incremental MLP.

Keywords: Machine Learning, Optimization
Abstract: We often observe manifolds embedded in a high-dimensional space, which are generated by a latent low-dimensional system via non-linear measurements. If we obtain push-forward metric tensors of the manifold, we can reconstruct a data representation which is intrinsic and isometric with respect to the latent geometry. For that purpose, assuming samples can be drawn from Gaussian distributions in the latent low-dimensional system, we study methods to estimate metric tensors. If we don't have enough amount of samples, the estimation will be failed. One solution to that problem is artificial neural networks, however it can be time-consuming. So, we propose a kernelized linear model which is easy to be implemented and has capability of estimating the metric tensor under the insufficient circumstance. Estimation capability of the proposed method is evaluated in the numerical experiment.

Keywords: Machine Learning, Machine Vision
Abstract: Conventional methods for object detection usually require substantial amounts of training data and annotated bounding boxes. If there are only a few training data and annotations, the object detectors easily overfit and fail to generalize. It exposes the practical weakness of the object detectors. On the other hand, human can easily master new reasoning rules with only a few demonstrations using previously learned knowledge. In this paper, we introduce a few-shot object detection via knowledge transfer, which aims to detect objects from a few training examples. Central to our method is prototypical knowledge transfer with an attached meta-learner. The meta-learner takes support set images that include the few examples of the novel categories and base categories, and predicts prototypes that represent each category as a vector. Then, the prototypes reweight each RoI (Region-of-Interest) feature vector from a query image to remodels R-CNN predictor heads. To facilitate the remodeling process, we predict the prototypes under a graph structure, which propagates information of the correlated base categories to the novel categories with explicit guidance of prior knowledge that represents correlations among categories. Extensive experiments on the PASCAL VOC dataset verifies the effectiveness of the proposed method.

Keywords: Machine Vision, Image Processing/Pattern Recognition, Neural Networks and their Applications
Abstract: We propose an efficient and robust segmentation network for automated seagrass region detection. The proposed network has a simple architecture to save computational demands as well as inference energy cost. More importantly, the scale of network can be feasibly adjusted, to balance the network computational demands and segmentation accuracy. Experimental results show that our proposed network is robust to segment the various seagrass patterns with 90.66% mIoU (mean Intersection over Union) accuracy. It had achieved 200 frames per second (FPS, 1.42 times faster than the second-best network GCN) on desktop GPU, and 18 FPS on NVIDIA Jetson TX2. It also has 3.45M parameters and 0.587 GMACs FLOPs (FLoating Point OPerations), only 14.6% and 10.8% of those in GCN respectively. To segment a single image on the Jetson TX2, our architecture requires an average energy of 0.26 Joule. This energy cost is only 46% of DeepLab, which shows the proposed network to be an energy efficient one. The proposed network demonstrates accurate and real-time segmentation capability, and it can be deployed to low-energy embedded AUVs for sea habitat protection.

Keywords: Machine Vision, Image Processing/Pattern Recognition, Neural Networks and their Applications
Abstract: With the development of deep learning and the emergence of massive video data, object tracking has great application prospects in many fields. However, most tracking algorithms can hardly get top performance with real-time speed. In this paper, we improved tracking model based on adversarial learning and to accelerate feature extraction we proposed an efficient and accurate method. We also present a Precise ROI Pooling (PrROIPooling) based algorithm for extracting more accurate representations of targets. Furthermore, a novel regularization term is defined to ensure the similarity between the generated features and the real features. Finally, the improved objective function with modulating factors is designed to handle the problem of imbalance in the number of positive and negative samples. Extensive experiments on three datasets have demonstrated our effectiveness and achieved competitive results compared with state-of-the-art methods.

Keywords: Machine Vision, Image Processing/Pattern Recognition, Neural Networks and their Applications
Abstract: Object tracking method based on Siamese Network has shown great performance, which expressed the tracking problem as dependency relationship between target template x and search template z. How to extract the characteristics of the target is a crucial problem. We explicitly model channels interdependencies within modules and adaptively calibrate the feature response between channels to enhance the representation of the entire network. To enable the network to perform feature recalibration, global information is learned to selectively enhance useful features and suppress few less useful features. We proposed a Siamese Network tracker based on SE-ResNet-50. In the training phase, meta-learning was introduced to solve the problem of lack of training data, so that the model can achieve a better result on an extremely small amount of data. Our method overcame the limitations of Siamese Network through weighted cross-correlation. Experimental results illustrate our algorithm is much better than some popular trackers in solving the problems of occlusion, illumination change and object deformation.

Keywords: Machine Vision, Machine Learning, Image Processing/Pattern Recognition
Abstract: In this paper, we propose a distribution based feature mapping technique to improve the baseline accuracy of SVM kernels in different computer vision tasks. The proposed technique is based on learning parameters from the data and use that parameters to make inference from new data. The learned parameters can be thought of as prior knowledge about the data representation. Utilizing such prior knowledge about the data distribution increases the discriminative power of the classifier. Our proposed feature mapping technique is based on inverted Dirichlet, generalized inverted Dirichlet and inverted Beta Liouville distributions. These distributions are efficient in modelling semi-bounded data which are prevalent in computer vision problems. Our experimental results demonstrate the effectiveness of the proposed method in texture recognition, natural scene recognition and human action recognition in videos.

Keywords: Machine Learning
Abstract: In this study, a comparison of time-series modeling with linear and nonlinear ML tools is performed for fresh produce (FP) yield forecast. The consecutive monthly weather and yield dataset of Oxnard, California corresponding to the years 2007 to 2014 are applied for models’ development and training by considering the different combinations of predictors. The forecast performance is then evaluated on the next two years ahead. The sensitivity analysis is conducted as the preprocessing approach to determine the effective lag-time of the predictors. Results reveal the efficiency of time-series analysis and modeling in FP yield forecast as the implementation of autoregressive predictors along with the exogenous variables significantly improves the forecast accuracy.

Keywords: Machine Learning, Neural Networks and their Applications, Computational Intelligence
Abstract: The importance of imputation for having highly performing prediction models is highlighted in this work. Three imputation techniques are tested against a non-imputation approach that discards records with any missing values; the complete-case analysis (CCA). The deep learning linear memory vector recurrent neural network-RNN (LIME) imputation model is tested along with two other nondeep learning models such as the linear function and Last Observation Carried Forward (LOCF). The simple LSTM deep learning (DL) prediction model is deployed to decide the best performing imputation model, the one resulting in the lowest price and yield prediction errors. Five performance evaluation measures are utilized; the mean absolute error (MAE), the root mean square error (RMSE), R2 correlation measure along with two aggregated measures summarizing these three measures to decide the overall prediction performance; the average aggregated measure (AGM) for each considered step ahead and the average of the AGM across all considered steps ahead (AAGM). Based on AGM, it is found that the LIME imputation model leads to the best prediction performance of the simple LSTM DL model across both applications of 5 weeks ahead strawberry price and yield predictions using weather; W2P and W2Y. Therefore, the LIME imputed file is reused to train two compound DL models, Convolutional Long Short-Term Memory RNN with attention (ATT-ConvLSTM) and ATT-CNN-LSTM along with their Voting Regressor ensemble (VR). The same models are retrained with files preprocessed with the non-imputation approach, CCA. It is found that the overall AAGM of the compound DL and ensemble prediction models across all the 1, 2, 3, and 4 weeks ahead price predictions confirm that using LIME highly improves the prediction performance of the ensemble and its compound DL components. The VR ensemble price prediction performance is improved by 72% and the ATT-ConvLSTM component is improved by 89% compared to their performances without imputation; using CCA preprocessed files.

Keywords: Agent-Based Modeling, Optimization
Abstract: Agent-based modeling has been proposed to simulate real-world situations where autonomous agents take their own decisions based on simple rules and data from the environment. The strawberry market in California is a challenging example as prices can vary suddenly due to change in supplies and the fruits cannot be stored for long durations. The microeconomic theory that is expected to model this market is implemented within the simulation model to predict the strawberry price based on the difference between total supply and total demand. In this study, the observed yield of strawberry for the two main suppliers in California is considered as total supply; and for predicting the demand, different demand functions are presented. To estimate the ABM model parameters, two optimization methods are applied with Python-Netlogo. Finally, the computational results are presented to show the performance of the prediction model with directions for future research for improving the results.

Keywords: Cybernetics for Informatics
Abstract: This paper is devoted to studying the effect of false data injection attacks on the state estimation of discrete linear time-invariant systems in the presence of unknown disturbance. The proposed scheme firstly decouples the disturbance signal from the estimation error by exploiting the concepts of unknown input observers. Then, the observer gain has been designed based on the Kalman filter algorithm while a saturation term has been assigned to the output error in the update rule of the estimated states. Thanks to the saturation-limit dynamics introduced into the error dynamics of the Kalman filter-based estimation, the proposed method is applicable for the real-time applications. The effectiveness of the proposed scheme has been validated through a numerical example by taking two different scenarios into considerations. First, the comparative results show the superiority of the proposed scheme in state estimation under the presence of high-frequency measurement noise. Next, further to the high-frequency measurement noise, it is assumed that the sensed measurements are also manipulated by an adversary, leading to outliers in the measurements. As for this scenario, the attained results show how successfully the proposed scheme can mitigate the effect of the outliers in the presence of unknown disturbances.

Keywords: Machine Learning
Abstract: Data-driven diagnostic frameworks for large-scale power grid networks usually deal with a large number of features collected by means of sparse measuring devices. As a preprocessing task, dimensionality reduction methods can improve the efficiency of data-driven diagnostic methods by extracting sets of informative and relevant features from the raw data through appropriate transformations. This work is devoted to studying the applicability of various well-known dimensionality reduction techniques in combination with four classification models in diagnosing open circuit faults in smart grids. By providing a comparative study, this work aims at finding the best combination of dimensionality reduction techniques and classification models for diagnosing faults under normal, high signal-to-noise-ratio, low sampling rate, and high fault-resistance conditions. Various fault scenarios have been simulated on the IEEE 39-bus system and a rigorous analysis of the attained results is fulfilled so as to determine the best combinations under different conditions.

Keywords: Neural Networks and their Applications, Machine Learning, Computational Intelligence
Abstract: Satellites are complicated systems, and there are many interconnected devices inside a satellite that needs to be healthy to ensure the proper functionality of a satellite. Uncertainty and mechanical failure in different integral parts of the satellite pose the major threat for the satellite to remain fully functional for its expected life span. One of the most common reasons for satellite failure is the reaction wheel (RW) failure. Satellite RW fault prognosis can be formed as a two-step process. In this paper, we study the first step where a data-driven approach is used for forecasting the RW parameters that can be used to predict the remaining useful life (RUL) of a reaction wheel onboard satellite. Autoregressive integrated moving average model (ARIMA) and a type of recurrent neural network (RNN) known as the long short-term memory (LSTM) are used for time-series forecasting in this paper. Both models can predict up to a degree of accuracy, even when limited historical data is available. ARIMA works very efficiently as it can capture a suite of different standard temporal structures in time series. Still, when it comes to accuracy, LSTM provides a better regression outcome for our dataset. The results obtained by the models are very optimistic when model parameters are tuned.

Keywords: Neural Networks and their Applications, Machine Learning
Abstract: In the actual industrial production processes, various faults occur at different frequencies and the resulting fault data may be class imbalanced. This means machine learning-driven fault diagnosis methods have to learn from imbalanced data, and accordingly lead to lower diagnostic accuracy or even directly errors in identifying minority class. To solve this problem, we present a novel Minority-class Sensitive Fault Diagnosis approach, which can reduce the imbalance of data and enhance the sensitivity of our diagnostic model to minority-class samples. Specifically, we first design a new generative oversampling method by combining Wasserstein Generative Adversarial Network (WGAN) with Synthetic Minority Oversampling Technique (SMOTE) to balance the whole dataset and improve the distribution of the minority-class samples. WGAN is adopted to learn the distribution of minority-class samples and generate some minority-class samples as a supplement to the original dataset, while SMOTE is applied to the resulting dataset to further enhance the diversity of synthetic samples for weakening the influence from WGAN’s mode collapse. In addition, a deep forest based minority-class aware fault classification model is developed. First, during multi-grained scanning processes, we score the forests and select the corresponding forests with higher scores to generate feature representations for accelerating model convergence. Second, weights are introduced for different forests in cascade levels to further improve the overall performance of our fault diagnostic model. A series of experiments are conducted to testify the effectiveness of our proposed method, and the experimental results show that our approach can synthesize new minority-class samples with higher qualities and improve the diagnosis performance for minority-class samples as well as its overall classification accuracy. Meanwhile, in case of extremely imbalanced datasets, the proposed approach still maintains a relatively high recognition rate for minority-class samples.

Keywords: Machine Learning
Abstract: Electricity theft can be considered as a Nontechnical Loss (NTL) in smart grids, which is very harmful to the power system. Electricity Theft Detection (ETD) is a procedure to detect atypical behaviours in smart grids, which can be achieved via the massive amount of data that is generated by these networks due to using smart meter tools and Information and Communications Technology (ICT). Since the existing methods are not exceptionally robust to detect this type of attack, also considering the strength of the convolutional neural network (CNN), an Ensemble Deep Convolutional Neural Network (EDCNN) algorithm for ETD in smart grids has been proposed. As the first layer of the model, a random under bagging technique is applied to deal with the imbalance data, then deep CNNs are utilized on each subset, and finally, a voting system is embedded as the last part. This study has been conducted on a dataset which contains consumption information of more than 42,000 customers over 24 months. Various performance parameters containing AUC, precision, recall, f1-score and accuracy have been reported as the results.

Keywords: Computational Intelligence, Heuristic Algorithms, Optimization
Abstract: Abstract—Ultrasound elastography is a modality that is uniquely suited to augment conventional B-mode ultrasound for various clinical applications. Motion tracking plays a critically important role during image formation for ultrasound elastography. In this study, the accuracy of four motion tracking methods tailored for acoustic radiation force-based elastography (e.g. acoustic radiation force imaging, shear wave elastography) is compared. In these elastography methods, external mechanical excitation results in small tissue displacements (i.e. 5-10 micrometers). This paper compares four published motion tracking methods: a quadratic sub-sample estimation method, a coupled sub-sample estimation method, a 2-D spline-based estimator, and a 2-D autocorrelation-based motion estimator. Those four methods are evaluated using computer-simulated and tissue-mimicking phantom data. Based on our preliminary data, we find that the autocorrelation-based method is the preferred estimator without considering the lateral displacement. Overall, the splinebased estimator is superior to the other two competitors when both axial and lateral displacements are estimated. Since the spline-based estimation algorithm is considerably time-intensive, the coupled sub-sample estimation method becomes a practical alternative.

Keywords: Image Processing/Pattern Recognition, Machine Vision, Biometric Systems and Bioinformatics
Abstract: Nuclei segmentation occupies an important position in medical image analysis, which helps to predict and diagnose diseases. With the further research of deep learning, the task of nuclei segmentation has been automated. However, most existing methods require a great deal of manually marked full masks for training, which is time-consuming and labor-intensive, and can only be done by professional personnel. For the purpose of reducing the cost of labeling, we propose a weakly supervised method using generative adversarial training for segmentation of nucleus. In the case of no boundary, but only the centroid of the nucleus, the proposed method segmented the nucleus region with blurred boundaries. We first use the generative adversarial network(GAN) to generate the likelihood map of the nuclear centroid, then use Guided Backpropagation to visualize the pixels that contributes to the detection of the centroid of each nucleus, and finally obtain the segmentation mask of the nucleus by graph-cut. In addition, for the purpose of training the network better, we performed stain normalization on each pathological image. We have verified the proposed method on a multi-organ nuclei dataset. The final experiment results show that our advanced method achieves better segmentation performance than other weakly supervised methods, and can even reach the level of full supervision.

Keywords: Neural Networks and their Applications, Image Processing/Pattern Recognition, Machine Vision
Abstract: The image and video of the giant panda captured by the infrared camera in the field are of great significance to the research and protection of the giant panda. However, there are few pandas in the field and many useless data are also captured by the infrared camera, which brings many difficulties to the detection of the panda image. In order to quickly find the panda image from a large number of images, this paper proposes a special method of fast detection of giant panda based on the WL-SSD (wavelet feature single shot multibox detection) network using Spatial domain and frequency domain feature. The algorithm extracts the frequency domain features of the giant panda image through wavelet feature network. The experimental results show that the giant panda detection algorithm proposed in this paper better use of the high frequency characteristics of the giant panda texture feature, and it can find the panda image in high accuracy and detection rate from a large number of images.

Keywords: Computational Life Science, Multimedia Computation
Abstract: Monte-Carlo ray tracing, which enables realistic simulation of ultrasound-tissue interactions such as soft shadows and fuzzy reflections, has been used to simulate ultrasound images. The main technical challenge presented with Monte-Carlo ray tracing is its computational efficiency. In this study, we investigated the use of a commercially-available ray-tracing engine (NVIDIA’s Optix 6.0), which provides a simple, recursive, and flexible pipeline for accelerating ray tracing algorithms. Our preliminary results show that our ultrasound simulation algorithm accelerated by the Optix engine can achieve a frame of 25 frames/second using an Nvidia RTX 2060 card. Furthermore, we compare ultrasound simulations built on the proposed Monte-Carlo ray-tracing algorithm with a deep-learning generative adversarial network (GANs)-based ultrasound simulator and a physics-based ultrasound simulator (Field II). The proposed ultrasound simulator was able to better visualize small-sized structures while the other two above-mentioned simulators could not. Our future work includes integration of our proposed simulator with a virtual reality platform and expansion to other ultrasound modalities such as elastography and flow imaging.

Keywords: Neural Networks and their Applications
Abstract: Recent research has revealed that the output of Deep Neural Networks (DNN) can be easily altered by adding relatively small perturbations to the input pixels. All pixels have to be filtered out for defending DNN that will cause lots of computations. To reduce the computation, instead of all pixels only the pixels that their slight changes will cause the neural network to have a wrong result for prediction are found out and filtered out for defense goal. Experiment results show that our defense method has achieved about 90% defense success rate with filtering out 50 mutation pixels.

Keywords: Human-Machine Interface, Human-Computer Interaction
Abstract: The classification of motor imagery (MI) task has gained lots of attention in brain-computer interface (BCI) field and numerous studies have proposed various methods for MI classification. In recent years, time-frequency analysis method and convolutional neural network (CNN) have been combined to extract more complex features and exhibit superiors performance in comparison with conventional methods. In this paper, we presented a new classification method based on Stockwell Transform of common spatial pattern and CNN. The proposed framework with two different activation functions (ReLU and ELU) was compared with CSP-SVM and CSP-CNN methods by validation on BCI competition IV dataset I calibration data to assess the performance of the proposed method. We achieved an average classification accuracy of 81.22% (ReLU) and 81.34% (ELU), which outperformed the conventional CSP-SVM (76.45%) and CSP-CNN methods (69.29%). Further interrogation showed that the higher accuracy was attributed to improved performance in the subjects with lower detection rate by CSP-SVM. In sum, our results showed that the proposed framework can be applied to MI-BCI systems with superior performance, leading new insights towards more practical MI-based BCI applications.

Keywords: Human-Machine Interface, Human-Computer Interaction
Abstract: Most brain-computer interfaces (BCIs) utilize univariate features (i.e., power spectrum or amplitude) for motor imagery (MI) pattern recognition, while less attention has been paid on multivariate analysis that considers information flow between various brain areas through brain connectivity estimations. Most recently, researches have proved that connectivity features were able to characterize different MI tasks. In this study, we investigated the performance of functional connectivity features measured by phase lag index (PLI), weighted phase lag index(wPLI) and phase-locking value(PLV) on MI classification. The widely-used filter-bank common spatial pattern (FBCSP) approach was employed here for performance assessment. A linear support vector machine was trained to classify different MI tasks using two publicly available datasets from BCI Competition III and IV. The classification results showed that connectivity features achieved classification accuracy >85% in most cases and PLI outperformed all other methods including FBCSP. Our work suggested that functional connectivity features could be utilized as a powerful tool for recognizing different motor intention.

Keywords: Human-Machine Interface, Human-Computer Interaction, Brain-based Information Communications
Abstract: Kinesthetic Motor Imagery (MKI) has been demonstrated to be a robust paradigm for Brain-Computer Interfaces (BCI). In this paper we present the characterization of KMI paradigm of three tasks of wrist and forearm of the right arm using Hurst Exponent and variogram, preceding for ICA to map signals into source space. The results show high persistency an average of 0.76 pm 0.07 for KMI Pronation/Supination (PS), 0.82 pm 0.05 for KMI Flexion-Extension (FE), and 0.90 pm 0.02 for KMI Abduction-Adduction (AA). We found a significant difference between the three KMI tasks, useful for multimodal command in BCI.

Keywords: Brain-based Information Communications, Human-Machine Interface, Information Visualization
Abstract: We present an investigation into the relationship between steady-state visually-evoked potentials (SSVEPs) and the magnitude, distance, shape, and spatial location of the flashing stimulus relative to the participant. We use a wearable electroencephalography (EEG) device with the addition of an external occipital electrode for the experiments. SSVEP responses are extracted using the lock-in amplifier and fast Fourier transform algorithms. We then map the responses to what the human eye sees. Our experiments pinpoint the optimal range of stimulus parameters required for stable SSVEP response, identify failure states for flashing stimulus, as well as create a visual map, a vidmap, of the participant's ability to see. The results show that locating the stimulus at the participant's central vision elicits stronger SSVEP response compared to the peripheral vision.

Keywords: Human-Computer Interaction, Augmented Cognition, Virtual and Augmented Reality Systems
Abstract: Brain stimulation therapies have been established as effective treatments for Parkinson’s disease, essential tremor, and epilepsy, as well as having high diagnostic and therapeutic potential in a wide range of neurological and psychiatric conditions. Novel interventions such as extended reality (XR), video games and exergames that can improve physiological and cognitive functioning are also emerging as targets for therapeutic and rehabilitative treatments. Previous studies have proposed specific applications involving non-invasive brain stimulation (NIBS) and virtual environments, but to date these have been uni-directional and restricted to specific applications or proprietary hardware. Here, we describe technology integration methods that enable invasive and non-invasive brain stimulation devices to interface with a cross-platform game engine and development platform for creating bi-directional brain-computer interfaces (BCI) and XR-based interventions. Furthermore, we present a highly-modifiable software framework and methods for integrating deep brain stimulation (DBS) in 2D, 3D, virtual and mixed reality applications, as well as extensible applications for BCI integration in wireless systems. The source code and integrated brain stimulation applications are available online at https://github.com/oxfordbioelectronics/brain-stim-game

Keywords: Human-Machine Cooperation and Systems
Abstract: Transmission systems that enable speed, torque, and direction conversion with high responsiveness and back-drivability are strongly required for higher performance robotic systems. Engagement states can be changed by clutches, but traditional clutches cannot change the direction and provide sufficient backdrivability. On the other hand, direction conver-sion can be realized by gear box such as bevel gears, but its backdrivability is poor. Thus, we newly develop a prototype transmission system with backdrivablity and responsiveness integrating clutch and direction converter using magnetorheo-logical fluid (MRF). MRF is a functional fluid consisted of magnetic particles and carrier fluids which can change its vis-cosity rapidly and continuously according to the strength of magnetic field. MRF clutch consists of driving and driven shaft connected by vanes and coils for controlling MRF. MRF direc-tion converter consists of bevel gears, brake, and MRF for re-versing the output direction. In addition to traditional functions, such as speed and torque conversion, the proposed power transmission system provides five working modes: forward di-rection, reverse direction, and three kinds of free, according to states of the MRF clutch, MRF direction converter, and brake. Preliminary experiments revealed that the proposed transmis-sion system could adequately realize implemented functions

Keywords: Human-Machine Cooperation and Systems
Abstract: In this paper, we propose an AI-FML agent with a patch learning (PL) mechanism for the robotic game of Go applications. The proposed AI-FML agent contains three kinds of intelligence, including a perception intelligence, a cognition intelligence, and computational intelligence, for the robotic application. Additionally, we embed the PL mechanism into the AI-FML agent. The method for running PL involves three steps. It first trains an initial global model, then trains a patch model for each identified patch, and finally updates the global model using the training data that do not fall into any patch. This paper adopts the Google DeepMind Master 60 games to be the training data and testing data set. The experimental results show the AI-FML agent with the patch learning mechanism can improve the performance of regression for the robotic game of Go applications.

Keywords: Human-Machine Cooperation and Systems, Human Factors, Human Performance Modeling
Abstract: In this research, an innovative framework that taking advantage of unsupervised clustering and variable selection method is proposed for the modeling of car-following behavior, suitable for incorporating explainable microscopic traffic models into understanding driver behavior. The proposed framework retains the advantages of both conventional and data-driven method. The experimental result presented in this paper shows that the unsupervised clustering method helps identify driver behaviors naturally in an intelligible way, while variable selection has shown a good property of efficiently reducing model complexity. Especially, the proposed framework is demonstrated using real-world data collected from a sequence of instruments install on a driving vehicle in Sakae, downtown area of Nagoya city, Japan. GHR models, one of the most extensively used non-linear car-following models is calibrated against the same data and used as a reference benchmark.

Keywords: Human-Machine Cooperation and Systems, Human Factors, Human-Computer Interaction
Abstract: Future automated vehicles may be equipped with external human-machine interfaces (eHMIs) capable of signaling to pedestrians whether or not they can cross the road. There is currently no consensus on the correct colors for eHMIs. Industry and academia have already proposed a variety of eHMI colors, including red and green, as well as colors that are said to be neutral, such as cyan. A confusion that can arise with red and green is whether the color refers to the pedestrian (egocentric perspective) or the automated vehicle (allocentric perspective). We conducted two crowdsourcing experiments (N = 2000 each) with images depicting an automated vehicle equipped with an eHMI in the form of a rectangular display on the front bumper. The eHMI had one out of 729 colors from the RGB spectrum. In Experiment 1, participants rated the intuitiveness of a random subset of 100 of these eHMIs for signaling ‘please cross the road’, and in Experiment 2 for ‘please do NOT cross the road’. The results showed that for ‘please cross’, colors close to pure green were considered the most intuitive. For ‘please do NOT cross’, colors close to pure red were rated as the most intuitive, but with high standard deviations among participants. In addition, some participants rated green colors as intuitive for ‘please do NOT cross’. Results were consistent for men and women and for colorblind and non-colorblind persons. It is concluded that eHMIs should be green if the eHMI is intended to signal ‘please cross’, but green and red should be avoided if the eHMI is intended to signal ‘please do NOT cross’. Various neutral colors can be used for that purpose, including cyan, yellow, and purple.

Keywords: Human-Machine Cooperation and Systems, Human-Machine Interface, Assistive Technology
Abstract: Verification of dispensed pharmaceutical packages is of paramount importance to prescription dispensing. Due to lack of identification peripherals like bar codes or RFID tags on blister packages, image-based solutions have been utilized. One such example is HBIN where paired front and back images of a handheld package are complementarily exploited for effective identification through a two-stage structure, of which the first stage is to crop partially occluded package images of front side and back side, amid unconstrained background, then rectify and juxtapose them into a fixed size template before the second stage of identification. However, two-stage solutions require more resources for implementation and training, in addition to more computational time. In contrast, this paper presents an end-to-end trained solution, called Fast Rotated Occluded Rectangular (Fast ROR) pattern recognition architecture, composed of modules of: rotational rectangular detection, affine transformation, and image recognition. In particular, the features used to localize the package and the features used for identification are the same and extracted by a common feature extractor. As a result, the overall architecture is more compact with only one training set needed and more efficient computation time. Comparison experiments have been conducted on the proposed end-to-end FOR and on a representative two-stage HBIN solution: Targeting a pool of 230 types of pharmaceutical packages, 30 paired front and back handheld images for each type were taken and randomly partitioned with 4:1 for training and testing, FOR (vs. HBIN) uses 41.79M (120.32M) network parameters, with a training time of 17 hours (119 hours), and a testing speed of 22.2 fps (10fps). The identification results in term of of F1-score by FOR (vs. HBIN) is 100% (98.67%) in familiar environment, whereas 91.67% (91.27%) when the identification is conducted in new environment.

Keywords: Assistive Technology, Information Visualization, Human-Computer Interaction
Abstract: Expert instructors frequently use verbal instructions in motor learning coaching. The instructors find points to be improved in learners’ forms and give advice. However, the verbal expressions strongly depend on instructors’ characteristics, and the process of choosing instruction is not systematic in many fields. This study aims to develop an agent that can automatically generate proper verbal instructions from learners’ motion data, and to systematize the instructions given by various expressions. We collected shooting motion data of free throws in basketball from novice players, and verbal instructions to the novices' shooting motions from experienced players. Using multi-label classification, a type of supervised learning, we developed an agent that generates essential instructions to the novices' motions. We also suggested a possibility to extract semantic relationships among instructions with various expressions in the context of motor learning by using network structure inference in graph theory.

Keywords: Human-Computer Interaction, Kansel (sense/emotion) Engineering, Information Visualization
Abstract: In conventional text mining, a systematic analysis method based on time series information is important for topic change recognition. However, such a method has yet to be established. Furthermore, the procedure to avoid the identification problem that occurs frequently in the calculation of structural equation modeling (SEM) is complicated. This study developed a system that renders a path model. This system is based on the visual representation of the time series transition of topics inherently spoken in text data, which contains time-series information. The system can calculate the hierarchical Dirichlet allocation method and SEM for easy recalculation when an identification problem occurs in SEM.

Keywords: Information Visualization
Abstract: Visualization techniques have received a lot of attention regarding their potential to interpret and analyze the data.One of the marked visualization methods is the Parallel Coordinates Plot (PCP) utilized to high-dimensional datasets (more than three dimensions). Due to that, in visualizing large-scale datasets, the method suffers from high clutters produced from numerous intersection lines between neighboring axes, numbers of researchers have conducted techniques to boost PCPs. For instance, reducing the number of crossing lines by utilizing the re-ordering the neighboring axes in the PCP technique is a useful procedure to reduce the clutter. Motivated by this goal, the acquisition of the optimal coordinate’s order can be classified as a combinatorial optimization problem. However, in high-dimensional datasets, the optimization algorithm may face difficulty to deal with this issue. In this paper, we propose a smart mutation operator to enhance the performance of Genetic Algorithm (GA) in finding the optimal order of PCP based on diminishing the numerous intersection lines. However, any other user-desired metric can be utilized as an objective function. To assess the introduced method, we conducted a Monte Carlo simulation and several experiments to find an optimal coordinates’ order in PCP to visualize the datasets with various numbers of samples and dimensions. In the experimental results, utilizing the smart mutation represents an improvement in PCP visualization in terms of reducing the intersection lines between the neighboring coordinates compared to the original GA.

Keywords: Information Visualization, User Interface Design, Information Systems for Design/Marketing
Abstract: In this paper, we propose a novel hybrid layout called Information Eye for relational information space, which supports interactive exploration of information. It is a new visual metaphor, which uses the circular layout to represent the relationship information and the radial tree in the center to represent the attribute information. It combines the two traditional layout methods and presents the shape of the eye. This method draws on the idea of force-directed algorithm and introduces the concept of force into a circular layout. The distance from the node to the center of the circle is used instead of the node-link diagram to express the relationship, making the relationship more intuitive. In the center of the circular layout, the radial tree is used to characterize the attribute information of the nodes, which is more layered. Information Eye also introduces a round lens method, similar to the focusing function of human eyes, to realize the context change when the center of the circle is selected or not, and realizes a smooth transition of the information view. It combines multiple interactive technologies to realize the interactive exploration of two types of information. Finally, we apply this technique to movie data to illustrate its usefulness.

Keywords: Interactive Design Science and Engineering, Information Visualization, User Interface Design
Abstract: Automated production systems (aPS) are variant-rich, design-to-order systems and an increasing proportion of their functionality is implemented by control software. In control software development, software reuse is still commonly performed via clone-and-own despite many drawbacks, e.g., copying errors. This unplanned reuse leads to a high amount of historically grown software variants, which contain valuable domain expertise. Therefore, to enable planned reuse of existing control software solutions, an analysis of legacy software, inducing documentation of identified variability, is required. While so-called Software Product Lines enable the documentation of variability, they lack suitable variability visualization tailored to the needs of aPS stakeholders such as application or module developers. To address this gap, this paper introduces a variability visualization concept tailored to the needs of aPS stakeholders with the aim of supporting them in their daily tasks. The concept was evaluated successfully within a master student’s course by use of a prototypical implementation of the visualization concept.

Keywords: Human Performance Modeling, Team Performance and Training Systems, Assistive Technology
Abstract: One of the external factors that affect human intellectual activity is the concentration of carbon dioxide in the environment. Previous studies have measured individual carbon dioxide concentrations in a room, but it is not clear how carbon dioxide concentrations change in the case of multiple individuals. In this study, we used a small sensor device to simultaneously measure the carbon dioxide concentration in a room with multiple people. Furthermore, by comparing the predicted values using existing prediction models with the measured values, we will verify whether the prediction models can be used effectively in this measurement method. The experimental results show that the carbon dioxide generated by human exhalation diffuses regardless of the distance from the devices, the height difference, or the placement of the person. It was shown that the existing model for predicting carbon dioxide concentration was sufficiently useful for rooms with more than a certain number of ventilations, but in rooms with fewer ventilation, there was an error between the measured value and the actual value.

Keywords: Human-Machine Cooperation and Systems, Human-Computer Interaction, Team Performance and Training Systems
Abstract: This paper discusses the problem of having cognitive agents display an ethical behavior when simulating humans engaged in collaborative activities. It reviews briefly the notion of ethics for humans and its implementation in systems of cognitive agents, reviews some research approaches, and lists the factors to take into account for simulation of collaborative work. An example of training a medical leader constituting a preliminary approach, is used to illustrate the discourse.

Keywords: Human-Machine Cooperation and Systems, Multi-User Interaction, Team Performance and Training Systems
Abstract: In this paper, we describe an AI-based system that recognizes the activity status of several people from video streams during brainstorming meetings. Deep learning is often used to recognize video characteristics but requires a huge amount of computer resources. This makes it difficult to keep track of the activities of multiple people whose circumstances change. On the other hand, many trained models of one person's motion recognition have been developed and are available. We propose to use the existing technology but to be able to do that we need to identify a single person’s activities within a group context. This is achieved by segmenting the video and cropping the area with a person, identifying the activity using pre-existing trained models. The activity of the group is recognized by a production rule system based on individual activities. To achieve our goal, we introduce the concept of atomic action to describe activities and propose categories of atomic actions. High-level collaborative categories that indicate the status of a group during collaborative meetings are based on the CIAO model. This paper ends with the results of the first experiments we conducted using video recordings of actual students’ work sessions.

Keywords: Human-Machine Interface, Team Performance and Training Systems, Multi-User Interaction
Abstract: The concept of the knowledge transfer from a human expert to another non-expert human through technological interfaces, where a task can be learned by using brain-to-brain or body-to-body connections has great potential for future applications in which traditional verbal or visual communication channels are not available. In this paper, we present a novel approach of human-to-human knowledge transfer using a system based on functional electrical stimulation (FES). Using the proposed approach, hand-arm movements from a human teacher are recognized through an electromyogram signal classification algorithm. Using a master-slave approach, the movement signals are then translated into electrical stimulation signals and transmitted to a human learner using a functional electrical stimulation device. In the experiment conducted, we show how a human expert teaches seven learners a task that consists of associating hand-arm movements with visual stimuli presented to the learners. Furthermore, cognitive engagement was monitored during the learning process using an electroencephalogram (EEG) system. Experimental results show that four out of seven participants were able to learn the task with an accuracy over 80% and their cognitive engagement correlates to their performance.

Keywords: Information Visualization, Virtual and Augmented Reality Systems, Team Performance and Training Systems
Abstract: This paper investigates the effects of real time visual biofeedback for improving sports performance using a large scale immersive mixed reality system in which users are able to play a simulated game of curling. The users slide custom curling stones across the floor onto a projected target whose size is dictated by the user's stress-related physiological measure; heart rate (HR). The higher HR the player has, the smaller the target will be, and vice-versa. In the experiment participants were asked to compete in three different conditions: baseline, with and without the proposed biofeedback. The results show that when providing a visual representation of the player's HR or "choking" in competition, it helped the player understand their condition and improve competition performance (P-value of 0.0391).

Keywords: Assistive Technology, Human-Computer Interaction
Abstract: Interacting with long web documents such as wiktionaries, manuals, tutorials, blogs, novels, etc., is easy for sighted users, as they can leverage convenient pointing devices such as a mouse/touchpad to quickly access the desired content either via scrolling with visual scanning or clicking hyperlinks in the available Table of Contents (TOC). Blind users on the other hand are unable to use these pointing devices, and therefore can only rely on keyboard-based screen reader assistive technology that lets them serially navigate and listen to the page content using keyboard shortcuts. As a consequence, interacting with long web documents with just screen readers, is often an arduous and tedious experience for the blind users. To bridge the usability divide between how sighted and blind users interact with web documents, in this paper, we present iTOC, a browser extension that automatically identifies and extracts TOC hyperlinks from the web documents, and then facilitates on-demand instant screen-reader access to the TOC from anywhere in the website. This way, blind users need not manually search for the desired content by moving the screen-reader focus sequentially all over the webpage; instead they can simply access the TOC from anywhere using iTOC, and then select the desired hyperlink which will automatically move the focus to the corresponding content in the document. A user study with 15 blind participants showed that with iTOC, both the access time and user effort (number of user input actions) were significantly lowered by as much as 42.73% and 57.9%, respectively, compared to that with another state-of-the-art solution for improving web usability.

Keywords: Multi-User Interaction
Abstract: Authentication on smartphones is performed at the initial entry, mostly utilizing knowledge-based authentication methods which are fast and convenient. Device-based authentication does not guarantee that the user will utilize effective authentication credentials as many users choose less robust and easy to remember credentials. To reduce the explicit intervention from users and to increase user adoption, implicit authentication should be present. This approach authenticates users based on temporal access patterns to mobile devices, such as modeling the access behavior to applications. This paper presents an intelligent risk-based authentication method based on temporal access behavior to general applications on mobile devices. The risk score is calculated from the modeled pattern on the mobile device and the approach minimizes the required credentials based on the quality of this pattern. The evaluation of the presented method is achieved on real datasets and the results show the effectiveness of the approach. Importantly, the approach requires only a short period of application usage to build the model in addition to adapting to new app usage. Ultimately, the results show that the approach provides a low false acceptance rate and false rejection rate, which enhances its usability.

Keywords: Human-Computer Interaction, Virtual and Augmented Reality Systems
Abstract: Building Information Modelling (BIM) is set to revolutionize not only the planning and construction of buildings, but the entire process of urban planning and facilitation. This paper addresses the effective use of BIM in urban context based on the Geographical Information Systems (GIS) for training, planning and utilizing crisis management activities. The proposed solution integrates the domain ontologies associated with BIM and GIS at building and urban level to filter the useful data and use that filtered data to increase the public awareness for better crisis and disaster preparedness by interactive gaming. The proposed proof of concept study presents an IoT-enabled Virtual Reality (VR) environment that relies on the visual data obtained from the BIM and GIS and also sensory inputs accumulated from the building or urban layout. Such visual layouts of buildings and their surrounding areas are presented in a layered format, considering the security and privacy concerns, to identify the safe and dangerous zones on the evacuation route starting from a person’s private residential area, e.g. home, and ends at a crisis gathering point (e.g. Points of Interest, PoI). The use case presented in this study simulates an earthquake and fire scenario in a VR environment.

Keywords: Web Intelligence and Interaction, Human-Computer Interaction
Abstract: Hierarchical topic model is the method used in considering topics with hierarchical relationships. Neural embedding topic labelling (NETL) is a method utilized to label topics with neural embedding, even though it labels topics without topic relationships. The labels of hierarchical topics should have hierarchical relationship with other labels. This study proposes a method for labeling hierarchical topics with hierarchical relationships, and uses NETL to generate candidate labels for bottom topics. Moreover, our proposed method calculates how small the overlap of the candidate labels compared with other sibling topics. To label the upper topics, our method adds the label of the bottom topics and generate labels in the same way as the bottom topics recursively. Our method succeeded label hierarchical topics with labels which is more qualitative labels to consider hierarchical relationship of topics.

Keywords: Interactive and Digital Media, Human-Computer Interaction
Abstract: Having highly engaged followers on social media allows us to spread information, seek feedback, and promote a sense of community efficiently. Crafting engaging posts, however, requires careful thoughts, creativity, and communication skills. This research studied tweets and explored the effect of content types on user engagement. More specifically, we compared the number of likes and retweets between text-only and multimedia tweets. We analyzed four Twitter accounts relevant to the City of Edmonton, Canada, and performed negative binomial regressions to model the expected count of likes and retweets based on accounts, content types, and their interaction. The results showed that multimedia content increased engagement in two of the four accounts but did not change engagement significantly in the other two. In other words, multimedia content had a positive or neutral effect on user engagement, depending on accounts. Our analysis also showed the effectiveness of well-written texts in attracting the attention of users. Tweets, by design, are text-oriented, and posting multimedia content may help, but is not a necessary condition to engage with followers effectively on Twitter.

Keywords: Augmented Cognition, Assistive Technology, Human-Machine Cooperation and Systems
Abstract: This paper presents a novel distributed probabilistic framework based on movement primitives for flexible robots assembly implementation. Since modern advanced industrial cell usually deals with various tasks that are not fixed via-point trajectories but highly reconfigurable application templates, the industrial robots used in these applications must be capable of adapting and learning new skills on-demand, without programming experts. Therefore, we propose a probabilistic framework that could accommodate various learning abilities trained with different movement-primitive datasets, separately. Thanks to the fusion theory of the Bayesian Committee Machine, this framework could infer new adapting trajectories with weighted contributions of every trained datasets. To verify the feasibility of our proposed imitation learning framework, state-of-the-art movement learning framework Task-parameterized GMM is compared from several crucial aspects, such as generalization capability, accuracy and robustness. Moreover. this framework is further tested on the YUMI collaborative robot with a rivet picking assembly scenario. Potential applications can be extended to more complicated industrial assembly manufacturing or service robotic applications.

Keywords: Mental Models, Brain-based Information Communications, Human-Machine Interface
Abstract: This paper discusses representation learning from electroencephalographic (EEG) signal with deep variational predictive coding networks. We introduce a hierarchical probabilistic network that minimises prediction error at multiple levels of spatio-temporal abstraction. While the lowest layer predicts brain activity directly, higher layers abstract away from the data and predict sequences of the hidden states in lower layers. The network captures both expected and actual uncertainty by relating predicted state posteriors. Each layer minimises (expected) surprise either with or without sampling new evidence from the layer below. This structure motivates both active learning and active inference as means to learn representations. Active learning refers to model parameter exploration which allows to learn regularities, especially when they are stable between trials. Active inference refers to hidden state exploration, a process that enables dynamic inference of the current context using the learned generative model. We train the model on EEG data recorded during free reading and evaluate adaptive EEG prediction in the context of Fixation Related Potentials (FRPs).

Keywords: Human Factors, Interactive Design Science and Engineering, Affective Computing
Abstract: This paper describes experiments to learn laughter co-occurrences with dialogue contributions. The dialogue data belongs to the special type of First Encounter Dialogues where the interlocutors meet each other for the first time and where laughter mainly functions as a sign of politeness or relief of embarrassment. Earlier studies have shown that there is a correlation between the speaker's utterance content (topic) and non-verbal communication (laughter and body movement) and in this paper we seek to learn the correlations via a neural model. The results show a weak correlation in our data. The paper also concerns important aspects of the use of small datasets in interaction studies.

Keywords: Human-Machine Cooperation and Systems, Information Visualization, Human-Computer Interaction
Abstract: How can there be Human-In-the-Loop-Learning (HILL) if datasets aimed at building classifiers have ever more dimensions? We make two contributions. First, we examine the few early results on the effectiveness of HILL for building autonomous classifiers and report on our own experiment that validates the merits of HILL. Second, we introduce a HILL system (by using parallel coordinates) for learning of decision tree classifiers (DTCs). DTCs importantly emphasise the relevance of attributes and enable attribute selection, and therefore are appreciated for their transparency. The proposed system addresses a number of the shortcomings of the many HILL systems and allows for easy exploration of datasets. In particular, we incorporate parallel coordinates effectively in our tool for visualisation of high dimensional datasets. We can not only focus the learning on the accuracy of classifiers, but we can enhance performance in other important factors such as system’s interpretability and the ability to gain insight into datasets. Finally, we show the advantages of our HILL system in the application area of mobile robotics using the case study of image segmentation in robotic soccer.

Keywords: Human Factors, Assistive Technology, Mental Models
Abstract: Closed-loop validation of autonomous vehicles is an open problem, significantly influencing development and adoption of this technology. The main contribution of this paper is a novel approach to reproducible, scenario-based validation that decouples the problem into several sub-problems, while avoiding to brake the crucial couplings. First, a realistic scenario is generated from the real urban traffic. Second, human participants, drive in a virtual scenario (in a driving simulator), based on the real traffic. Third, human and automated driving trajectories are reproduced and compared in the real vehicle on an empty track without traffic. Thus, benefits of automation with respect to safety, efficiency and comfort can be clearly benchmarked in a reproducible manner. Presented approach is used to benchmark performance of SBOMP planner in one scenario and validate SuperHuman driving performance.

Keywords: Information Visualization
Abstract: Interest in visual data analytics related to many-objective optimization has recently risen. This paper introduces a novel visualization scheme based on the Radial Coordinate Visualization (RadVis) for analysis of Pareto fronts during the optimization process. This method illustrates the ranks of the Pareto front, the relative location, and the distribution of candidate solutions. The results show that the proposed method is capable of showing different ranks of Pareto fronts simultaneously. The simplicity of the P-RadVis visualization and its compatibility to work with many-objective algorithms could be beneficial in terms of visual analytics for real-time monitoring of optimization process.

Keywords: Team Performance and Training Systems, Human-Machine Cooperation and Systems, Augmented Cognition
Abstract: PV power generation has an important place in sustainable energy production. The proportion of power generation systems in various countries has increased year by year. However, PV power generation is probabilistic and cannot be accurately predicted, so in recent years it has been an important and popular research topic. This paper proposes an ANN with days tuning to predict the PV power generation of hour-ahead using every ten minutes power generation of three hours ago (a total of 19 data). The neural network has an excellent learning ability, which can easily learn when using days tuning to make prediction results more accurate and yield a slight error. Different forecasting conditions have validated the effectiveness and performance of the proposed idea. The result can trigger an early response to the power system dispatcher and solve PV power generation probabilistic and forecasting.

Keywords: Wearable Computing
Abstract: This study aims to design and develop a virtual fitness-coach information system for barbell bench press based on deep-learning Long Short Term Memory (LSTM) mechanism and wearable devices. We utilizes a set of three-axis accelerometers, gyroscopes and Electromyography (EMG) sensing modules to design our proposed wearable devices. Through computer and smartphone, the analysis and real-time assessment of the weight training in barbell free bench press can be performed to avoid injury in weight training and improve the quality of training performance. In this study, 21 subjects are recruited to use our proposed wearable devices for weight training in barbell free bench press. In the training, the subject's physiological signals and videos are captured, and the subject’s signals are extracted according to the 11 most common kinds of errors marked by the fitness instructor, including 7 posture errors and 4 kinds of muscle force errors. After the extracted signal is normalized, the data is fed for the Recurrent Neural Network (RNN) training through the Long Short Term Memory (LSTM) to classify the weight training errors. The experimental results show that the classification threshold used in the classification has the best classification result when set at 0.5, and the overall average accuracy, accuracy, recall rate, F1 Score, FPR and FNR are 91.84%, 89.25%, 88.17%, 88.18%, 6.50% and 11.83%, respectively. We found that in some categories, because the sensors are not powerful enough to capture the characteristics of the errors, the accuracy is low. While the overall accuracy of the other categories is higher than 85%. In order to accelerate the training speed of LSTM, we also try to use the common factor extraction analysis to reduce the data of accelerometers and gyroscopes from 24 to 18, 12 and 6 dimensions for training. When the total dimension including EMG is 30 dimensions, there is not much difference in the accuracy when the dimension is reduced to 24 or 18. However when it is reduced to 12 dimensions, the evaluation metrics are reduced to below 70%, and the False Negative Rate (FNR) has risen sharply to 30.21%. We therefore choose to reduce the training data from 30 dimensions to 18 dimensions to maintain

Keywords: Wearable Computing, User Interface Design, Human-Machine Interface
Abstract: In this paper, the authors propose a smart home-care system built on an Android smartphone. The database and application programming interface (API) are set up on the server side. The database collects information from various sensors and stores it, and the API acts as a bridge between the mobile phone and the database. The API prevents the leakage of private data. In the associated Android smartphone app, two functions are provided: instant monitoring based on in-home sensor data and fall detection using the three-axis accelerometer, gyroscope, and orientation sensor inbuilt into the smartphone. When the sensor data are abnormal, the remote controller is notified immediately. Moreover, the remote controller can view real-time images by using an IP camera to guarantee home safety. As for fall detection, given that falls cause severe injuries in elder people and children, the proposed app can detect a fall event, send a help message, and indicate the user’s location by using the global positioning system and Google Maps API. According to the simulation results obtained in this study, the proposed system exhibited a fall-detection sensitivity of 92.5% and specificity of 97.6%, thus proving that the system can be effectively used for home care.

Keywords: Information Systems for Design/Marketing, Augmented Cognition, Mental Models
Abstract: There has been a recent explosion of interest in digital twins, namely data driven virtual replicas that can provide insights about a physical system and support decision making. This paper deals with cognitive digital twins, namely twins that can exhibit a high level of intelligence that can replicate human cognitive processes and execute conscious actions autonomously. The paper brings together the concepts of digital twins and self-awareness and discusses how the different levels of self-awareness can be harnessed for the design of cognitive-digital twins. A discussion of digital twins in relation to the Dynamic Data Driven Application Systems (DDDAS) paradigm and a classification of digital twins based on their analytics capability are also provided.

Keywords: Human-Machine Cooperation and Systems, Information Visualization
Abstract: It is evident that corticomuscular coherence (CMC), representing the functional coupling between motor cortex and muscle tissues, plays a crucial role in neurophysiologic studies and applications. It is hypothesized that there is an unknown time delay comprising at least neural conduction time in the process of corticomuscular interaction. In this study, we developed a novel delay estimation method, defined as the rate of voxels change (RVC) for the estimation of time delay in two coupled physiological signals. The RVC is the dynamic variation of the local CMCs observed in different time offsets. Both simulation and physiological data confirm the capability of RVC in estimating cortical-muscular delay. The underlying mechanisms of individual discrepancy of the latency is also investigated via exploring the correlation between delays, brain activity and motor performance. Correlation analyses indicate an intrinsic link between the connectivity strength of the brain network and the length of time delay in cortical-muscular interactions.

Keywords: Wearable Computing, Assistive Technology, Companion Technologies
Abstract: Catastrophic forgetting is a major problem that affects neural networks during progressive learning. In it, the previously learned representation vanishes as the network learns new information. The extreme learning machine is one of the variants of the neural network. It is used in many domains due to fast training and good generalization ability. However, like other neural networks, it suffers from catastrophic forgetting and negative forward and backward transfer during the progression of neurons in incremental learning. The study hypothesizes that it is due to overlapping in hidden neurons and output weights. The global representation by an activation function further supports this hypothesis. To address this, the study proposes a neuron clustering approach to mitigate it in an adaptive class incremental extreme learning machine. The neuron clustering method activates k nearest neurons during learning and testing. It helps to partition the network to select overlapping subnetwork. Experimental results on four food datasets show that the proposed approach reduces negative forward and backward transfer when neurons are added incrementally during progressive learning.

Keywords: Human-Machine Interface, Human-Computer Interaction, Wearable Computing
Abstract: Multi-modal sensory fusion can obtain higher accuracy in gesture recognition. Its difficulty lies in mining discriminative features and fusing features from different modalities. Surface electromyography(sEMG) and ultrasound signals are typical signal modalities in gesture recognition. It is expected that the fusion of them can take advantage of the complementarity of electrophysiological information and muscle morphology information. This paper proposed two kinds of feature fusion method. The one is concatenating the manual designed sEMG and ultrasound features, and the other is a convolutional neural network (CNN) based feature exaction and fusion method for sEMG and ultrasound signals. Eight able-bodied subjects were involved to participate in the experiments. In the experiments, four channels of sEMG and A-mode ultrasound signals corresponding to 20 gestures were collected synchronously to evaluate the proposed method. The experimental results demonstrated that the fusion sEMG-ultrasound feature always outperformed the separate sEMG or ultrasound feature regardless of the feature extraction method, and as for fusion sEMG-ultrasound feature, the CNN based method achieve a high accuracy (97:38 ± 1:49%) in 20 gestures, which surpassed the method of concatenating the manual designed features and applying machine learning algorithm (LDA, KNN, SVM).

Keywords: Human-Machine Interface, Human-Computer Interaction, Kansel (sense/emotion) Engineering
Abstract: Several studies on microsaccades have been performed; however, those studies measured microsaccades under the fixation state. For practical applications, such as cognitive science, psychology, brain science, and user interface, it is ideal to measure microsaccades from the eye-tracking data that includes saccades. In this study, we propose the visual attention level based on the eye-tracking data including microsaccades after saccades. Based on the research on the visual attention level of humans, the microsaccades and pupil diameter were simultaneously measured in three experimental tasks that required different visual attention levels. Through statistical analysis, the results of the experiment indicated that the behaviors of the microsaccades and the pupil were different between the three experimental tasks, and suggested that the visual attention level can be classified by microsaccades after saccades.

Keywords: Assistive Technology
Abstract: In this study, we aim to develop a measurement system for evaluating walking ability in daily life. Health promotion is one of the most important tasks to improve quality of life and quality of community for elderly people. Disabilities related to loss of independence in performing activities of daily living can lead to their social isolation and loneliness that can induce immobility and depression, producing the vicious cycle. Various methods have been proposed to measure lower limb joint angles and positions by using wearable systems and motion capture systems. However, such systems are too expensive and big for elderly’s daily self-monitoring. This paper presents a method of lower limb joint angle estimation using a Kinect sensor. The sensor has a built-in processor to detect joint positions. However, inverse kinematics problem is required to be addressed in order to derive the joint angles. We therefore propose a hybrid approach for lower limb joint angle estimation using genetic algorithm and feed forward neural network.

Keywords: Intelligent transportation systems, Robotic Systems
Abstract: Detection and classification of road-objects like cars, pedestrians, and cyclists is the first step in autonomous driving. In particular, point-wise object segmentation for 3D point clouds is essential to estimate the precise appearances of the road-objects. In this paper, we propose SPriorSeg, a fast and accurate point-level object segmentation for point clouds by integrating the strengths of deep convolutional auto-encoder and region growing algorithm. Semantic segmentation using the light-weighted convolutional auto-encoder generates semantic prior by labeling a spherical projection image of point clouds pixel-by-pixel with classes of road-objects. The region growing algorithm achieves pixel-wise instance segmentation by taking into account semantic prior and geometric features between neighboring pixels. We build a well-balanced, pixel-level labeled dataset for all classes using 3D bounding boxes and point clouds from the KITTI object dataset. The dataset is employed to train our light-weighted neural network for semantic segmentation and demonstrate the performance of both semantic and instance segmentation of SPriorSeg.

Keywords: Intelligent Learning in Control Systems, Intelligent transportation systems, Robotic Systems
Abstract: Decision making for autonomous driving is a safety-critical control problem. Prior works of safe reinforcement learning either tackle the problem with reward shaping or with modifying the reinforcement learning exploration process. However, the former cannot guarantee the safety during the learning process, while the latter relies heavily on expertise to design exquisite exploration policy. Currently, only short-term decision makings for low-speed driving were achieved in road scenes with basic geometries. In this paper, we propose a two-stage safe reinforcement learning algorithm to automatically learn a long-term policy for high-speed driving that guarantees safety during the entire training. In the first learning stage, model-free reinforcement learning is followed by a rule-based safeguard module to avoid danger at low speed without expert fine-tuning. In the second learning stage, the rule-based module is replaced with a data-driven counterpart to develop a closedform analytical safety solution for high-speed driving. Moreover, an adaptive reward function is designed to match the different objectives of the two learning stages for faster convergence to an optimal policy. Experiments are conducted on a racing simulator TORCS which has complex racing tracks (e.g. sharp turns, hills). Compared with the state-of-the-art baselines, the results show that our method achieves zero safety violation and quickly converges to a more efficient and stable policy with an average speed of 127 km/h (3.3% higher than the best result of baselines) and an average swing of 3.96 degrees.

Keywords: Robotic Systems, Technology Assessment, Distributed Intelligent Systems
Abstract: Vision-based target localization in the air-to-ground scenes of quadcopter is a hot topic in the military and civilian applications. The current localization approaches mostly rely on the intrinsic parameters of camera which are needed to calibrate in advance. But if the calibration is inaccurate, the localization based on the focal length of camera will result in inaccuracy. Moreover, due to the autonomous flight of quadcopter, the pose changes of the quadcopter will lead to the erroneous locating results. In this paper, we propose an accurate ground-based target localization method by means of monocular vision on a quadcopter. Based on the field of view (FOV) angle of camera, we present a localization algorithm requiring no calibration. Meanwhile, we introduce the quadcopter pose superposition strategy and the IMU quaternion denoising method to eliminate the impact of quadcopter pose changes in flight. The simulation experimental results demonstrate the high performances of our locating method in both accuracy and robustness.

Keywords: Cooperative Systems, Robotic Systems, Intelligent transportation systems
Abstract: In order to fully exploit the advantages inherent to cooperating heterogeneous multi-robot teams, sophisticated coordination algorithms are essential. Time-extended multi-robot task allocation approaches assign and schedule a set of tasks to a group of robots such that certain objectives are optimized and operational constraints are met. This is particularly challenging if cooperative tasks, i.e. tasks that require two or more robots to work directly together, are considered. In this paper, we present an easy-to-implement criterion to validate the feasibility, i.e. executability, of solutions to time-extended multi-robot task allocation problems with cross schedule dependencies arising from the consideration of cooperative tasks and precedence constraints. Using the introduced feasibility criterion, we propose a local improvement heuristic based on a neighborhood operator for the problem class under consideration. The initial solution is obtained by a greedy constructive heuristic. Both methods use a generalized cost structure and are therefore able to handle various objective function instances. We evaluate the proposed approach using test scenarios of different problem sizes, all comprising the complexity aspects of the regarded problem. The simulation results illustrate the improvement potential arising from the application of the local improvement heuristic.

Keywords: Intelligent transportation systems, Smart urban Environments, Robotic Systems
Abstract: Trajectory prediction for vehicles is a popular subject since it is beneficial for efficient and secure trajectory planning. In structured traffic scenarios, the behaviour and motion of vehicles are heavily dependent on the social interaction constraints, such as road geometry and surrounding vehicles, and the kinematics model constraints, such as continuous heading and maximum acceleration. To take these factors into account, we analyse the particular characteristics of driving vehicles and propose a model that predicts the possible and feasible trajectory for host vehicle in 3 seconds. In this model, the trajectory of host vehicle takes the center-line as reference, imitates the leader vehicle and focuses on the social vehicles through attention concentration mechanism (ACM) with spatial and temporal information encoded in a fusion hidden state. Furthermore, in order to make the trajectory feasible for vehicle dynamics and kinematics, we introduce a prediction diagnosis method to check the continuous heading and maximum acceleration condition, pruning and adjusting the prediction candidates. Experiments on released public datasets show that this framework can well evaluate the traffic interactions and forecast the trajectory more accurately than common networks.

Keywords: Human-Computer Interaction, Human Factors, Virtual and Augmented Reality Systems
Abstract: This paper describes validity testing of the NeuLog NUL-217 GSR measurement device. This was accomplished by comparing the NeuLog device to readings from the Biopac Student Lab Systems EDA system. The results of this research found that the NeuLog device is significantly comparable to the Biopac system for the purposes of its intended use in psychological and technological research.

Keywords: System of Systems, Conflict Resolution, Decision Support Systems
Abstract: Cauvery River, in the southern part of India, has experienced conflict about the right to use water for the last 130 years. Historically, the states/provinces in conflict have used the water from the river for agricultural purposes. In our research, we are developing a novel system of systems approach which takes in scenario-based methods for analyzing the conflict. We have employed four methodologies which feed into each other and overlap, namely, an expert elicitation, an integrated water resources management approach to generate the streamflow requirements in the basin, a socio-political and economic impact approach which analyses the various relevant factors affecting the conflict in the region, and finally, a graph model to map out the progression of the conflict to potentially resolve. We present our methods and the preliminary findings in this paper.

Keywords: Augmented Cognition, Brain-based Information Communications, Assistive Technology
Abstract: Vigilance is a primary job-performance requirement for human operators in domains that demand sustained attention, including air traffic control (ATC), surveillance, emergency response and many others. In this pilot study we introduce the prerequisites and conditions that facilitate a novel closed-loop, adaptive neurostimulation system to alleviate vigilance decrements during prolonged time-on-task efforts. Here, we investigate the use of transcranial Direct Current Stimulation (tDCS) with preset stimulation parameters -- intensity (I), duration (t), and, probe location to augment the operators' vigilance state during an under-arousing task. To this end, we employ an app-based psychomotor-vigilance test (PVT), where performance metrics are analyzed along with physiological and cognitive bio-markers to explore opportunities toward a predictive framework. Initial observations (N=19) suggest that -- (1) a prolonged version of the PVT (40 minutes) can function both as a diagnostic and an inductive mechanism for vigilance loss, (2) tDCS can serve to restore/ improve operator vigilance states relative to baseline performance levels, and (3) short-term Heart Rate Variability (HR/V) features (3 min) and the fNIRS signal are sensitive to state changes during the PVT, and to the effects of stimulation.

Keywords: Social Network Systems, Decision Support Systems
Abstract: Anchor user identification across social networks is a classification task which determines whether a pair of accounts from different social networks belong to the same user. It is a fundamental research of information dissemination across social networks. Based on the observation that users prefer to use similar or identical display names in different social network, some researchers utilized the similarity between display names to build models. However, due to Chinese social network setting and pronunciation and font characteristics of Chinese display names, these methods do not perform well in Chinese social network datasets. To address this problem, we analyze the display name pairs of Chinese anchor users which are obtained by a crawler build in this paper. Then we define 4 special features to extract the pronunciation and font similarities. Finally, we use Gradient Boosting to establish the identification model. The experiments based on the ground-truth datasets we obtained show that these features can improve the performance of display name-based anchor user identification between Chinese social networks.

Keywords: Affective Computing, Human-Computer Interaction
Abstract: There has been much research on facial affect detection, but many of them fall short on accurately identifying expressions, due to changes in illumination, occlusion, or noise in uncontrolled environments. Also, not much research has been conducted on implementing the algorithms using multiple datasets, varying the size of dataset and the dimension of each image in the dataset. Our ultimate goal is to develop an optimized algorithm that can be used for real-time affect detection in automated vehicles. To this end, in this study we implemented the facial affect detection algorithms with various datasets and conducted a comparative analysis of performance across the algorithms. The algorithms implemented in the study included a Convolutional Neural Network (CNN) in Tensorflow, FaceNet using Transfer Learning, and Capsule Network. Each of these algorithms was trained using the three datasets (FER2013, CK+, and Ohio) to get the predicted results. The Capsule Network showed the best detection accuracy (99.3%) with the CK+ dataset. Results are discussed with implications and future work.

Keywords: Cybernetics for Informatics, Swarm Intelligence, Agent-Based Modeling
Abstract: Multi-agent pathfinding (MAPF) represents a core problem in robotics. In its abstract form, the task is to navigate agents in an undirected graph to individual goal vertices so that conflicts between agents do not occur. Many algorithms for finding feasible or optimal solutions have been devised. We focus on the execution of MAPF solutions with a swarm of simple physical robots. Such execution is important for understanding how abstract plans can be transferred into reality and vital for educational demonstrations. We show how to use a swarm of reflex-based Ozobot Evo robots for MAPF execution. We emulate centralized control of the robots using their reflex-based behavior by putting them on a screen's surface, where control curves are drawn in real-time during the execution. We identify critical challenges and ways to address them to execute plans successfully with the swarm. The MAPF execution was evaluated experimentally on various benchmarks.

Keywords: Brain-based Information Communications, Affective Computing
Abstract: Understanding the neurophysiological mechanisms underlying curiosity and therefore being able to identify the curiosity level of a person, would provide useful informa- tion for researchers and designers in numerous fields such as neuroscience, psychology, and computer science. A first step to uncovering the neural correlates of curiosity is to collect neurophysiological signals during states of curiosity, in order to develop signal processing and machine learning (ML) tools to recognize the curious states from the non-curious ones. Thus, we ran an experiment in which we used electroencephalography (EEG) to measure the brain activity of participants as they were induced into states of curiosity, using trivia question and answer chains. We used two ML algorithms, i.e. Filter Bank Common Spatial Pattern (FBCSP) coupled with a Linear Discriminant Algorithm (LDA), as well as a Filter Bank Tangent Space Classifier (FBTSC), to classify the curious EEG signals from the non-curious ones. Global results indicate that both algorithms obtained better performances in the 3-to-5s time windows, suggesting an optimal time window length of 4 seconds (63.09% classification accuracy for the FBTSC, 60.93% classification accuracy for the FBCSP+LDA) to go towards curiosity states estimation based on EEG signals.

Keywords: Human-Machine Interface, Brain-based Information Communications, Human Factors
Abstract: There is increasing interest in understanding the cognitive and physiological state of operators in safety critical situations (e.g. pilots), specifically as it relates to task difficulty and mental workload. Herein, we evaluate the potential of electrocardiography (ECG) and electroencephalography (EEG) for detecting in-flight changes in helicopter pilot workload. Two National Research Council Canada test pilots performed a series of flight maneuvers in a NRC Bell 205 helicopter which involved a target tracking task with three levels of difficulty. Subjective ratings of pilot workload were collected using the Cooper-Harper handling quality ratings scale and pilot control activity was quantified based on cyclic control movements. ECG derived measures of heart rate and heart rate variability, as well as EEG derived measures of power in three frequency bands (theta 4-8Hz; alpha 8-13Hz; beta 13-22Hz), were computed and compared across task difficulty levels. A set of support vector machine (SVM) regressors were trained and tested to differentiate the three difficulty levels from ECG and EEG features. Differences in subjective ratings and control activity metrics confirmed the task difficulty manipulations (p<0.01). ECG-derived physiological metrics were able to partially resolve differences among the task difficulty levels. Similarly, EEG-derived cognitive measures confirmed the capture of differential neural functioning levels for the task difficulty conditions in the alpha and beta bands (p<0.05), though substantial individual differences were observed between pilots. SVM regressors trained on ECG and EEG features successfully differentiated levels of workload, with the ECG-based regressor (minimum cross-validation MSEECG = 0.17) performing better than the EEG-based regressor (minimum cross-validation MSEEEG = 0.29). This study provides an initial application demonstration of physiological and cognitive metrics and machine learning approaches for detecting differences in task difficulty during helicopter flight. This is the necessary first step for further development of passive brain computer interfaces for real-time in-flight monitoring of helicopter pilot workload.

Keywords: Human-Computer Interaction, Brain-based Information Communications, Human-Machine Interface
Abstract: New research and development efforts are highlighting the ways in which electroencephalogram-based (EEG) brain-computer interface (BCI) technology can be used to improve the quality of life for healthy individuals. One such application incorporates cognitive state monitoring into passive BCI (pBCI) systems. Among the challenges facing this development, a significant barrier to adoption is the time-intensive calibration typically needed to tune the system to account for variations in neural activity patterns. An open research question is understanding the relationship between underlying user state and user performance in real-world situations and environments. However, user states are often derived and defined as a function of observed user performance for a particular analysis. Understanding the relationship between user state and user performance ideally requires the definition of user state be independent of observed user performance. This work represents our initial steps towards this goal by using cross-dataset learning, where we define user state from a dataset recorded in a highly-controlled experiment, building a subject-independent pBCI model to predict user state using deep learning approaches, and applying this pBCI model to analyze user performance in a new, unobserved dataset. We show that user performance varies smoothly across a continuum of pBCI model outputs. Our results highlight a promising approach for dealing with one of the major hurdles in the development of BCI systems for healthy users.

Keywords: Human-Computer Interaction
Abstract: Non-invasive brain imaging techniques are popular tools for monitoring the cognitive state of human participants. This work builds on our previous studies using the HydroCel Geodesic Sensor Net, 256 electrodes dense-array electro-encephalography (EEG). The studies analyze dominant frequencies of temporal power spectral densities for each of the EEG electrodes. The experiments involve three modalities: Meditation, Math Mind, and (c) Open Eyes condition. Here we perform an analysis of the Shannon entropy index and Pearson's skewness coefficient in order to test their fitness to classify different brain states. The results help to develop a comprehensive methodology to understand brain dynamics.

EEG, Cognition, Shannon Entropy, Skewness, Emotion, Meditation, Awareness, Spiritual Values, Intentionality.

Keywords: Human-Computer Interaction, Brain-based Information Communications, Human Factors
Abstract: Abstract— Using EEG signals for mental workload detection has received particular attention in passive BCI research aimed at increasing safety and performance in high-risk and safety-critical occupations, like pilots and air traffic controllers. Along with detecting the level of mental workload, it has been suggested that being able to automatically detect the type of mental workload (e.g., auditory, visual, motor, cognitive) would also be useful. In this study, we developed a novel experimental protocol in which subjects performed a task involving one of two different types of mental workload (specifically, Auditory and Visual), each under two different levels of task demand (Easy and Difficult). The tasks were designed to be nearly identical in terms of visual and auditory stimuli, and differed only in the type of stimuli the subject was monitoring/attending to. EEG power spectral features were extracted and used to train linear discriminant classifiers. Preliminary results on three subjects suggested that the Auditory and Visual tasks could be distinguished from one another, and individually from a Baseline condition (which also contained nearly identical stimuli that the subject did not need to attend to), with accuracy significantly exceeding chance. This was true when classification was done within a workload level, and when data from the two workload levels was combined. Though further investigation is required, these preliminary results are promising, and suggest the feasibility of a passive BCI for detecting both type and level of mental workload.

Keywords: Optimization, Evolutionary Computation, Computational Intelligence
Abstract: Selecting a portfolio with high return and low risk is a difficult problem that is worthy of research. When solving a portfolio optimization problem, research studies usually employ various assessing indicators. This paper utilizes the trend ratio as the measure indicator. The value of the trend ratio represents how much the return is from the portfolio per unit of risk. A portfolio selected by the trend ratio based on simple linear regression is more in accordance with investors’ psychology. This paper proposes a new investment strategy with fund allocation to solve a portfolio optimization problem, as fund allocation can make the portfolio more flexible and also can reduce risk effectively. This paper uses the Global-best Guided Quantum-inspired Tabu Search with Quantum-NOT Gate (GNQTS) to find how many funds are allocated to each stock and then uses the trend ratio to evaluate the portfolio. Because overfitting is a common problem in the stock market, this paper uses thirteen types of sliding windows to avoid overfitting. The results show that fund allocation is more flexible than allocating equal funds, thus allowing the proposed method to find a portfolio with a higher return and lower risk.

Keywords: Optimization, Evolutionary Computation, Heuristic Algorithms
Abstract: Future force design is a crucial task that assists in the creation of an effective future defence force. The primary objective of this task is to select a set of projects, within a fixed planning window and subject to budgetary constraints, that will lead to improved capabilities. While inherently related to the well-known multi-period knapsack problem, addressing this problem in the context of the defence sector gives rise to a number of unique nuances and associated challenges. Furthermore, the literature pertaining to the selection and scheduling of projects for capability-based planning in the defence sector is rather limited. To address this literature gap, this paper formalizes a multi-period project selection and scheduling problem inspired by future force design. Numerous heuristics, both random and deterministic, along with a hybrid genetic algorithm, are employed to optimize a set of instances of the proposed problem formulation with various characteristics derived from real-world, public defence data made available by the Australian Department of Defence.

Keywords: Optimization, Evolutionary Computation, Heuristic Algorithms
Abstract: Many crucial pieces of infrastructure we use on a daily basis, such as power grids and telecommunication systems, can be described as networks. The robustness of a network to failure is an important part of the reliability of its functionality. To help ensure that these networks stay stable and operational, it is useful to know how to increase their robustness to failure. Previous work has focused on increasing network robustness either through rewiring the network's existing edges or augmenting it with additional new edges. This paper proposes a new hybrid method that combines these two approaches and shows that this hybrid technique can be used to create networks considerably more robust than techniques that rely only on edge rewiring.

Keywords: Optimization, Machine Learning, Agent-Based Modeling
Abstract: Modern compilers provide a huge number of optional compilation optimization options. Not only the selection of compilation optimization options represents a hard problem to be solved, but also the ordering of the phases is adding further complexity, making it a long standing problem in compilation research. A large number of experiments have shown that different ordering of the phases has varying degrees of influence on the program. Currently, most research focuses on the traditional optimization goals, such as execution speedup and code size optimization. In this paper, we focus on the impact of the phase-ordering on program reliability. We propose a new model with reinforcement learning algorithm A3C for finding the phase order that can improve the reliability of the program. We performed our experiments with LLVM compiler framework, considering 130 LLVM optimization options. The experimental results show that when compared with LLVM standard options and the existing phase-ordering method with genetic algorithm, the phase order found by our model can bring higher reliability gain to the program.

Keywords: Optimization, Machine Learning, Industry 4.0
Abstract: Robots are nowadays increasingly required to deal with (partially) unknown tasks and situations. The robot has, therefore, to adapt its behavior to the specific working conditions. Classical control methods in robotics require manually programming all actions of a robot. While very effective in fixed conditions, such model-based approaches cannot handle variations, demanding tedious tuning of parameters for every new task. Reinforcement learning (RL) holds the promise of autonomously learning new control policies through trial-and-error. However, RL approaches are prone to learning with high samples, particularly for continuous control problems. In this paper, a learning-based method is presented that leverages simulation data to learn an object manipulation task through RL. The control policy is parameterized by a neural network and learned using modern Proximal Policy Optimization (PPO) algorithm. A dense reward function has been designed for the task to enable efficient learning of an agent. The proposed approach is trained entirely in simulation (exploiting the MuJoCo environment) from scratch without any demonstrations of the task. A grasping task involving a Franka Emika Panda manipulator has been considered as the reference task to be learned. The task requires the robot to reach the part, grasp it, and lift it off the contact surface. The proposed approach has been demonstrated to be generalizable across multiple object geometries and initial robot/parts configurations, having the robot able to learn and re-execute the target task.

Keywords: Machine Learning
Abstract: Positive vectors appear naturally in many applications, yet heir modeling has not been adequately addressed in the past. Inverted Dirichlet offers a good representation and modeling of positive non-Gaussian vectors, and its generalizations offer even more practical and flexible alternatives. In this paper, we propose three regression models based on flexible distributions for semi-bounded data, namely, inverted Dirichlet, inverted generalize Dirichlet, and inverted Beta-Liouville. The efficiency of these models is tested via real-world applications, including software defects prediction, age estimation, spam filtering, and disease diagnosis. Our results show that the three proposed regression models outperform other commonly used regression approaches

Keywords: Machine Learning, Computational Intelligence, Image Processing/Pattern Recognition
Abstract: Hidden Markov models (HMM) have recently risen as a key generative machine learning approach for time series data study and analysis. While early works focused only on applying HMMs for speech recognition, HMMs are now prominent in various fields such as stock market forecasting, video classification, and genomics. In this paper, we develop a Maximum A Posteriori (MAP) framework for learning the Dirichlet and Beta-Liouville HMMs that have been proposed recently as an efficient way for modeling sequential proportional data. In contrast to the conventional Baum Welch algorithm, commonly used for learning HMMs, the proposed algorithm places priors for the learning of the desired parameters; hence, regularizing the estimation process. We validate our proposed approach on two challenging real applications; namely, dynamic texture classification and infrared action recognition.

Keywords: Machine Learning, Optimization, Machine Vision
Abstract: As a natural extension of the traditional graph model, hypergraph has been extensively exploited and applied in many tasks such as image clustering, classification, etc. The performance of these tasks highly depends on building an informative hypergraph to accurately and robustly formulate the underlying data correlation. Existing hypergraph construction methods can only be suitable for simple Gaussian or outlier noise assumptions, which cannot be applied to more complex noise scenario in practical applications. To address this challenge, we propose a robust hypergraph learning model by adopting the Mixture of Gaussians (MoG) noise modeling strategy. In particular, our model adopts low-rank representation and sparse representation simultaneously to construct an informative hypergraph. The Correlation among nodes and the weights of edges can be obtained by seeking a low-rank and sparse representation matrix. The so-obtained hypergraph can capture both the global mixture of subspaces structure (by low-rank) and the locally linear structure (by sparse) of the data. Furthermore, an efficient Expectation-Maximization-like optimization algorithm is designed to solve the proposed model. Finally, the superiority of our model is demonstrated by extensive experiments on image clustering.

Keywords: Neural Networks and their Applications, Machine Learning, Image Processing/Pattern Recognition
Abstract: In recent years, there has been substantial research on uncertainty and explainability to improve the reliability of deep learning predictions. In the field of image processing, previous methods can be intuitively understood by humans and can be used to evaluate the reliability of predictions. However, they cannot be applied to complex time series prediction because of the difficulty for humans in evaluating the reliability of predictions. Previous methods focused on either uncertainty or explainability; therefore, the evaluation of the methods and predictions depended only on human knowledge. In this paper, we propose a time series prediction method focusing on both uncertainty and explainability. Our method allows the evaluation of predictions from multiple perspectives by using uncertainty and explainability without relying solely on human knowledge. The explanation obtained from the unimodal distribution prediction model is unreliable because the predictive distribution of time series prediction can be an inherently multi-peaked. The unimodal model does not correspond to the actual phenomenon and does not match with human knowledge. Therefore, we solve the time series prediction as a multi-class classification problem. From the verification of the proposed model with real time series, we confirm the effectiveness of the (1) prediction based on the expected value of the class probability distribution, (2) confidence of the prediction, and (3) series importance based on the confidence. Moreover, we show that our model can improve the robustness of the time series prediction and evaluate its reliability.

Keywords: Machine Vision, Machine Learning, Image Processing/Pattern Recognition
Abstract: The task of crowd counting is to estimate the accurate number of people in photos taken from unconstrained surveillance scenes. It is in general a challenging problem due to the input scale variations and perspective distortions. Previous methods make efforts to enhance the representation ability by using multi-scale features of the scene pictures. However, most of these methods directly add or fuse the features, in which the influences of different feature sizes are equally considered. In this paper, we propose a novel architecture called adaptive context learning network (ACLNet) to incorporate context of features in multiple levels. In this architecture, the original image features are enhanced by a multi-level feature generating module, and then the multi-level features are up-sampled to the same size and re-weighted for fusing. The ACLNet incorporates the context information existed in sub-regions of various scales adaptively, thus it is able to enhance the representative ability of multi-level features. We perform several experiments on public ShanghaiTech (A and B), UCF CC 50 and NWPU-crowd datasets. Our proposed ACLNet achieves the state-of-the-art results compared with existing methods.