Primate vision depends on recurrent processing for reliable perception [1-3]. A growing body of literature also suggests that recurrent connections improve the learning efficiency and generalization of vision models o...
详细信息
ISBN:
(纸本)9781713829546
Primate vision depends on recurrent processing for reliable perception [1-3]. A growing body of literature also suggests that recurrent connections improve the learning efficiency and generalization of vision models on classic computer vision challenges. Why then, are current large-scale challenges dominated by feedforward networks? We posit that the effectiveness of recurrent vision models is bottlenecked by the standard algorithm used for training them, "back-propagation through time" (BPTT), which has O(N) memory-complexity for training an N step model. Thus, recurrent vision model design is bounded by memory constraints, forcing a choice between rivaling the enormous capacity of leading feedforward models or trying to compensate for this deficit through granular and complex dynamics. Here, we develop a new learning algorithm, "contractor recurrent back-propagation" (C-RBP), which alleviates these issues by achieving constant O(1) memory-complexity with steps of recurrent processing. We demonstrate that recurrent vision models trained with C-RBP can detect long-range spatial dependencies in a synthetic contour tracing task that BPTT-trained models cannot. We further show that recurrent vision models trained with C-RBP to solve the large-scale Panoptic Segmentation MS-COCO challenge outperform the leading feedforward approach, with fewer free parameters. C-RBP is a general-purpose learning algorithm for any application that can benefit from expansive recurrent dynamics. Code and data are available at https://***/c-rbp.
For the analysis, simulation, and controller design of large-scale systems, a surrogate model with small complexity is mostly required. A standard approach to determine such a model is given by modelling the system an...
详细信息
For the analysis, simulation, and controller design of large-scale systems, a surrogate model with small complexity is mostly required. A standard approach to determine such a model is given by modelling the system and applying model-order-reduction techniques. Contrary, we propose a data-driven approach, where the surrogate model of the input-output behaviour of an LTI system is determined from data without modelling the system beforehand. Moreover, we provide a guaranteed bound on the maximal error between the system and the surrogate model in case of noise-free measurements. We analyse the stability and convergence of the presented schemes and apply them on a benchmark system from the model-order-reduction literature. Copyright (C) 2020 The Authors.
In federated learning, the local devices train the model with their local data, independently;and the server gathers the locally trained model to aggregate them into a shared global model. Therefore, federated learnin...
详细信息
ISBN:
(纸本)9781450381826
In federated learning, the local devices train the model with their local data, independently;and the server gathers the locally trained model to aggregate them into a shared global model. Therefore, federated learning is an approach to decouple the model training from directly assessing the local data. However, the requirement of periodic communications on model parameters results in a primary bottleneck for the efficiency of federated learning. This work proposes a novel federated learning algorithm, Federated Weight Recovery(FEWER), which enables a sparsely pruned model in the training phase. FEWER starts with the initial model training with an extremely sparse state, and FEWER gradually grows the model capacity until the model reaches a dense model at the end of the training. The level of sparsity becomes the leverage to either increasing the accuracy or decreasing the communication cost, and this sparsification can be beneficial to practitioners. Our experimental results show that FEWER achieves higher test accuracies with less communication costs for most of the test cases.
Within the simulation community, the prevailing wisdom seems to be that when solving a simulation optimization problem, biased gradient estimators should not be used to guide a local-search algorithm. On the contrary,...
详细信息
ISBN:
(纸本)9781728194998
Within the simulation community, the prevailing wisdom seems to be that when solving a simulation optimization problem, biased gradient estimators should not be used to guide a local-search algorithm. On the contrary, we argue that for certain problems, biased gradient estimators may still provide useful directional information. We focus on the infinitesimal perturbation analysis (IPA) gradient estimator, which is biased when an interchange of differentiation and expectation fails. Although a local-search algorithm guided by biased gradient estimators will likely not converge to a local optimal solution, it might be expected to reach a neighborhood of one. We test such a gradient-based search on an ambulance base location problem, demonstrating its effectiveness in a non-trivial example, and present some supporting theoretical results.
The node replication attack is considered one of the most dangerous attacks against wireless sensor networks (WSNs). In this attack, an adversary captures one or more normal nodes of the network, extracts its key mate...
详细信息
In this paper, we develop the Koopman operator theory for dynamical systems with symmetry. In particular, we investigate how the Koopman operator and eigenfunctions behave under the action of the symmetry group of the...
详细信息
In this paper, we develop the Koopman operator theory for dynamical systems with symmetry. In particular, we investigate how the Koopman operator and eigenfunctions behave under the action of the symmetry group of the underlying dynamical system. Further, exploring the underlying symmetry, we propose an algorithm to construct a global Koopman operator from local Koopman operators. In particular, we show, by exploiting the symmetry, data from all the invariant sets are not required for constructing the global Koopman operator;that is, local knowledge of the system is enough to infer the global dynamics. Copyright (C) 2020 The Authors.
Atmospheric flows are governed by a broad variety of spatio-temporal scales, thus making real-time numerical modeling of such turbulent flows in complex terrain at high resolution computationally unmanageable. In this...
详细信息
Atmospheric flows are governed by a broad variety of spatio-temporal scales, thus making real-time numerical modeling of such turbulent flows in complex terrain at high resolution computationally unmanageable. In this paper, we demonstrate a novel approach to address this issue through a combination of fast coarse scale physics based simulator and a family of advanced machine learning algorithm called the Generative Adversarial Networks. The physics-based simulator generates a coarse wind field in a real wind farm and then ESRGANs enhance the result to a much finer resolution. The method outperforms state of the art bicubic interpolation methods commonly utilized for this purpose.
The support of coexisting Ultra-Reliable and Low-Latency Communication (URLLC) and enhanced Mobile BroadBand (eMBB) services is a cornerstone challenge to wireless communication networks. Those two types of services i...
详细信息
ISBN:
(纸本)9780738131269
The support of coexisting Ultra-Reliable and Low-Latency Communication (URLLC) and enhanced Mobile BroadBand (eMBB) services is a cornerstone challenge to wireless communication networks. Those two types of services introduce strict resource allocation requirements resulting in a power-struggle between reliability, latency, and resource utilization. The difficulty in addressing that challenge could be rooted in the predominant reactive approach to resource allocation in wireless networks, where the allocation operation is carried out based on received service requests and global network statistics. This paper proposes a novel framework termed service identification to develop proactive resource allocation algorithms. The framework is based on visual data and deep learning, and its objective is to equip future wireless networks with the ability to anticipate incoming services and perform proactive resource allocation. To demonstrate the potential of the framework, a wireless network scenario with two coexisting URLLC and eMBB services is considered, and a deep learning algorithm is designed to utilize RGB video frames and predict incoming service type and its request time. An evaluation dataset is developed and used to evaluate the performance of the algorithm. The results show that the algorithm enables a 78% utilization of idle network resources, which emphasizes the value of proaction.
Anomalies are pervasive in time series data, such as sensor readings. Existing methods for anomaly detection cannot distinguish between anomalies that represent data errors, such as incorrect sensor readings, and nota...
详细信息
ISBN:
(纸本)9781728129037
Anomalies are pervasive in time series data, such as sensor readings. Existing methods for anomaly detection cannot distinguish between anomalies that represent data errors, such as incorrect sensor readings, and notable events, such as the watering action in soil monitoring. In addition, the quality performance of such detection methods highly depends on the configuration parameters, which are dataset specific. In this work, we exploit active learning to detect both errors and events in a single solution that aims at minimizing user interaction. For this joint detection, we introduce an algorithm that accurately detects and labels anomalies with a non-parametric concept of neighborhood and probabilistic classification. Given a desired quality, the confidence of the classification is then used as termination condition for the active learning algorithm. Experiments on real and synthetic datasets demonstrate that our approach achieves F-score above 80% in detecting errors by labeling 2 to 5 points in one data series. We also show the superiority of our solution compared to the state-of-the-art approaches for anomaly detection. Finally, we demonstrate the positive impact of our error detection methods in downstream data repairing algorithms.
In this paper, we study adaptive online convex optimization, and aim to design a universal algorithm that achieves optimal regret bounds for multiple common types of loss functions. Existing universal methods are limi...
详细信息
In this paper, we study adaptive online convex optimization, and aim to design a universal algorithm that achieves optimal regret bounds for multiple common types of loss functions. Existing universal methods are limited in the sense that they are optimal for only a subclass of loss functions. To address this limitation, we propose a novel online algorithm, namely Maler, which enjoys the optimal O(root T), O(d log T) and O(log T) regret bounds for general convex, exponentially concave, and strongly convex functions respectively. The essential idea is to run multiple types of learning algorithms with different learning rates in parallel, and utilize a meta-algorithm to track the best on the fly. Empirical results demonstrate the effectiveness of our method.
暂无评论