检索结果-内蒙古大学图书馆

Variable Binding for sparse distributed representations: Theory and Applications

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2023年第5期34卷 2191-2204页

作者： Frady, Edward Paxon Kleyko, Denis Sommer, Friedrich T. Intel Labs Neuromorph Comp Lab Santa Clara CA 95054 USA Univ Calif Berkeley Redwood Ctr Theoret Neurosci Berkeley CA 94720 USA Res Inst Sweden Intelligent Syst Lab S-16440 Kista Sweden

Variable binding is a cornerstone of symbolic reasoning and cognition. But how binding can be implemented in connectionist models has puzzled neuroscientists, cognitive psychologists, and neural network researchers for many decades. One type of connectionist model that naturally includes a binding operation is vector symbolic architectures (VSAs). In contrast to other proposals for variable binding, the binding operation in VSAs is dimensionality-preserving, which enables representing complex hierarchical data structures, such as trees, while avoiding a combinatoric expansion of dimensionality. Classical VSAs encode symbols by dense randomized vectors, in which information is distributed throughout the entire neuron population. By contrast, in the brain, features are encoded more locally, by the activity of single neurons or small groups of neurons, often forming sparse vectors of neural activation. Following Laiho et al. (2015), we explore symbolic reasoning with a special case of sparse distributed representations. Using techniques from compressed sensing, we first show that variable binding in classical VSAs is mathematically equivalent to tensor product binding between sparse feature vectors, another well-known binding operation which increases dimensionality. This theoretical result motivates us to study two dimensionality-preserving binding methods that include a reduction of the tensor matrix into a single sparse vector. One binding method for general sparse vectors uses random projections, the other, block-local circular convolution, is defined for sparse vectors with block structure, sparse block-codes. Our experiments reveal that block-local circular convolution binding has ideal properties, whereas random projection based binding also works, but is lossy. We demonstrate in example applications that a VSA with block-local circular convolution and sparse block-codes reaches similar performance as classical VSAs. Finally, we discuss our results in the contex

关键词： Data structures Neurons Cognition Tensors Convolution Compounds sparse matrices Classification cognitive reasoning compressed sensing (CS) sparse block-codes sparse distributed representations tensor product variable binding vector symbolic architectures (VSAs)

来源：评论

学校读者我要写书评

暂无评论

"What-Where" sparse distributed invariant representations of visual patterns

引用

NEURAL COMPUTING & APPLICATIONS 2022年第8期34卷 6207-6214页

作者： Sa-Couto, Luis Wichert, Andreas Univ Lisbon Inst Super Tecn INESC ID Lisbon Portugal

Although modern deep learning approaches have achieved astounding results in most visual pattern recognition tasks, they do it using large datasets of labeled data. Besides the fact that, in many applications, such labels are costly to obtain, the need for them is not observed in a biologically intelligent machine like the human brain. "What-Where" sets were proposed as a way to represent visual patterns in a manipulatable manner, where two-dimensional geometric transformations can be exploited to increase invariance, and thus reduce the need for large amounts of training data. However, the cornerstone of classification using these sets is a similarity measure that implicates a time-consuming computation due to the unstructured nature of sets. In this work, we propose a grid-based coding strategy to represent the sets as sparse binary vectors. By doing so, we achieve three main advantages: first, leveraging pointer-coding of active bits, we reduce the time complexity of the similarity computation from quadratic to linear in the number of elements of the smaller set being compared;second, we use the theoretical framework of sparse representations to justify the classification robustness exhibited in the original work;third, we bring the model under the widely accepted biological constraint that populations of neurons in the brain code sparse representations.

关键词： Brain inspired architectures sparse distributed representations "What-Where" sets Steep learning curves

来源：评论

学校读者我要写书评

暂无评论

Binding of sparse distributed representations in Hierarchical Temporal Memory 19

Binding of Sparse Distributed Representations in Hierarchica...

引用

7th Annual Neuro-Inspired Computational Elements Workshop (NICE)

作者： Boudreau, Luke Kudithipudi, Dhireesha Rochester Inst Technol Neuromorph AI Lab Rochester NY 14623 USA

ISBN: (纸本)9781450361231

Hierarchical Temporal Memory is a brain inspired theory of intelligence, which emulates the homogeneous structure and connectivity of the pyramidal neurons in the mammalian neocortex. Similar to the neocortex, Hierarchical Temporal Memory processes spatiotemporal information to produce an internal representation of objects in the world. A fundamental component in Hierarchical Temporal Memory is the spatial pooler, which is responsible for processing feedforward sensory inputs into sparse distributed representations. In this work, we apply an explicit binary binding vector operation from vector symbolic architectures to HTM's sparse distributed representations through local inhibition. We demonstrate that the binding operation can be used as the basis for a content addressable memory, which has the potential for storage and retrieval of sparse distributed representations. Furthermore, the bound vectors and their semantic relationships are visualized through dimensionality reduction techniques and a uniqueness matrix.

关键词： Online-learning HTM sparse distributed representations Binding FREAK

来源：评论

学校读者我要写书评

暂无评论

Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?

引用

BIOLOGICAL CYBERNETICS 2024年第5-6期118卷 267-276页

作者： Osorio, Maria Sa-Couto, Luis Wichert, Andreas Univ Lisbon Dept Comp Sci & Engn INESC ID Ave Prof Dr Anibal Cavaco Silva P-2744016 Lisbon Portugal Univ Lisbon Inst Super Tecn Ave Prof Dr Anibal Cavaco Silva P-2744016 Lisbon Portugal

It is generally assumed that the brain uses something akin to sparse distributed representations. These representations, however, are high-dimensional and consequently they affect classification performance of traditional Machine Learning models due to the "curse of dimensionality". In tasks for which there is a vast amount of labeled data, Deep Networks seem to solve this issue with many layers and a non-Hebbian backpropagation algorithm. The brain, however, seems to be able to solve the problem with few layers. In this work, we hypothesize that this happens by using Hebbian learning. Actually, the Hebbian-like learning rule of Restricted Boltzmann Machines learns the input patterns asymmetrically. It exclusively learns the correlation between non-zero values and ignores the zeros, which represent the vast majority of the input dimensionality. By ignoring the zeros the "curse of dimensionality" problem can be avoided. To test our hypothesis, we generated several sparse datasets and compared the performance of a Restricted Boltzmann Machine classifier with some Backprop-trained networks. The experiments using these codes confirm our initial intuition as the Restricted Boltzmann Machine shows a good generalization performance, while the Neural Networks trained with the backpropagation algorithm overfit the training data.

关键词： Hebbian learning Restricted Boltzmann machines sparse distributed representations Curse of dimensionality

来源：评论

学校读者我要写书评

暂无评论

Cyberattack Detection in the Industrial Internet of Things Based on the Computation Model of Hierarchical Temporal Memory

引用

AUTOMATIC CONTROL AND COMPUTER SCIENCES 2023年第8期57卷 1040-1046页

作者： Krundyshev, V. M. Markov, G. A. Kalinin, M. O. Semyanov, P. V. Busygin, A. G. Peter Great St Petersburg Polytech Univ St Petersburg 195251 Russia Jet Infosystems Moscow 127015 Russia

This study considers the problem of detecting network anomalies caused by computer attacks in the networks of the industrial Internet of things. To detect anomalies, a new method is proposed, built using a hierarchical temporal memory (HTM) computation model based on the neocortex model. An experimental study of the developed method of detecting computer attacks based on the HTM model showed the superiority of the developed solution over the LSTM analog. The developed prototype of the anomaly detection system provides continuous training on unlabeled data sets in real time, takes into account the current network context, and applies the accumulated experience by supporting the memory mechanism.

关键词： hierarchical temporal memory artificial intelligence cyberattacks neocortex real-time training sparse distributed representations HTM

来源：评论

学校读者我要写书评

暂无评论

A Radically New Theory of How the Brain Represents and Computes with Probabilities 9th

A Radically New Theory of How the Brain Represents and Compu...

引用

9th Annual Conference on Machine Learning, Optimization and Data science (LOD)

作者： Rinkus, Gerard Neurithm Syst Newton MA 02465 USA

ISBN: (纸本)9783031539657;9783031539664

It is widely believed that the brain implements probabilistic reasoning and that it represents information via some form of population (distributed) code. Most prior probabilistic population coding (PPC) theories share basic properties: 1) continuous-valued units;2) fully/densely distributed codes;3) graded synapses;4) rate coding;5) units have innate low-complexity, usually unimodal, tuning functions (TFs);and 6) units are intrinsically noisy and noise is generally considered harmful. I describe a radically different theory that assumes: 1) binary units;2) sparse distributed codes (SDC);3) functionally binary synapses;4) a novel, atemporal, combinatorial spike code;5) units initially have flat TFs (all weights zero);and 6) noise is a controlled resource used to cause similar inputs to be mapped to similar codes. The theory, sparsey, was introduced 25 + years ago as: a) an explanation of the physical/computational relationship of episodic and semantic memory for the spatiotemporal (sequential) pattern domain;and b) a canonical, mesoscale cortical probabilistic circuit/algorithm possessing fixed-time, unsupervised, single-trial, non-optimization-based, unsupervised learning and fixed-time best-match (approximate) retrieval;but was not described in terms of probabilistic computation. Here, we show that: a) the active SDC in a sparsey coding field (CF) simultaneously represents not only the likelihood of the single most likely input but the likelihoods of all hypotheses stored in the CF;and b) that entire explicit distribution can be transmitted, e.g., to a downstream CF, via a set of simultaneous single spikes from the neurons comprising the active SDC.

关键词： sparse distributed representations probabilistic population coding cell assemblies canonical cortical circuit/algorithm

来源：评论

学校读者我要写书评

暂无评论

Hierarchical intrinsically motivated agent planning behavior with dreaming in grid environments

引用

BRAIN INFORMATICS 2022年第1期9卷 8页

作者： Dzhivelikian, Evgenii Latyshev, Artem Kuderov, Petr Panov, Aleksandr I. Moscow Inst Phys & Technol MIPT Dolgoprudnyi Russia Russian Acad Sci Fed Res Ctr Comp Sci & Control Moscow Russia Artificial Intelligence Res Inst AIRI Moscow Russia

Biologically plausible models of learning may provide a crucial insight for building autonomous intelligent agents capable of performing a wide range of tasks. In this work, we propose a hierarchical model of an agent operating in an unfamiliar environment driven by a reinforcement signal. We use temporal memory to learn sparse distributed representation of state-actions and the basal ganglia model to learn effective action policy on different levels of abstraction. The learned model of the environment is utilized to generate an intrinsic motivation signal, which drives the agent in the absence of the extrinsic signal, and through acting in imagination, which we call dreaming. We demonstrate that the proposed architecture enables an agent to effectively reach goals in grid environments.

关键词： Model-based reinforcement learning Intrinsic motivation Hierarchical temporal memory sparse distributed representations

来源：评论

学校读者我要写书评

暂无评论

Hebbian spatial encoder with adaptive sparse connectivity

引用

COGNITIVE SYSTEMS RESEARCH 2024年 88卷

作者： Kuderov, Petr Dzhivelikian, Evgenii Panov, Aleksandr I. Moscow Inst Phys & Technol MIPT Dolgoprudnyi Russia Artificial Intelligence Res Inst AIRI Moscow Russia Russian Acad Sci Fed Res Ctr Comp Sci & Control Moscow Russia

Biologically plausible neural networks have demonstrated efficiency in learning and recognizing patterns in data. This paper proposes a general online unsupervised algorithm for spatial data encoding using fast Hebbian learning. Inspired by the Hierarchical Temporal Memory (HTM) framework, we introduce the SpatialEncoder algorithm, which learns the spatial specialization of neurons' receptive fields through Hebbian plasticity and k-WTA (k k winners take all) inhibition. A key component of our model is a two-part synaptogenesis algorithm that enables the network to maintain a sparse connection matrix while adapting to non-stationary input data distributions. In the MNIST digit classification task, our model outperforms the HTM SpatialPooler in terms of classification accuracy and encoding stability. Compared to another baseline, a two-layer artificial neural network (ANN), our model achieves competitive classification accuracy with fewer iterations required for convergence. The proposed model offers a promising direction for future research on sparse neural networks with adaptive neural connectivity.

关键词： Spatial encoding sparse distributed representations sparse neural networks Adaptive neural connectivity

来源：评论

学校读者我要写书评

暂无评论

Intrinsic Motivation to Learn Action-State Representation with Hierarchical Temporal Memory 14th

Intrinsic Motivation to Learn Action-State Representation wi...

引用

14th International Conference on Brain Informatics (BI)

作者： Dzhivelikian, Evgenii Latyshev, Artem Kuderov, Petr Panov, Aleksandr, I Moscow Inst Phys & Technol Dolgoprudnyi Russia Artificial Intelligence Res Inst Moscow Russia Russian Acad Sci Fed Res Ctr Comp Sci & Control Moscow Russia

ISBN: (纸本)9783030869939;9783030869922

In this paper, we propose a biologically plausible model for learning the decision-making sequence in an external environment with internal motivation. As a computational model, we propose a hierarchical architecture of an intelligent agent acquiring experience based on reinforcement learning. We use the basal ganglia model to aggregate a reward, and sparse distributed representation of states and actions in hierarchical temporal memory elements. The proposed architecture allows the agent to build a compact model of the environment and to form an effective strategy, which is experimentally demonstrated to search for resources in grid environments.

关键词： Model-based reinforcement learning Intrinsic motivation Hierarchical temporal memory sparse distributed representations

来源：评论

学校读者我要写书评

暂无评论

Using the hierarchical temporal memory spatial pooler for short-term forecasting of electrical load time series

引用

APPLIED COMPUTING AND INFORMATICS 2021年第2期17卷 264-278页

作者： Osegi, E. N. Natl Open Univ Nigeria Dept Informat Technol Lagos Nigeria

In this paper, an emerging state-of-the-art machine intelligence technique called the Hierarchical Temporal Memory (HTM) is applied to the task of short-term load forecasting (STLF). A HTM Spatial Pooler (HTM-SP) stage is used to continually form sparse distributed representations (SDRs) from a univariate load time series data, a temporal aggregator is used to transform the SDRs into a sequential bivariate representation space and an overlap classifier makes temporal classifications from the bivariate SDRs through time. The comparative performance of HTM on several daily electrical load time series data including the Eunite competition dataset and the Polish power system dataset from 2002 to 2004 are presented. The robustness performance of HTM is also further validated using hourly load data from three more recent electricity markets. The results obtained from experimenting with the Eunite and Polish dataset indicated that HTM will perform better than the existing techniques reported in the literature. In general, the robustness test also shows that the error distribution performance of the proposed HTM technique is positively skewed for most of the years considered and with kurtosis values mostly lower than a base value of 3 indicating a reasonable level of outlier rejections.

关键词： Machine intelligence Short-term load forecasting sparse distributed representations Time series Overlapping temporal classification

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：