检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Burns, Thomas F. Fukai, Tomoki Earls, Christopher J. SciAI Center Cornell University United States Neural Coding and Brain Computing Unit OIST Japan SciAI Center Center for Applied Mathematics School of Civil and Environmental Engineering Cornell University United States

Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability is known as in-context learning (ICL). Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification. We then apply our architecture in small language models with 8 million parameters, focusing on attention head values, with results also indicating improved ICL performance at this larger and more naturalistic *** Codes 92B20, 68T01, 68T37, 68T50 © 2024, CC BY.

关键词： Associative storage

来源：评论

学校读者我要写书评

暂无评论

Detecting danger in gridworlds using Gromov's Link Condition

arXiv

引用

arXiv 2022年

作者： Burns, Thomas F. Tang, Robert Neural Coding and Brain Computing Unit OIST Graduate University 1919-1 Tancha Onna-son Kunigami-gun Okinawa 904-0495 Japan Department of Pure Mathematics Xi'an Jiaotong Liverpool University 111 Ren'ai Road Suzhou Industrial Park Suzhou Jiangsu Province 215123 China

Gridworlds have been long-utilised in AI research, particularly in reinforcement learning, as they provide simple yet scalable models for many real-world applications such as robot navigation, emergent behaviour, and operations research. We initiate a study of gridworlds using the mathematical framework of reconfigurable systems and state complexes due to Abrams, Ghrist & Peterson. State complexes represent all possible configurations of a system as a single geometric space, thus making them conducive to study using geometric, topological, or combinatorial methods. The main contribution of this work is a modification to the original Abrams, Ghrist & Peterson setup which we introduce to capture agent braiding and thereby more naturally represent the topology of gridworlds. With this modification, the state complexes may exhibit geometric defects (failure of Gromov's Link Condition). Serendipitously, we discover these failures occur exactly where undesirable or dangerous states appear in the gridworld. Our results therefore provide a novel method for seeking guaranteed safety limitations in discrete task environments with single or multiple agents, and offer useful safety information (in geometric and topological forms) for incorporation in or analysis of machine learning systems. More broadly, our work introduces tools from geometric group theory and combinatorics to the AI community and demonstrates a proof-of-concept for this geometric viewpoint of the task domain through the example of simple gridworld *** Codes 57Z25 (Primary) 68R01, 51F99 (Secondary) © 2022, CC BY.

关键词： Geometry

来源：评论

学校读者我要写书评

暂无评论

26th Annual Computational Neuroscience Meeting (CNS*2017) of the Organization for Computational Neuroscience Antwerp, Belgium, July 15-20, 2017

引用

BMC NEUROSCIENCE 2017年第SUPPL 1期18卷 59-59页

作者： [Anonymous] Indiana University Purdue University Indianapolis Indianapolis IN 46032 USA Stark Neurosciences Research Institute Indiana University School of Medicine Indianapolis IN 46032 USA Department of Mathematics East Carolina University Greenville NC 27858 USA Jülich Supercomputing Centre Forschungszentrum Jülich 52425 Jülich Germany Future Systems Swiss National Supercomputing Centre 8092 Zurich Switzerland User Engagement and Support Swiss National Supercomputing Centre 6900 Lugano Switzerland Institut de Neurosciences des Systèmes Aix Marseille Univ 13005 Marseille France Simulation Lab Neuroscience Forschungszentrum Jülich Jülich Germany Department of Experimental Psychology Ghent University 9000 Ghent Belgium Donders Center for Cognitive Neuroimaging Radboud University 6525HR Nijmegen The Netherlands Department of Electrical Computer and Energy Engineering University of Colorado Boulder CO 80309 USA Department of Neurosurgery Johns Hopkins School of Medicine Baltimore MD 21287 USA Department of Neurology Johns Hopkins School of Medicine Baltimore MD 21287 USA Department of Otolaryngology Johns Hopkins School of Medicine Baltimore MD 21287 USA INSERM U968 Paris France Sorbonne Universités UPMC University Paris 06 UMR_S 968 Institut de la Vision Paris France CNRS UMR_7210 Paris France Department of Computer Architecture and Technology University of Granada (CITIC) Granada Spain Sorbonne Universités UPMC Univ Paris 06 INSERM CNRS Institut de la Vision Paris France Department of Adaptive Machine Systems Osaka University Osaka Japan Department of Computer Science University of Cergy-Pontoise Cergy-Pontoise France Department of Physics and Astronomy College of Charleston Charleston SC 29424 USA School of Physics Faculty of Science University of Sydney Sydney NSW 2006 Australia Center of Excellence for Integrative Brain Function Australian Research Council Sydney Australia Max Planck Institute for Human Cognitive and Brain Sciences Saxony Lei

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：