检索结果-内蒙古大学图书馆

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

JOURNAL OF MACHINE LEARNING RESEARCH 2022年第1期23卷 1-39页

作者： Fedus, William Zoph, Barret Shazeer, Noam Google Mountain View CA 94043 USA

In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) models defy this and instead select different parameters for each in-coming example. The result is a sparsely-activated model-with an outrageous number of parameters-but a constant computational cost. However, despite several notable suc-cesses of MoE, widespread adoption has been hindered by complexity, communication costs, and training instability. We address these with the introduction of the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques mitigate the instabilities, and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large (Raffel et al., 2019) to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the "Colossal Clean Crawled Corpus", and achieve a 4x speedup over the T5-XXL model.1

关键词： mixture-of-experts natural language processing sparsity large-scale machine learning distributed computing

来源：评论

学校读者我要写书评

暂无评论

Scalable high-dimensional multivariate linear regression for feature-distributed data

引用

The Journal of Machine Learning Research 2024年第1期25卷 9858-9916页

作者： Shuo-Chieh Huang Ruey S. Tsay Booth School of Business University of Chicago Chicago IL

Feature-distributed data, referred to data partitioned by features and stored across multiple computing nodes, are increasingly common in applications with a large number of features. This paper proposes a two-stage relaxed greedy algorithm (TSRGA) for applying multivariate linear regression to such data. The main advantage of TSRGA is that its communication complexity does not depend on the feature dimension, making it highly scalable to very large data sets. In addition, for multivariate response variables, TSRGA can be used to yield low-rank coefficient estimates. The fast convergence of TSRGA is validated by simulation experiments. Finally, we apply the proposed TSRGA in a financial application that leverages unstructured data from the 10-K reports, demonstrating its usefulness in applications with many dense large-dimensional matrices.

关键词： Frank-Wolfe algorithm distributed computing reduced-rank regression feature selection multi-view and multi-modal data

来源：评论

学校读者我要写书评

暂无评论

An Efficient Scheme for Data Pattern Matching in IoT Networks

引用

Computers, Materials & Continua 2022年第8期72卷 2203-2219页

作者： Ashraf Ali Omar A.Saraereh Department of Electrical Engineering Engineering FacultyThe Hashemite UniversityZarqa13133Jordan

The Internet has become an unavoidable trend of all things due to the rapid growth of networking technology,smart home technology encompasses a variety of sectors,including intelligent transportation,allowing users to communicate with anybody or any device at any time and from ***,most things are different ***:Structured data is a form of separated storage that slows down the rate at which everything is *** pattern matching is commonly used in data connectivity and can help with the issues mentioned ***:The present pattern matching system is ineffective due to the heterogeneity and rapid expansion of large IoT *** method requires a lot of manual work and has a poor match with real-world *** the modern IoT context,solving the challenge of automatic pattern matching is ***:A three-layer mapping matching is proposed for heterogeneous data from the IoT,and a hierarchical pattern matching *** feature classification matching,relational feature clustering matching,and mixed element matching are all examples of feature classification *** layer-by-layer matching,the algorithm gradually narrows the matching space,improving matching quality,reducing the number of matching between components and the degree of manual participation,and producing a better automatic mode ***:The algorithm’s efficiency and performance are tested using a large number of data samples,and the results show that the technique is practical and ***:the proposed algorithm utilizes the instance information of the data *** deploys three-layer mapping matching approach and mixed element matching and realizes the automatic pattern matching of heterogeneous data which reduces the matching space between elements in complex *** improves the efficiency and accuracy of automatic matching.

关键词： Internet of things distributed computing optimization feature classification

来源：评论

学校读者我要写书评

暂无评论

A quad-form clustered mapping approach for large-scale applications of reconfigurable computing systems

引用

COMPUTERS & ELECTRICAL ENGINEERING 2022年 97卷 107637-107637页

作者： Mohtavipour, Seyed Mehdi Shahhoseini, Hadi Shahriar Iran Univ Sci & Technol Sch Elect Engn Tehran Iran

Reconfigurable computing (RC) systems with hardware implementation feature demonstrated a promising solution for the demand of enormous processing power. However, RC systems suffer from long compilation phases when preparing large-scale applications to execute on reconfigurable hardware. In this paper, a novel quad-form clustered approach has been introduced to manage the hardware resources with low overhead and acceptable quality. For this purpose, two quad-form resource clustering and application partitioning algorithms are proposed to solve the entire mapping problem in intra and inter-cluster parts. Moreover, an analytical distance estimation is derived to properly manage clustered resources in the reconfigurable hardware. Several extensive experimental scenarios on both synthetic and real applications have been conducted to evaluate the effectiveness of the proposed approach in comparison with the state-of-art methods and the results showed that significant improvements have been achieved in terms of quality and time overhead for large-scale applications.

关键词： Resource management Resource mapping Reconfigurable hardware Application compilation distributed computing

来源：评论

学校读者我要写书评

暂无评论

Coded Sparse Matrix Computation Schemes That Leverage Partial Stragglers

引用

IEEE TRANSACTIONS ON INFORMATION THEORY 2022年第6期68卷 4156-4181页

作者： Das, Anindya Bijoy Ramamoorthy, Aditya Iowa State Univ Dept Elect & Comp Engn Ames IA 50011 USA

distributed matrix computations over large clusters can suffer from the problem of slow or failed worker nodes (called stragglers) which can dominate the overall job execution time. Coded computation utilizes concepts from erasure coding to mitigate the effect of stragglers by running "coded" copies of tasks comprising a job;stragglers are typically treated as erasures. While this is useful, there are issues with applying, e.g., MDS codes in a straightforward manner. Several practical matrix computation scenarios involve sparse matrices. MDS codes typically require dense linear combinations of submatrices of the original matrices which destroy their inherent sparsity. This is problematic as it results in significantly higher worker computation times. Moreover, treating slow nodes as erasures ignores the potentially useful partial computations performed by them. Furthermore, some MDS techniques also suffer from significant numerical stability issues. In this work we present schemes that allow us to leverage partial computation by stragglers while imposing constraints on the level of coding that is required in generating the encoded submatrices. This significantly reduces the worker computation time as compared to previous approaches and results in improved numerical stability in the decoding process. Exhaustive numerical experiments on Amazon Web Services (AWS) clusters support our findings.

关键词： Sparse matrices Task analysis Encoding Codes Decoding Web services Numerical stability distributed computing MDS code stragglers condition number sparsity

来源：评论

学校读者我要写书评

暂无评论

On distributability

引用

THEORETICAL COMPUTER SCIENCE 2022年 913卷 151-173页

作者： Peters, Kirstin Nestmann, Uwe Schmitt, Anna Tech Univ Darmstadt Darmstadt Germany Tech Univ Berlin Berlin Germany

The Actor model of computation is the backbone for the design of some of the most successful distributed programming platforms, Erlang and Akka. It builds upon asynchronous message-passing (liberating senders from waiting) and unique message routing (using unique Actor addresses), which enables relatively simple distributed implementations. The Actor model shares these principles with the model of computation underlying JoCaml, which supports the distributed implementation of OCaml components. In this paper, we support the distributability of the two models on theoretical grounds. For this, we investigate three refined variants of the pi-calculus that can be understood as the essence of the two models. We show that neither of the three calculi is able to express a synchronization pattern that is known from the world of Petri nets and that was used by us as one criterion in a distributability hierarchy of process calculi. Moreover, we exhibit three encodings between the three calculi and prove that they each preserve the degree of distributability between source and target terms. (C) 2022 Elsevier B.V. All rights reserved.

关键词： Concurrency theory distributed computing

来源：评论

学校读者我要写书评

暂无评论

Rook Coding for Batch Matrix Multiplication

引用

IEEE TRANSACTIONS ON COMMUNICATIONS 2022年第6期70卷 3641-3654页

作者： Soto, Pedro Fan, Xiaodi Saldivia, Angel Li, Jun CUNY Grad Ctr New York NY 10016 USA Microsoft Redmond WA 98052 USA CUNY Queens Coll New York NY 10016 USA

Matrix multiplication is a fundamental building block in various distributed computing algorithms. In order to multiply large matrices, it is common practice to distribute the computation into multiple tasks running on different nodes. In order to tolerate stragglers among such nodes, various coding schemes have been proposed by adding additional coded tasks. However, most existing coding schemes for matrix multiplication are constructed for only one matrix multiplication, while batch matrix multiplication is common in large-scale distributed computing workloads. In this paper, we propose Rook Coding (RC), a novel polynomial-based coding framework for computing the multiplication of n pairs of matrices in batch. Designed to achieve lower encoding time in practice, we construct RC as polynomials of much simpler forms than existing coding schemes for batch matrix multiplication, achieving a recovery threshold of O(n(log2 3)). Compared to existing coding schemes, RC achieves a lower encoding complexity in practice, because of its simpler forms in the encoding polynomials. Through extensive experiments, we show that RC can save the time of the whole job thanks to its low overhead of encoding.

关键词： distributed computing batch matrix multiplication straggler mitigation coded computing

来源：评论

学校读者我要写书评

暂无评论

WfCommons: A framework for enabling scientific workflow research and development

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2022年 128卷 16-27页

作者： Coleman, Taina Casanova, Henri Pottier, Loic Kaushik, Manav Deelman, Ewa da Silva, Rafael Ferreira Univ Southern Calif Informat Sci Inst 4676 Admiralty Way Suite 1001 Marina Del Rey CA 90292 USA Univ Hawaii Informat & Comp Sci Honolulu HI 96822 USA Univ Southern Calif Dept Comp Sci Los Angeles CA 90007 USA

Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in real-world workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions. (c) 2021 Elsevier B.V. All rights reserved.

关键词： Scientific workflows Workflow management systems Simulation distributed computing Workflow instances

来源：评论

学校读者我要写书评

暂无评论

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2022年 134卷 414-429页

作者： Ejarque, Jorge Badia, Rosa M. Albertin, Loic Aloisio, Giovanni Baglione, Enrico Becerra, Yolanda Boschert, Stefan Berlin, Julian R. D'Anca, Alessandro Elia, Donatello Exertier, Francois Fiore, Sandro Flich, Jose Folch, Arnau Gibbons, Steven J. Koldunov, Nikolay Lordan, Francesc Lorito, Stefano Lovholt, Finn Macias, Jorge Marozzo, Fabrizio Michelini, Alberto Monterrubio-Velasco, Marisol Pienkowska, Marta de la Puente, Josep Queralt, Anna Quintana-Orti, Enrique S. Rodriguez, Juan E. Romano, Fabrizio Rossi, Riccardo Rybicki, Jedrzej Kupczyk, Miroslaw Selva, Jacopo Talia, Domenico Tonini, Roberto Trunfio, Paolo Volpe, Manuela [a]Barcelona Supercomputing Center (BSC) Spain [b]Centre Internacional de Mètodes Numèrics a l’Enginyeria (CIMNE) Spain [c]Universitat Politècnica de Catalunya (UPC) Spain [d]Jülich Supercomputing Centre (JSC) Germany [e]Universitat Politècnica de València (UPV) Spain [f]Atos BDS R&D HPC & AI Software France [g]DtoK Lab Srl Italy [h]Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC) Italy [i]Department of Information Engineering and Computer Science University of Trento Italy [j]Universidad de Málaga (UMA) Spain [k]Istituto Nazionale di Geofisica e Vulcanologia (INGV) Italy [l]Alfred-Wegener-Institut Helmholtz-Zentrum für Polar- und Meeresforschung Germany [m]Consejo Superior Investigaciones Cientificas (CSIC) Spain [n]Eidgenössische Technische Hochschule (ETH) Zürich Switzerland [o]Siemens AG Germany [p]Norwegian Geotechnical Institute (NGI) Norway [q]Poznan Supercomputing and Networking Center (PSNC) Poland

The evolution of High-Performance computing (HPC) platforms enables the design and execution of progressively larger and more complex workflow applications in these systems. The complexity comes not only from the number of elements that compose the workflows but also from the type of computations they perform. While traditional HPC workflows target simulations and modelling of physical phenomena, current needs require in addition data analytics (DA) and artificial intelligence (AI) tasks. However, the development of these workflows is hampered by the lack of proper programming models and environments that support the integration of HPC, DA, and AI, as well as the lack of tools to easily deploy and execute the workflows in HPC systems. To progress in this direction, this paper presents use cases where complex workflows are required and investigates the main issues to be addressed for the HPC/DA/AI convergence. Based on this study, the paper identifies the challenges of a new workflow platform to manage complex workflows. Finally, it proposes a development approach for such a workflow platform addressing these challenges in two directions: first, by defining a software stack that provides the functionalities to manage these complex workflows;and second, by proposing the HPC Workflow as a Service (HPCWaaS) paradigm, which leverages the software stack to facilitate the reusability of complex workflows in federated HPC infrastructures. Proposals presented in this work are subject to study and development as part of the EuroHPC eFlows4HPC project. (C) 2022 Elsevier B.V. All rights reserved.

关键词： High performance computing distributed computing Parallel programming HPC-DA-AI convergence Workflow development Workflow orchestration

来源：评论

学校读者我要写书评

暂无评论

Strategies and System Implementations for Secure Electronic Written Exams

引用

IEEE ACCESS 2022年 10卷 20559-20570页

作者： Gervasi, Osvaldo Perri, Damiano Simonetti, Marco Univ Perugia Dept Math & Comp Sci I-06123 Perugia Italy Univ Florence Dept Math & Comp Sci I-50121 Florence Italy

The pandemic has accelerated the process of digitization in many fields, highlighting some critical issues that have slowed the development of digital technologies in many areas. One of these is undoubtedly that of skills assessment, which on a digital platform is a more streamlined, reliable and faster process and allows some individuals from disadvantaged groups to be able to carry out the exam more easily. We present some strategies for enabling online remote secure written exams and their implementation through the open-source system LibreEOL. The system can be used for university exams (in presence or from remote locations) and training courses of all types to evaluate the effectiveness of learning in itinere, i.e., before, during and after a course, to provide quantitative data on training effectiveness. We outline the architecture of the services underpinning the system and the logical organisation of the various user roles and content relating to questions and exam delivery. We describe the algorithms that have allowed us to develop an environment that appears to be an essential aid both for teachers and students in light of the questionnaires conducted on students. The system is successfully used in the EchemTest, promoted by the European Chemistry Thematic Network, and allows comparison of chemistry training outcomes in different European countries. Despite the importance, it has taken on in the last two years due to the pandemic, such a system makes it possible to carry out a valuable skills assessment.

关键词： Containers Mathematics Computer science Europe Databases Chemistry COVID-19 E-assessment cloud computing distributed computing artificial neural networks human-computer interaction user eXperience COVID-19

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：