检索结果-内蒙古大学图书馆

Hidden Markov Models for Multivariate Panel Data

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Neal, Mackenzie R. Sochaniwsky, Alexa A. McNicholas, Paul D. Department of Mathematics & Statistics McMaster University ON Canada

While advances continue to be made in model-based clustering, challenges persist in modeling various data types such as panel data. Multivariate panel data present difficulties for clustering algorithms because they are often plagued by missing data and dropouts, presenting issues for estimation algorithms. This research presents a family of hidden Markov models that compensate for the issues that arise in panel data. A modified expectation-maximization algorithm capable of handling missing not at random data and dropout is presented and used to perform model estimation. Copyright © 2024, The Authors. All rights reserved.

关键词： expectation maximization algorithm

BAYESIAN EXPERIMENTAL DESIGN VIA CONTRASTIVE DIFFUSIONS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Iollo, Jacopo Heinkelé, Christophe Alliez, Pierre Forbes, Florence Université Grenoble Alpes Inria CNRS G-INP France Cerema Endsum Strasbourg France Université Côte d’Azur Inria France

Bayesian Optimal Experimental Design (BOED) is a powerful tool to reduce the cost of running a sequence of experiments. When based on the Expected Information Gain (EIG), design optimization corresponds to the maximization of some intractable expected contrast between prior and posterior distributions. Scaling this maximization to high dimensional and complex settings has been an issue due to BOED inherent computational complexity. In this work, we introduce a pooled posterior distribution with cost-effective sampling properties and provide a tractable access to the EIG contrast maximization via a new EIG gradient expression. Diffusion-based samplers are used to compute the dynamics of the pooled posterior and ideas from bi-level optimization are leveraged to derive an efficient joint sampling-optimization loop. The resulting efficiency gain allows to extend BOED to the well-tested generative capabilities of diffusion models. By incorporating generative models into the BOED framework, we expand its scope and its use in scenarios that were previously impractical. Numerical experiments and comparison with state-of-the-art methods show the potential of the approach. © 2024, CC BY-SA.

关键词： expectation maximization algorithm

Mini-batch Submodular maximization

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Schwartzman, Gregory JAIST Japan

We present the first mini-batch algorithm for maximizing a non-negative monotone decomposable submodular function, (Equation presented), under a set of constraints. We consider two sampling approaches: uniform and weighted. We first show that mini-batch with weighted sampling improves over the state of the art sparsifier based approach both in theory and in practice. Surprisingly, our experimental results show that uniform sampling is superior to weighted sampling. However, it is impossible to explain this using worst-case analysis. Our main contribution is using smoothed analysis to provide a theoretical foundation for our experimental results. We show that, under very mild assumptions, uniform sampling is superior for both the mini-batch and the sparsifier approaches. We empirically verify that these assumptions hold for our datasets. Uniform sampling is simple to implement and has complexity independent of N, making it the perfect candidate to tackle massive real-world datasets. Copyright © 2024, The Authors. All rights reserved.

关键词： expectation maximization algorithm

Probabilistic Targeted Factor Analysis

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Herculano, Miguel C. Montoya-Blandón, Santiago Adam Smith Business School University of Glasgow United Kingdom

We develop a probabilistic variant of Partial Least Squares (PLS) we call Probabilistic Targeted Factor Analysis (PTFA), which can be used to extract common factors in predictors that are useful to predict a set of predetermined target variables. Along with the technique, we provide an efficient expectation-maximization (EM) algorithm to learn the parameters and forecast the targets of interest. We develop a number of extensions to missing-at-random data, stochastic volatility, and mixed-frequency data for real-time forecasting. In a simulation exercise, we show that PTFA outperforms PLS at recovering the common underlying factors affecting both features and target variables delivering better in-sample fit, and providing valid forecasts under contamination such as measurement error or outliers. Finally, we provide two applications in Economics and Finance where PTFA performs competitively compared with PLS and Principal Component Analysis (PCA) at out-of-sample forecasting. © 2024, CC BY-NC-SA.

关键词： expectation maximization algorithm

Learning Mixtures of Experts with EM

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fruytier, Quentin Mokhtari, Aryan Sanghavi, Sujay Department of Electrical and Computer Engineering The University of Texas at Austin AustinTX United States

Mixtures of Experts (MoE) are Machine Learning models that involve partitioning the input space, with a separate "expert" model trained on each partition. Recently, MoE have become popular as components in today's large language models as a means to reduce training and inference costs. There, the partitioning function and the experts are both learnt jointly via gradient descent on the log-likelihood. In this paper we focus on studying the efficiency of the expectation maximization (EM) algorithm for the training of MoE models. We first rigorously analyze EM for the cases of linear or logistic experts, where we show that EM is equivalent to Mirror Descent with unit step size and a Kullback-Leibler Divergence regularizer. This perspective allows us to derive new convergence results and identify conditions for local linear convergence based on the signal-to-noise ratio (SNR). Experiments on synthetic and (small-scale) real-world data show that EM outperforms the gradient descent algorithm both in terms of convergence rate and the achieved accuracy. Copyright © 2024, The Authors. All rights reserved.

关键词： expectation maximization algorithm

Variational Probabilistic Multi-Hypothesis Tracking

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Shin, Hyo-Sang Xu, Shuoyuan Tsourdos, Antonios School of Aerospace Transport and Manufacturing Cranfield University CranfieldMK43 0AL United Kingdom Cho Chun Shik Graduate School of Mobility Korea Advanced Institute of Science and Technology Daejeon34141 Korea Republic of

This paper proposes a novel multi-target tracking (MTT) algorithm for scenarios with arbitrary numbers of measurements per target. We propose the variational probabilistic multi-hypothesis tracking (VPMHT) algorithm based on the variational Bayesian expectation-maximisation (VBEM) algorithm to overcome limitations in the classic probabilistic multi-hypothesis tracking (PMHT) algorithm. The introduction of variational inference allows VPMHT to handle track-loss much better than the conventional PMHT, while preserving a similar or even better tracking accuracy. Extensive numerical simulations are conducted to validate the proposed algorithm. © 2024, The Authors. All rights reserved.

关键词： expectation maximization algorithm

Fairness in Social Influence maximization via Optimal Transport

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chowdhary, Shubham De Pasquale, Giulia Lanzetti, Nicolas Stoica, Ana-Andreea Dörfler, Florian ETH Zürich Switzerland Eindhoven University of Technology Netherlands Max Planck Institute Tübingen Germany

We study fairness in social influence maximization, whereby one seeks to select seeds that spread a given information throughout a network, ensuring balanced outreach among different communities (e.g. demographic groups). In the literature, fairness is often quantified in terms of the expected outreach within individual communities. In this paper, we demonstrate that such fairness metrics can be misleading since they overlook the stochastic nature of information diffusion processes. When information diffusion occurs in a probabilistic manner, multiple outreach scenarios can occur. As such, outcomes such as "In 50% of the cases, no one in group 1 gets the information, while everyone in group 2 does, and in the other 50%, it is the opposite", which always results in largely unfair outcomes, are classified as fair by a variety of fairness metrics in the literature. We tackle this problem by designing a new fairness metric, mutual fairness, that captures variability in outreach through optimal transport theory. We propose a new seed-selection algorithm that optimizes both outreach and mutual fairness, and we show its efficacy on several real datasets. We find that our algorithm increases fairness with only a minor decrease (and at times, even an increase) in efficiency. © 2024, CC BY.

关键词： expectation maximization algorithm

Cross-Entropy Optimization for Hyperparameter Optimization in Stochastic Gradient-based Approaches to Train Deep Neural Networks

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Kevin Li, Fulu

In this paper, we present a cross-entropy optimization method for hyperparameter optimization in stochastic gradient-based approaches to train deep neural networks. The value of a hyperparameter of a learning algorithm often has great impact on the performance of a model such as the convergence speed, the generalization performance metrics, etc. While in some cases the hyperparameters of a learning algorithm can be part of learning parameters, in other scenarios the hyperparameters of a stochastic optimization algorithm such as Adam [5] and its variants are either fixed as a constant or are kept changing in a monotonic way over time. We give an in-depth analysis of the presented method in the framework of expectation maximization (EM). The presented algorithm of cross-entropy optimization for hyperparameter optimization of a learning algorithm (CEHPO) can be equally applicable to other areas of optimization problems in deep learning. We hope that the presented methods can provide different perspectives and offer some insights for optimization problems in different areas of machine learning and beyond. © 2024, CC BY.

关键词： expectation maximization algorithm

A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Tuan, Yi-Lin Wang, William Yang University of California Santa Barbara United States

Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance. © 2024, CC BY.

关键词： expectation maximization algorithm