检索结果-内蒙古大学图书馆

A hydrological knowledge-informed LSTM model for monthly streamflow reconstruction using distributed data: Application to typical rivers across the Tibetan plateau

引用

JOURNAL OF HYDROLOGY 2025年 649卷

作者： Hou, Shengling Wei, Jiahua Hou, Minglei Xu, Jiaqi Han, Lu Qinghai Univ Sch Civil Engn & Water Resources State Key Lab Plateau Ecol & Agr Lab Ecol Protect & High Qual Dev Upper Yellow Rive Xining 810016 Peoples R China Tsinghua Univ State Key Lab Hydrosci & Engn Beijing 100084 Peoples R China

Long-term streamflow data are essential for water resources planning and management, cascade reservoir scheduling, and understanding the response of water resources to climate change and human activities. Streamflow reconstructions can effectively "fill-in" missing runoff data gaps. However, considering the scarcity of observational monitoring stations and the limitations of distributed hydrological models, the reconstruction of long-term time series of runoff under varying surface and climatic conditions remains a challenge. Here, we propose a hydrological knowledge-informed Long Short-Term Memory (LSTM) model (Hydro-LSTM) for monthly streamflow reconstruction using open-access distributed data. Hydrological knowledge was derived from hydrological governing equations and parameters for each independent water cycle component. The Hydro-LSTM addresses the lack of physical consistency inherent in data-driven models, along with missing observations. The approach was applied to simulate monthly runoff of representative rivers in the Tibetan Plateau (TP) from 1980 to 2018. The results show that streamflow reconstructions for these eight stations yielded favorable levels of performance;trends in dynamic change and the range of runoff in the model training period and test period are consistent with the measured values. Values of NSE, CC, and KGE range between 0.715-0.968, 0.847-0.985, and 0.786-0.969, respectively. The influence of hydrological expertise and distributed data on the model is discussed. The introduction of hydrological knowledge makes the driving elements have hydrological significance, which improves the physical consistency and interpretability of the Hydro-LSTM model. The proposed Hydro-LSTM is expected to (1) achieve accurate and efficient reconstructions of long-term runoff time series using open-access distributed data and limited observations and (2) provide a new perspective for runoff reconstruction and prediction, with promising application prospec

关键词： Long short-term memory network (LSTM) Hydrological knowledge-informed LSTM (Hydro-LSTM) Monthly streamflow reconstruction distributed data data-driven model

来源：评论

学校读者我要写书评

暂无评论

Composite quantile regression for a distributed system with non-randomly distributed data

引用

STATISTICAL PAPERS 2025年第1期66卷 1-30页

作者： Jin, Jun Hao, Chenyan Chen, Yewen Yangzhou Univ Coll Math Sci Yangzhou Peoples R China Univ Georgia Coll Publ Hlth Athens GA USA

The composite quantile regression estimator is widely acknowledged for its robustness and efficiency, offering a compelling alternative to both ordinary least squares and quantile regression estimators in linear models. However, when data is not randomly distributed across different workers in distributed settings, existing methods for composite quantile regression become statistically inefficient. To address this limitation, we present a novel one-step upgraded pilot composite quantile regression method. Our proposed approach involves two essential steps. In the first step, we obtain a pilot estimator by leveraging a small random sample collected from different workers. Subsequently, in the second step, we perform one-step updating based on the pilot estimator, involving the summarization of sample moment quantities on each worker. The resulting estimator is theoretically proven to be as statistically efficient as the composite quantile regression estimator using the entire sample, without relying on restrictive assumptions about randomness. Furthermore, the resulting estimator inherits the robustness and efficiency advantages of the composite quantile regression estimator, while also being computationally efficient in terms of communication cost and storage usage. To validate the practical performance of our proposed method, we conduct numerical studies using simulated and real data, demonstrating its effectiveness in real-world scenarios.

关键词： Communication efficiency distributed system distributed data Robust estimation Statistical efficiency.

来源：评论

学校读者我要写书评

暂无评论

Efficient iterative programs with distributed data collections

引用

JOURNAL OF LOGICAL AND ALGEBRAIC METHODS IN PROGRAMMING 2025年 144卷

作者： Chlyah, Sarah Gesbert, Nils Geneves, Pierre Layaida, Nabil Univ Grenoble Alpes CNRS Inria Grenoble INPLIG F-38000 Grenoble France

Big data programming frameworks have become increasingly important for the development of applications for which performance and scalability are critical. In those complex frameworks, optimizing code by hand is hard and time-consuming, making automated optimization particularly necessary. In order to automate optimization, a prerequisite is to find suitable abstractions to represent programs;for instance, algebras based on monads or monoids to represent distributed data collections. Currently, however, such algebras do not represent recursive programs in a way which allows for analyzing or rewriting them. In this paper, we extend a monoid algebra with a fixpoint operator for representing recursion as a first class citizen and show how it enables new optimizations. Experiments with the Spark platform illustrate performance gains brought by these systematic optimizations.

关键词： Fixpoint operator distributed data Rewrite rules Optimization

来源：评论

学校读者我要写书评

暂无评论

Sequoia: An Accessible and Extensible Framework for Privacy-Preserving Machine Learning over distributed data

引用

Proceedings of the ACM on Management of data 2025年第1期3卷 1-27页

作者： Kaiqiang Xu Di Chai Junxue Zhang Fan Lai Kai Chen Hong Kong University of Science and Technology Hong Kong Hong Kong University of Illinois Urbana-Champaign Urbana-Champaign IL USA

Privacy-preserving machine learning (PPML) algorithms use secure computation protocols to allow multiple data parties to collaboratively train machine learning (ML) models while maintaining their data confidentiality. However, current PPML frameworks couple secure protocols with ML models in PPML algorithm implementations, making it challenging for non-experts to develop and optimize PPML applications, limiting their accessibility and *** propose Sequoia, a novel PPML framework that decouples ML models and secure protocols to optimize the development and execution of PPML applications across data parties. Sequoia offers JAX-compatible APIs for users to program their ML models, while using a compiler-executor architecture to automatically apply PPML algorithms and system optimizations for model execution over distributed data. The compiler in Sequoia incorporates cross-party PPML processes into user-defined ML models by transparently adding computation, encryption, and communication steps with extensible policies, and the executor efficiently schedules code execution across multiple data parties, considering data dependencies and device *** to existing PPML frameworks, Sequoia requires 64%-92% fewer lines of code for users to implement the same PPML algorithms, and achieves 88% speedup of training throughput in horizontal PPML.

关键词： compiler-executor architecture distributed data privacy-preserving machine learning (PPML) secure computation training throughput

来源：评论

学校读者我要写书评

暂无评论

distributed data Clustering by Neural Network Algorithms

Distributed Data Clustering by Neural Network Algorithms

引用

IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus)

作者： Kholod, Ivan I. Rukavitsyn, Andrey N. Reva, Nikita Shorov, Andrey V. St Petersburg Electrotech Univ LETI St Petersburg Russia

ISBN: (纸本)9781728103396

The paper describes an analysis of the possibilities of applying Kohonen self-organizing maps for distributed data. We consider two strategies of distributed data analysis: with intermediate synchronization of results and with synchronization after full or partial construction of maps. Each strategy is considered taking into account both horizontal and vertical data distribution. We provide recommendations for the choice of a strategy depending on the type of data distribution. The obtained conclusions are confirmed by the results of an experiment.

关键词： self-organizing maps neural network deep learning clustering data mining distributed data Big data

来源：评论

学校读者我要写书评

暂无评论

On the suitability of Prototype Selection methods for kNN classification with distributed data

引用

NEUROCOMPUTING 2016年 203卷 150-160页

作者： Valero-Mas, Jose J. Calvo-Zaragoza, Jorge Rico-Juan, Juan R. Univ Alicante Dept Software & Comp Syst Carretera San Vicente Raspeig S-N Alicante 03690 Spain

In the current Information Age, data production and processing demands are ever increasing. This has motivated the appearance of large-scale distributed information. This phenomenon also applies to Pattern Recognition so that classic and common algorithms, such as the k-Nearest Neighbour, are unable to be used. To improve the efficiency of this classifier, Prototype Selection (PS) strategies can be used. Nevertheless, current PS algorithms were not designed to deal with distributed data, and their performance is therefore unknown under these conditions. This work is devoted to carrying out an experimental study on a simulated framework in which PS strategies can be compared under classical conditions as well as those expected in distributed scenarios. Our results report a general behaviour that is degraded as conditions approach to more realistic scenarios. However, our experiments also show that some methods are able to achieve a fairly similar performance to that of the non-distributed scenario. Thus, although there is a clear need for developing specific PS methodologies and algorithms for tackling these situations, those that reported a higher robustness against such conditions may be good candidates from which to start. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Prototype Selection distributed data k-Nearest Neighbour Experimental study

来源：评论

学校读者我要写书评

暂无评论

Incremental Detection of Inconsistencies in distributed data

引用

IEEE TRANSACTIONS ON KNOWLEDGE AND data ENGINEERING 2014年第6期26卷 1367-1383页

作者： Fan, Wenfei Li, Jianzhong Tang, Nan Yu, Wenyuan Univ Edinburgh Sch Informat Lab Fdn Comp Sci LFCS Edinburgh EH8 9AB Midlothian Scotland Harbin Inst Technol Sch Comp Sci & Technol Dept Comp Sci & Engn Heilongjiang 150001 Peoples R China Qatar Fdn Qatar Comp Res Inst QCRI Doha Qatar

This paper investigates incremental detection of errors in distributed data. Given a distributed database D, a set Sigma of conditional functional dependencies (CFDs), the set V of violations of the CFDs in D, and updates Delta D to D, it is to find, with minimum data shipment, changes Delta V to V in response to Delta D. The need for the study is evident since real-life data is often dirty, distributed and frequently updated. It is often prohibitively expensive to recompute the entire set of violations when D is updated. We show that the incremental detection problem is NP-complete for database D that is partitioned either vertically or horizontally, even when Sigma and D are fixed. Nevertheless, we show that it is bounded: there exist algorithms to detect errors such that their computational cost and data shipment are both linear in the size of Delta D and Delta V, independent of the size of the database D. We provide such incremental algorithms for vertically partitioned data and horizontally partitioned data, and show that the algorithms are optimal. We further propose optimization techniques for the incremental algorithm over vertical partitions to reduce data shipment. We verify experimentally, using real-life data on Amazon Elastic Compute Cloud (EC2), that our algorithms substantially outperform their batch counterparts.

关键词： Incremental algorithms distributed data conditional functional dependencies error detection

来源：评论

学校读者我要写书评

暂无评论

Least Squares Model Averaging for distributed data

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2023年第1期24卷 1-59页

作者： Zhang, Haili Liu, Zhaobo Zou, Guohua Shenzhen Polytech Univ Inst Appl Math Shenzhen 518055 Peoples R China Shenzhen Univ Inst Adv Study Shenzhen 518060 Peoples R China Capital Normal Univ Sch Math Sci Beijing 100048 Peoples R China

Divide and conquer algorithm is a common strategy applied in big data. Model averaging has the natural divide-and-conquer feature, but its theory has not been developed in big data scenarios. The goal of this paper is to fill this gap. We propose two divide-and conquer-type model averaging estimators for linear models with distributed data. Under some regularity conditions, we show that the weights from Mallows model averaging criterion converge in L-2 to the theoretically optimal weights minimizing the risk of the model averaging estimator. We also give the bounds of the in-sample and out-of-sample mean squared errors and prove the asymptotic optimality for the proposed model averaging estimators. Our conclusions hold even when the dimensions and the number of candidate models are divergent. Simulation results and a real airline data analysis illustrate that the proposed model averaging methods perform better than the commonly used model selection and model averaging methods in distributed data cases. Our approaches contribute to model averaging theory in distributed data and parallel computations, and can be applied in big data analysis to save time and reduce the computational burden.

关键词： consistency distributed data divide and conquer algorithm Mallows' criterion model averaging optimality

来源：评论

学校读者我要写书评

暂无评论

Robustness analysis of arbitrarily distributed data-based recommendation methods

引用

EXPERT SYSTEMS WITH APPLICATIONS 2016年 44卷 217-229页

作者： Yilmazel, Burcu Yurekli Kaleli, Cihan Anadolu Univ Dept Comp Engn TR-26555 Eskisehir Turkey

Due to different shopping routines of people, rating preferences of many customers might be partitioned between two parties. Since two different e-companies might sell products from the same range to the identical set of customers, the type of data partition is called arbitrarily. In the case of arbitrarily distributed data, it is a challenge to produce accurate recommendations for those customers, because their ratings are split. Therefore, researchers propose methods for enabling data holders' collaboration. In this scenario, privacy becomes a deterrent barrier for collaboration, accordingly, the introduced solutions include private protocols for protecting parties' confidentiality. Although, private protocols encourage data owners to collaborate, they introduce a new drawback for partnership. Since, whole data is distributed and parties do not have full control of data, any malicious user, who knows that two parties collaborate, can easily insert shilling profiles to system by partitioning them between data holders. Parties can have trouble to find such profile injection attacks by employing existing detection methods because of they are arbitrarily distributed. Since profile injection attacks can easily performed on arbitrarily distributed data-based recommender systems, quality, and reliability of such systems decreases, and it causes angry customers. Therefore, in this paper, we try to describe aforementioned problems with arbitrarily distributed data-based recommender systems. As a first step, we analyze robustness of proposed arbitrarily distributed data-based recommendation methods against six well-known shilling attack types. Secondly, we explain why existing detection methods cannot detect malicious user profiles in distributed data. We perform experiments on a well-known movie data set, and according to our results, arbitrarily distributed data-based recommendation methods are vulnerable to shilling attacks. (C) 2015 Elsevier Ltd. All rights reserved.

关键词： Robustness Detection Shilling distributed data Arbitrarily Privacy Collaborative filtering

来源：评论

学校读者我要写书评

暂无评论

PRIVACY-PRESERVING NAIVE BAYESIAN CLASSIFIER-BASED RECOMMENDATIONS ON distributed data

引用

COMPUTATIONAL INTELLIGENCE 2015年第1期31卷 47-68页

作者： Kaleli, Cihan Polat, Huseyin Anadolu Univ Dept Comp Engn TR-26470 Eskisehir Turkey

data collected for recommendation purposes might be distributed among various e-commerce sites, which can collaboratively provide more accurate predictions. However, because of privacy concerns, they might not want to work together. If privacy measures are provided, they may decide to become involved in prediction generation processes. We propose privacy-preserving schemes eliminating e-commerce sites' privacy concerns for providing predictions on distributed data. We investigate how to achieve naive Bayesian classifier-based recommendations when data are distributed horizontally or vertically among multiple parties, even competing ones, without greatly violating their confidentiality. We analyze our schemes in terms of privacy and additional costs and show that they do not deeply violate online vendors' secrecy and they cause insignificant overhead costs. We also perform experiments on real data, evaluate our outcomes, and provide suggestions. Our empirical results show that our schemes produce more accurate predictions.

关键词： privacy recommendation distributed data naive Bayesian classifier electronic commerce

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：