In the big data era, scalability has become a crucial requirement for any useful computational model. Probabilistic graphical models are very useful for mining and discovering data insights, but they are not scalable ...
详细信息
ISBN:
(纸本)9781479956661
In the big data era, scalability has become a crucial requirement for any useful computational model. Probabilistic graphical models are very useful for mining and discovering data insights, but they are not scalable enough to be suitable for big dataproblems. Bayesian Networks particularly demonstrate this limitation when their data is represented using few random variables with a massive set of outcome values for each of them. With hierarchicaldata -data that is arranged in a treelike structure with several levels -one would expect to see hundreds of thousands or millions of values distributed over even just a small number of levels. When modeling this kind of hierarchicaldata across large data sets, Bayesian networks become unsuitable for representing the probability distributions for the following reasons: i) each level represents a single random variable with hundreds of thousands of values, ii) the number of levels is usually small, so there are also few random variables, and iii) the structure of the network is predefined since the dependency is modeled top-down from each parent to each of its child nodes. In this paper we propose a scalable probabilistic graphical model to overcome these limitations for massivehierarchicaldata. We believe the proposed model will lead to an easily-scalable, more readable, and expressive implementation for problems that require probabilistic-based solutions for massive amounts of hierarchicaldata. We successfully applied this model to solve two different challenging probabilistic-based problems on massivehierarchicaldata sets for different domains, namely, bioinformatics and latent semantic discovery over search logs.
暂无评论