We observe an infinite sequence of independent identically distributed random variables X-1, X-2,& mldr;drawn from an unknown distribution p over [n] , and our goal is to estimate the entropy H(p) = -E[logp(X)] wi...
详细信息
We observe an infinite sequence of independent identically distributed random variables X-1, X-2,& mldr;drawn from an unknown distribution p over [n] , and our goal is to estimate the entropy H(p) = -E[logp(X)] within an epsilon -additive error. To that end, at each time point we are allowed to update a finite-state machine with S states, using a possibly randomized but time-invariant rule, where each state of the machine is assigned an entropy estimate. Our goal is to characterize the minimax memory complexity S & lowast;of this problem, which is the minimal number of states for which the estimation task is feasible with probability at least 1 - delta asymptotically, uniformly in p. Specifically, we show that there exist universal constants C-1 and C-2 such that S* <= C-1 & sdot;n(log n)(4)/epsilon(2)delta for epsilon not too small, and S* >= C-2 & sdot;max{n, log n/epsilon} for epsilon not too large. The upper bound is proved using approximate counting to estimate the logarithm of p, and a finitememory bias estimation machine to estimate the expectation operation. The lower bound is proved via a reduction of entropy estimation to uniformity testing. We also apply these results to derive bounds on the memory complexity of mutual information estimation.
The problem of statistical inference in its various forms has been the subject of decades-long extensive research. Most of the effort has been focused on characterizing the behavior as a function of the number of avai...
详细信息
The problem of statistical inference in its various forms has been the subject of decades-long extensive research. Most of the effort has been focused on characterizing the behavior as a function of the number of available samples, with far less attention given to the effect of memory limitations on performance. Recently, this latter topic has drawn much interest in the engineering and computer science literature. In this survey paper, we attempt to review the state-of-the-art of statistical inference under memory constraints in several canonical problems, including hypothesis testing, parameter estimation, and distribution property testing/estimation. We discuss the main results in this developing field, and by identifying recurrent themes, we extract some fundamental building blocks for algorithmic construction, as well as useful techniques for lower bound derivations.
This correspondence presents a new recursive least squares (RLS) adaptive algorithm. The proposed computational scheme uses a finite window by means of a lemma for the system matrix inversion that is, for the first ti...
详细信息
This correspondence presents a new recursive least squares (RLS) adaptive algorithm. The proposed computational scheme uses a finite window by means of a lemma for the system matrix inversion that is, for the first time, stated and proven here. The new algorithm has excellent tracking capabilities. Moreover, its particular structure allows for stabilization by means of a quite simple method. Its stabilized version performs very well not only for a white noise input but also for nonstationary inputs as well. It is shown to follow music, speech, environmental noise, etc, with particularly good tracking properties. The new algorithm can be parallelized via a simple technique. Its parallel form is very fast when implemented with four processors.
暂无评论