中国人民大学王菲菲副教授：非均匀和非随机分布式数据中的分布式一步改进估计

嘉宾简介
王菲菲，中国人民大学统计学院副教授，北京大学光华管理学院统计学博士。研究上关注文本挖掘、社交网络分析、大数据建模等，研究论文发表于JOE，JBES，Statistics in Medicine、中国科学（数学）等国内外权威杂志上，主持并参与了多项省部级项目，包括国家自然科学基金、国家重点研发项目等。

报告摘要
One-shot-type (or divide-and-conquer) estimators havebeen widely used for distributed statistical analysis. However, theiroutstanding statistical efficiency hinges on two critical conditions. The firstis the uniformity condition, which requires that the sample sizes allocated todifferent Workers should be as comparable as possible. The second one is therandomness condition, which requires that the data should be distributed acrossWorkers as randomly as possible. Considering that both conditions are oftenviolated in practice, we prove both theoretically and empirically in this workthat the violation of either condition can seriously degrade the statisticalefficiency of one-shot estimators, or even make them inconsistent. To fix thisproblem, we propose a novel one-step upgraded pilot (OSUP) method. In the firststep of the algorithm, a pilot estimate is computed based on randomly selectedsamples from different Workers. In the second step, we conduct one-stepupdating based on the pilot estimate by summarizing the derivative informationon each Worker. We show theoretically that the resulting OSUP estimator can beas statistically efficient as the whole sample maximum likelihood estimatorwithout any restrictive assumption about distribution uniformity andrandomness. Extensive numerical studies are presented to demonstrate the finitesample performance of the OSUP estimator. Finally, by way of an illustration,an American Airlines dataset is analyzed on a Spark cluster.