[读书笔记][更新中] Aad van der Vaart "Asymptotic Statistics"

[hitori_bocchi]

今天太忙了，在读导师要求的文献，周末再更吧

[lemma_]

支持优质创作（（

[nomana]

喔喔喔好好好我之后可能也会需要学这个
关注，支持

[hitori_bocchi]

关于接下来的几次更新: 内容是导师要求看的一篇文献，虽然不是van der Vaart上的，但也是关于semi-parametric model （里面的一些内容应该是来自Aad van der Vaart早年的讲义），因此也放在这里更。

Semiparametric Doubly Robust Targeted Double Machine Learning: A Review by Edward H. Kennedy (Associate Professor, CMU Stats & Data Sciences Dept.)

1.Introduction

"In this review we cover the basics of efficient nonparametric parameter estimation, with a focus on parameters that arise in causal inference problems. We review both efficiency bounds (i.e., what is the best possible performance for estimating a given parameter?) and the analysis of particular estimators (i.e., what is this estimator’s error, and does it attain the efficiency bound?) under weak assumptions."

Comment: Why nonparametric model? 想了解背后的思想可以看Mark van der Laan（UC Berkeley 统计系杰出教授, COPSS奖得主，在半参理论、生存分析等领域有重要贡献）的一篇短文："Why We Need a Statistical Revolution" https://senseaboutscienceusa.org/super-learning-and-the-revolution-in-knowledge/

1.1 Set-up and notations

Suppose we observe a sample of independent observations $(Z_1,\cdots ,Z_n)$ all identically distributed according to some unknown probability distribution $\mathbb{P}$, which is assumed to lie in some model (i.e., set of distributions) $\mathcal{P}$. Our goal is to estimate some structured combination of components, called a target parameter $\psi :\mathcal{P}\to {\mathbb{R}}^q$ (which is a functional).

At times we subscript expectations and other quantities with the distribution under which they are taken, i.e., ${\mathbb{E}}_P(Y\mid X=x)$ for an expectation under distribution $P$. When the distribution is clear from context, we sometimes omit subscripts; in general, quantities without subscripts are meant to be taken under some generic $P$ in the model, or else under the true distribution $\mathbb{P}$. We denote convergence in distribution by $⇝$ and convergence in probability by $\stackrel{p}{\to }$. We use standard big-oh and little-oh notation, i.e., $X_n=O_{\mathbb{P}}\left(r_n\right)$ means $X_n/r_n$ is bounded in probability and $X_n=o_{\mathbb{P}}\left(r_n\right)$ means $X_n/r_n\stackrel{p}{\to }0$. To ease notation we sometimes omit arguments for functions of multiple arguments, e.g., $\phi =\phi (z;P)$ when the arguments are clear or secondary to the discussion. We use ${\mathbb{P}}_n$ to denote the empirical measure so that sample averages are written as ${\mathbb{P}}_n(f)={\mathbb{P}}_n\{f(Z)\}=\frac{1}{n}{\sum }_{i}f\left(Z_i\right)$. For a possibly random function $\hat{f}$, we similarly write $\mathbb{P}(\hat{f})=\mathbb{P}\{\hat{f}(Z)\}=\int \hat{f}(z)d\mathbb{P}(z)$, and we let $\Vert \hat{f}{\Vert }^2=\int \hat{f}(z{)}^{2}d\mathbb{P}(z)$ denote the squared $L_2(\mathbb{P})$ norm.

2.Benchmarks: Nonparametric Efficiency Bounds

“After having selected an appropriate target parameter $\psi$ matching the scientific question of interest, identifying (or bounding) it under appropriate causal (在因果推断中，identification指在一些假设下，将target parameter表示为观测数据的函数) or other assumptions, and laying out a statistical model $\mathcal{P}$ (which in our case will be nonparametric), a next line of business is to understand lower bounds or benchmarks for estimation error. In other words, how well can we possibly hope to estimate the parameter $\psi$ over the model $\mathcal{P}$?

There are two parts to showing optimality: (i) that no estimator can do better than some benchmark, and (ii) that a particular estimator does in fact attain that benchmark. Part (i) is discussed in this section, and part (ii) in the next section.
”
A classic benchmarking or lower bound result for smooth parametric models is the Cramer-Rao bound. In its simplest form, this result states that for smooth parametric models $\mathcal{P}=\left\{P_{\theta }:\theta \in \mathbb{R}\right\}$ and smooth functionals (i.e., with $P_{\theta }$ and $\psi (\theta )$ differentiable in $\theta$ ), the variance of any unbiased estimator $\hat{\psi }$ must satisfy

$${\mathrm{var}}_{\theta }(\hat{\psi })\ge \frac{{\psi }^{\prime }(\theta {)}^2}{{\mathrm{var}}_{\theta }\left\{s_{\theta }(Z)\right\}}$$

where $s_{\theta }(z)=\frac{\partial }{\partial \theta }\log p_{\theta }(z)$ is the score function, i.e., no unbiased estimator can have smaller variance than the above ratio.

A standard way to benchmark estimation error more generally is through minimax lower bounds of the form

$$\underset{\hat{\psi }}{\inf }\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P\left[\{\hat{\psi }-\psi (P){\}}^2\right]\ge R_n.$$

Comment: 在这里先介绍一下minimax theory，主要参考Larry Wasserman（CMU统计系冠名教授，COPSS奖得主，在非参、贝叶斯等领域有重要贡献）的讲义 https://www.stat.cmu.edu/~larry/=sml/Minimax.pdf

“When solving a statistical learning problem, there are often many procedures to choose from. This leads to the following question: how can we tell if one statistical learning procedure is better than another? One answer is provided by minimax theory which is a set of techniques for finding the minimum, worst case behavior of a procedure.”

Let $\mathcal{P}$ be a set of distributions and let $X_1,\dots ,X_n$ be a sample from some distribution $P\in \mathcal{P}$. Let $\theta (P)$ be some function of $P$. For example, $\theta (P)$ could be the mean of $P$, the variance of $P$ or the density of $P$. Let $\hat{\theta }=\hat{\theta }\left(X_1,\dots ,X_n\right)$ denote an estimator. Given a metric $d$, the minimax risk is

$$R_n\equiv R_n(\mathcal{P})=\underset{\hat{\theta }}{\inf }\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P[d(\hat{\theta },\theta (P))]$$

where the infimum is over all estimators.

Comment: 如何理解minimax risk？这里参考了一个知乎回答 https://www.zhihu.com/question/347730562/answer/835333769

max指的是 ${\sup }_{P\in \mathcal{P}}{\mathbb{E}}_P[d(\hat{\theta },\theta (P))]$ 是对所有可能的'真实'的 $\theta (P)$ 取得的上界，也就是在选定了估计量 $\hat{\theta }$ 之后，让 $\theta (P)$ 在statistical model（$\mathcal{P}$，包含真实分布，即 $\mathbb{P}\in \mathcal{P}$）内到处乱跑来最大化risk，以此来确定这个选定的估计量 $\hat{\theta }$ 可能遇到的最大的risk是多少。

而min才是选取估计量 $\hat{\theta }$ 来得到这些最大的risk的下界。所以实际上一个估计量是minimax的就说明它在最坏情况（无论 $\theta (P),P\in \mathcal{P}$ 怎么取）下的表现 ${\sup }_{P\in \mathcal{P}}{\mathbb{E}}_P[d(\hat{\theta },\theta (P))]$ 也是最好的。

[hitori_bocchi]

之后慢慢更（凑满8个字）

[wumingshi]

我感觉某些名词直接用原文，用中英（是不是真的英也无所谓）混写的风格就挺好的，Borel 连我一个纯外行都觉得比音译更自然（

[hitori_bocchi]

@wumingshi 好的，谢谢建议

[hitori_bocchi]

"Can the above Cramer-Rao bounds (work for smooth parametric models) be exploited to construct lower bound benchmarks for larger semi- or nonparametric models as well?"

2.1 Parametric Submodels

"The standard way to connect classic Cramer-Rao bounds for parametric models to larger more complicated nonparametric models is through a technical device called the parametric submodel"

Definition 1. A parametric submodel is a smooth parametric model ${\mathcal{P}}_{ϵ}=\left\{P_{ϵ}:ϵ\in \mathbb{R}\right\}$ that satisfies (i) ${\mathcal{P}}_{ϵ}\subseteq \mathcal{P}$, and (ii) $P_{ϵ=0}=\mathbb{P}$.

Thus, in words, a parametric submodel is a parametric model that (i) is contained in the larger model $\mathcal{P}$ of interest, and (ii) equals the true distribution at $ϵ=0$, i.e., contains the truth $\mathbb{P}$.

The high-level idea behind using submodels is that it is never harder to estimate a parameter over a smaller model, relative to a larger one in which the smaller model is contained. So any lower bound for a submodel will also be a valid lower bound for the larger model $\mathcal{P}$.

Comment: 如何理解 “any lower bound for a submodel will also be a valid lower bound for the larger model $\mathcal{P}$”？这里附上van der Vaart "Asymptotic Statistics" Chap. 25 Semiparametric Models 里的讲解。

"To estimate the parameter $\psi (P)$ given the model $\mathcal{P}$ is certainly harder than to estimate this parameter given that $P$ belongs to a submodel ${\mathcal{P}}_0\subset \mathcal{P}$. For every smooth parametric submodel ${\mathcal{P}}_0=\left\{P_{\theta }:\theta \in \Theta \right\}\subset \mathcal{P}$, we can calculate the Fisher information for estimating $\psi \left(P_{\theta }\right)$. Then the information for estimating $\psi (P)$ in the whole model is certainly not bigger than the infimum of the informations over all submodels (recall that The Cramér–Rao bound states that the inverse of the Fisher information is a lower bound on the variance of any unbiased estimator of $\theta$). We shall simply define the information for the whole model as this infimum. A submodel for which the infimum is taken (if there is one) is called least favorable or a "hardest" submodel."

Comment: （以下内容改写自Mark van der Laan的 STAT C245B Survival Analysis and Causality 的课程材料）The benchmark/lower bound in a minimax theory sense for the target parameter $\psi$ is tightly connected to looking at the derivative of it (the functional derivative). We are interested in the behavior of $\psi$ for local perturbations around $\mathbb{P}$. In particular, the derivative of $\psi$ and the steepness of this derivative define the difficulty of the estimation problem. 因此，我们需要一套关于functional derivative的理论。

[hitori_bocchi]

Directional derivative & pathwise derivative for functional的东西下次再更新。

[hitori_bocchi]

半参后面的理论确实有点复杂，会涉及一些泛函的东西，我不打算写的过于理论，更多还是intuition吧

[hitori_bocchi]

不打算更directional derivative & pathwise derivative for functional和更多的半参理论的东西了，实在是太多了，够写一本书的。这篇文章本来也是一个偏实用的指南，还是不想太偏离主旨...

[hitori_bocchi]

It turns out that, for the purposes of constructing lower bound benchmarks for functional estimation, it often suffices to use one-dimensional parametric submodels. A common choice of submodel for nonparametric $\mathcal{P}$ is, for some mean-zero function $h:\mathcal{Z}\to \mathbb{R}$,

$$p_{ϵ}(z)=d\mathbb{P}(z)\{1+ϵh(z)\}$$

where $\Vert h{\Vert }_{\infty }\le M<\infty$ and $ϵ<1/M$ so that $p_{ϵ}(z)\ge 0$. Note for this submodel the score function is ${\frac{\partial }{\partial ϵ}\log p_{ϵ}(z)|}_{ϵ=0}={\frac{\partial }{\partial ϵ}\log \{1+ϵh(z)\}|}_{ϵ=0}=h(z)$. Therefore the Cramer-Rao lower bound for some $P_{ϵ}$ in the example one-dimensional submodel ${\mathcal{P}}_{ϵ}$ above is given by

$$\frac{{\psi }^{\prime }{\left(P_{ϵ}\right)}^2}{{\mathrm{var}}_{P_{ϵ}}\left\{s_{ϵ}(Z)\right\}}=\frac{{\left\{{\frac{\partial }{\partial ϵ}\psi \left(P_{ϵ}\right)|}_{ϵ=0}\right\}}^2}{{\mathbb{E}}_{P_{ϵ}}\left\{h(Z{)}^2\right\}}.$$

Comment: Why one-dimensional submodel? 详细的说明见 Michael Kosorok "Introduction to Empirical Processes and Semiparametric Inference" Chap. 18。

还需要说明的一点是为什么我们选择了$p_{ϵ}(z)=d\mathbb{P}(z)\{1+ϵh(z)\}$作为submodel（以下内容改写自Mark van der Laan的 STAT C245B Survival Analysis and Causality 的课程材料）。

We want to define a type of differentiability of $\psi :\mathcal{P}\to {\mathbb{R}}^q$, where $\psi$ is the target parameter.

We could use the definition of a directional derivative in direction $h$ :

$$d\psi (\mathbb{P})(h)={\frac{d}{dϵ}\psi (\mathbb{P}+ϵh)|}_{ϵ=0}$$

However, $\mathbb{P}+ϵh$ might not be a path through $\mathcal{P}$, and thus ill defined. We need to define a derivative along paths that are submodels of $\mathcal{P}$.

Let $\mathcal{P}$ be nonparametric. We define a class of paths such that:

$$p_{ϵ}(z)=d\mathbb{P}(z)\{1+ϵh(z)\}$$

Two key assumptions necessary for it to be a proper submodel are as follows:

• $h$ is uniformly bounded
• ${\mathbb{E}}_{P}h(z)=0$

For $ϵ\in (-\delta ,\delta )$ with $\delta =\frac{1}{\Vert h{\Vert }_{\infty }}$, this is a submodel.

To see why, first note that for the paths to be a proper density, we need:

1. $d\mathbb{P}(z)\{1+ϵh(z)\}⩾0$

Sketch proof:

Let $h(z)$ be uniformly bounded and $h(z)=\Vert h{\Vert }_{\infty }$. If $ϵ⩽|\delta |,\{1+ϵh(z)\}⩾0$. Therefore, for $ϵ$ sufficiently small and $h$ uniformly bounded, $d\mathbb{P}(z)\{1+ϵh(z)\}⩾0$.

2. $\int \{1+ϵh(z)\}d\mathbb{P}(z)=1$

Sketch proof:

Note that $\int \{1+ϵh(z)\}d\mathbb{P}(z)=\int d\mathbb{P}(z)+ϵ\int h(z)d\mathbb{P}(z)=1$ since $p$ is a proper density and $\int h(z)d\mathbb{P}(z)={\mathbb{E}}_{P}h(z)=0$ by assumption.

Now consider the score of this submodel.

$$\begin{array}{rl}{\frac{\delta }{\delta ϵ}\log \frac{d{P}_{ϵ}}{d\mathbb{P}}|}_{ϵ=0}& ={\frac{\delta }{\delta ϵ}\log \{1+ϵh(z)\}|}_{ϵ=0}\\ & ={\frac{h(z)}{1+ϵh(z)}|}_{ϵ=0}\\ & =h(z).\end{array}$$

"Since any lower bound for the submodel ${\mathcal{P}}_{ϵ}$ is also a lower bound for $\mathcal{P}$, the best and most informative is the greatest such lower bound. Can we say anything about the best such lower bound for generic functionals and/or submodels?"

2.2 Pathwise Differentiability

Recall the Cramer-Rao bound

$$\frac{{\left\{{\frac{\partial }{\partial ϵ}\psi \left(P_{ϵ}\right)|}_{ϵ=0}\right\}}^2}{{\mathbb{E}}_{P_{ϵ}}\left\{s_{ϵ}(Z{)}^2\right\}}$$

for submodel ${\mathcal{P}}_{ϵ}$ described in the previous subsection. To find the best such lower bound, we would like to optimize the above over all $P_{ϵ}$ in some submodels. It is not a priori clear how generally this can be accomplished, since different functionals $\psi$ could yield very different numerators. Therefore let us first consider what we can say about the derivative in the numerator, for a large class of pathwise differentiable functionals.

Namely, suppose the functional $\psi :\mathcal{P}\mapsto \mathbb{R}$ is smooth, as a map from distributions to the reals, in the sense that it admits a kind of distributional Taylor expansion

$$\psi (\bar{P})-\psi (P)=\int \phi (z;\bar{P})d(\bar{P}-P)(z)+R_2(\bar{P},P)$$

for distributions $\bar{P}$ and $P$, often called a von Mises expansion, where $\phi (z;P)$ is a mean-

zero, finite-variance function satisfying $\int \phi (z;P)dP(z)=0$ and $\int \phi (z;P{)}^{2}dP(z)<\infty$, and $R_2(\bar{P},P)$ is a second-order remainder term (which means it only depends on products or squares of differences between $\bar{P}$ and $P)$.

Intuitively, the von Mises expansion above is just an infinite-dimensional or distributional analog of a Taylor expansion, with $\phi (z;Q)$ acting as a usual derivative term; it describes how the functional $\psi$ changes locally when the distribution changes from $P$ to $\bar{P}$. For example, when $Z\in \{1,\dots ,k\}$ is discrete and so $\bar{P}$ and $P$ have $k$ countable components, the von Mises expansion reduces to a standard multivariate Taylor expansion with

$$R_2(\bar{P},P)=\psi \left({\bar{p}}_1,\dots ,{\bar{p}}_k\right)-\psi \left(p_1,\dots ,p_k\right)-{\sum _j\frac{\partial }{\partial t_j}\psi \left(t_1,\dots ,t_k\right)|}_{t=\bar{p}}\left({\bar{p}}_j-p_j\right).$$

[hitori_bocchi]

今天就更到这里，去看碧蓝档案3周年fes了（

[nomana]

我爱你🤟波奇酱

[hitori_bocchi]

@nomana 谢谢~

[nomana]

波奇酱好久没更了

[hitori_bocchi]

@nomana 最近科研太忙了，等放假了再更

[hashhash]

@hitori_bocchi 原来你也是档友!

[hitori_bocchi]

@hashhash 阿罗娜可爱!

[hitori_bocchi]

@hashhash 可以加贵校碧蓝档案群155199376聊天吹水