# Shannon entropy and space vector dimension

Define the entropy of a random variable $$X$$ by : $$H(X):=\sum_{x\in X(\Omega)}p(x)\log\left(\frac{1}{p(x)} \right),$$ with $$0\log 0:=0$$ and $$p(x)=P(X=x)$$.

A fundamental propriety satisfied by $$H$$ is : $$H(X)=\log |range(X)|,$$ if and only if $$X$$ is a uniform random variable.

Let $$E$$ be a finite-dimensional space vector over a finite field $$\mathbb{F}$$, then and $$E^{*}$$ the dual space. Then, if $$X$$ is a uniform variable over $$E^{*}$$ : $$H(X)=\log |range(X)|=\log |E^{*}|=\log |\mathbb{F}^{\text{dim}(E)}|)=\text{dim}(E)\log|\mathbb{F}|.$$

My question is : if $$U,V$$ are two linear subspaces of $$E$$ an $$X,Y$$ two uniform variables over $$U$$ and $$V$$, how do you prove that : $$H(X,Y)=\text{dim}(U+V)\log|\mathbb{F}|.$$

Note : In the following article

https://terrytao.wordpress.com/2017/03/01/special-cases-of-shannon-entropy/

Tao wrote that "the joint random variable $$(X,Y)$$ determines the random linear function $$f$$ $$\color{red}{\text{on the union } U \cup V}$$ on the two spaces, and $$\color{red}{\text{thus by linearity on the Minkowski sum }U+V}$$ as well; thus $$(X,Y)$$ is equivalent to the restriction of $$f$$ to $$U+V.$$ In particular, $$H(X,Y) = \mathrm{dim}(U+V) \log |\mathbf{F}|$$.",

but I don't really understand the 'red' argument.

• You left out a key part from Tao's writing. $X$ and $Y$ are not just any random variables. $X$ is gotten by restricting a random function $f\in E^*$ to the subspace $U$, and similarly $Y$ is gotten by restricting the same random function to $V$. Consequently the components $(X,Y)$ automatically agree on $U\cap V$, and thus yield a well defined linear function on $U+V$. Jun 19 '19 at 6:06
• It's ok for the definition of $X$ and $Y$, but I don't understand the link between '$(X,Y)$ automatically agree on $U\cap V$' and it defines a 'linear function on $U+V$'. Jun 19 '19 at 7:11
• In the beginning Tao has a random linear function $f:E\to\Bbb{F}$. Then $X$ is the restriction of $f$ to the subspace $U$. Similarly $Y$ is the restriction to the subspace $V$. Therefore the restrictions of both $X$ and $Y$ to the subspace $U\cap V$ agree. Given two linear function $X:U\to\Bbb{F}$ and $Y:V\to\Bbb{F}$ that agree on $U\cap V$, we can define a unique linear function $(X,Y)$ from $U+V$ to $\Bbb{F}$ by declaring $(X,Y)(u+v)=X(u)+Y(v)$. This is well-defined even though we may have $u+v=u'+v'$ with $u'\neq u$ and $v\neq v'$. Jun 19 '19 at 10:06
• (cont'd) This is because $u-u'=v'-v$ is an element of $U\cap V$, so $X(u-u')=Y(v'-v)$. My understanding is that this may have been your problem, but I'm prepared to be wrong. Jun 19 '19 at 10:08

Let $$X\sim\mathcal{U}(E^{*})$$ and $$X_1=X_{|U}$$ and $$X_2=X_{|V}$$. These two functions are equal on $$U\cap V$$, and we can define a unique function $$(X,Y)$$ from $$U+V$$ to $$\mathbb{F}$$ by declaring : $$(X,Y)(u+v)=X(u)+Y(v).$$ This function is well defined because if $$u+v=u'+v'$$, $$(X,Y)(u+v)=(X,Y)(u'+v')$$.