1. ML中為什么需要矩陣求導(dǎo)

• 簡(jiǎn)潔
  用方程式表示如下：
  $$y_1=w_1{X}_{11}+w_2{X}_{12}\text{\hspace{1em}(1)}$$
  $$y_2=w_1{X}_{21}+w_2{X}_{22}\text{\hspace{1em}(2)}$$
  轉(zhuǎn)換成矩陣表示如下：
  $$Y=XW\text{\hspace{1em}(3)}$$
  $$Y=\left[\begin{array}{c}{y}_1\\ \\ y_2\end{array}\right],X=\left[\begin{array}{ccc}{x}_{11}& & x_{12}\\ & & \\ x_{21}& & x_{22}\end{array}\right],W=\left[\begin{array}{c}{w}_1\\ \\ w_2\end{array}\right]\text{\hspace{1em}(4)}$$

• 快速
  當(dāng)使用python 中的numpy 庫(kù)時(shí)候，在相對(duì)于 for 循環(huán)，Numpy 本身的計(jì)算提速相當(dāng)快

• 源代碼

import time
import numpy as npif __name__ == "__main__":N = 1000000a = np.random.rand(N)b = np.random.rand(N)start = time.time()c = np.dot(a,b)stop = time.time()print(f"c={c}")print("vectorized version: " + str(1000*(stop-start))+"ms")c = 0start1 = time.time()for i in range(N):c += a[i]*b[i]stop1 = time.time()print(f"c={c}")print("for loop: " + str(1000*(stop1-start1))+"ms")times1 = (stop1-start1)/(stop-start)print(f"times1={times1}")

• 結(jié)果

c=250071.8870070607
vectorized version: 6.549358367919922ms
c=250071.88700706122
for loop: 265.43641090393066ms
times1=40.52861303239898# 向量化居然比單獨(dú)的for循環(huán)快40倍

2. 向量函數(shù)與矩陣求導(dǎo)初印象

• 標(biāo)量函數(shù):輸出為標(biāo)量的函數(shù)
  $$f(x)=x^2?x\in R\to x^2\in R$$
  $$f(x)=x_1^2+x_2^2?\left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2\to x_1^2+x_2^2\in R$$
• 向量函數(shù):輸出為向量或矩陣的函數(shù)
  <1> 輸入標(biāo)量，輸出向量
  $$f(x)=\left[\begin{array}{c}{f}_1(x)=x\\ \\ f_2(x)=x^2\end{array}\right]?x\in R,\left[\begin{array}{c}x\\ \\ x^2\end{array}\right]\in R^2$$
  <2> 輸入標(biāo)量，輸出矩陣
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x& & f_{12}(x)=x^2\\ & & \\ f_{21}(x)=x^3& & f_{22}(x)=x^4\end{array}\right]?x\in R,\left[\begin{array}{ccc}x& & x^2\\ & & \\ x^3& & x^4\end{array}\right]\in R^{2\times 2}$$
  <3> 輸入向量，輸出矩陣
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x_1+x_2& & f_{12}(x)=x_1^2+x_2^2\\ & & \\ f_{21}(x)=x_1^3+x_2^3& & f_{22}(x)=x_1^4+x_2^4\end{array}\right]?\left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2,\left[\begin{array}{ccc}{x}_1+x_2& & x_1^2+x_2^2\\ & & \\ x_1^3+x_2^3& & x_1^4+x_2^4\end{array}\right]\in R^{2\times 2}$$
• 總結(jié)
  矩陣求導(dǎo)的本質(zhì)
  $$\frac{\mathrm{d}A}{\mathrm{d}B}=矩陣A中的每個(gè)元素對(duì)矩陣B中的每個(gè)元素求導(dǎo)$$

3. YX 拉伸術(shù)

3.1 f(x)為標(biāo)量，X為列向量

• 標(biāo)量不變，向量拉伸
• YX中，Y前面橫向拉，X后面縱向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x},Y=f(x)為標(biāo)量，X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]為列向量$$
  $$f(x)=f(x_1,x_2,....,x_n)為標(biāo)量$$
• 標(biāo)量$f(x)$不變，向量X 因?yàn)樵赮X拉伸術(shù)中在Y后面，所以向量X縱向拉伸，實(shí)際上就是將多元函數(shù)的偏導(dǎo)寫(xiě)在一個(gè)列向量中
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{?f(x)}{?x_1}\\ \\ \frac{?f(x)}{?x_2}\\ \\ ?\\ \\ \frac{?f(x)}{?x_n}\end{array}\right]$$

3.2 f(x)為列向量，X 為標(biāo)量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ?\\ \\ f_n(x)\end{array}\right];X為標(biāo)量$$

• 標(biāo)量不變，向量拉伸
• YX中，Y前面橫向拉，X后面縱向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{?f_1(x)}{?x}& & \frac{?f_2(x)}{?x}& & \dots & & \frac{?f_n(x)}{?x}\end{array}\right]$$

3.3 f(x)為列向量，X 為列向量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ?\\ \\ f_n(x)\end{array}\right];X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]為列向量$$

• 第一步先固定Y ，將 X 縱向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{?f(x)}{?x_1}\\ \\ \frac{?f(x)}{?x_2}\\ \\ ?\\ \\ \frac{?f(x)}{?x_n}\end{array}\right]$$
• 第二步，看每一個(gè)項(xiàng)$\frac{?f(x)}{?x_1}$,其中f(x)為列向量，$x_1$為標(biāo)量，那么可以看出要進(jìn)行 Y 橫向拉
  $$\frac{?f(x)}{?x_1}=\left[\begin{array}{ccccccc}\frac{?f_1(x)}{?x_1}& & \frac{?f_2(x)}{?x_1}& & \dots & & \frac{?f_n(x)}{?x_1}\end{array}\right]$$
• 第三步 ，將每項(xiàng)整合如下
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{?f_1(x)}{?x_1}& & \frac{?f_2(x)}{?x_1}& & \dots & & \frac{?f_n(x)}{?x_1}\\ & & & & & & \\ \frac{?f_1(x)}{?x_2}& & \frac{?f_2(x)}{?x_2}& & \dots & & \frac{?f_n(x)}{?x_2}\\ & & & & & & \\ ?& & ?& & \dots & & ?\\ & & & & & & \\ \frac{?f_1(x)}{?x_n}& & \frac{?f_2(x)}{?x_n}& & \dots & & \frac{?f_n(x)}{?x_n}\end{array}\right]$$

4. 常見(jiàn)矩陣求導(dǎo)公式

4.1 $Y=A^{T}X$

$$f(x)=A^{T}X;A=[a_1,a_2,\dots ,a_n{]}^T;X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$

• 由于$A^T=1\times n,X=n\times 1,那么f(x)為標(biāo)量，即表示數(shù)值$，
• 標(biāo)量不變，向量拉伸
• YX中，Y前面橫向拉，X后面縱向拉
  $$f(x)=\sum _{i=1}^N{a}_i{x}_i$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{?f(x)}{?x_1}\\ \\ \frac{?f(x)}{?x_2}\\ \\ ?\\ \\ \frac{?f(x)}{?x_n}\end{array}\right]$$
• 可以計(jì)算$\frac{?f(x)}{?x_i}$
  $$\frac{?f(x)}{?x_i}=a_i$$
• 可得如下：
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}{a}_1\\ \\ a_2\\ \\ ?\\ \\ a_n\end{array}\right]=A$$
• 結(jié)論：
  $$當(dāng)f(x)=A^{T}X$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=A$$

4.2 $Y=X^{T}AX$

$$f(x)=X^{T}AX;A=\left[\begin{array}{ccccccc}{a}_{11}& & a_{12}& & \dots & & a_{1n}\\ & & & & & & \\ a_{21}& & a_{22}& & \dots & & a_{2n}\\ & & & & & & \\ ?& & ?& & \dots & & ?\\ & & & & & & \\ a_{n1}& & a_{n2}& & \dots & & a_{nn}\end{array}\right];X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$
$$f(x)=\sum _{i=1}^N\sum _{j=1}^N{a}_{ij}{x}_i{x}_j$$

• 標(biāo)量不變，YX拉伸術(shù)，X縱向拉伸
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{?f(x)}{?x_1}\\ \\ \frac{?f(x)}{?x_2}\\ \\ ?\\ \\ \frac{?f(x)}{?x_n}\end{array}\right]$$
  $$\frac{?f(x)}{?x_i}=\left[\begin{array}{cccc}{a}_{i1}& a_{i2}& \dots & a_{in}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{1i}& a_{2i}& \dots & a_{ni}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ?& ?& \dots & ?\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ?& ?& \dots & ?\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ?\\ \\ x_n\end{array}\right]$$
• 已知$A,A^T$表示如下：
  $$A=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ?& ?& \dots & ?\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right];A^T=\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ?& ?& \dots & ?\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]$$
• 綜上所述如下：
  當(dāng) $f(x)=X^{T}AX$時(shí)
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=AX+A^{T}X=(A+A^T)X$$