What is the Derivative of (x^T P x)?

In the realm of matrix calculus, understanding the gradient of the quadratic form (x^T P x) is crucial for many optimization algorithms and machine learning models. This article explores the detailed derivation of the gradient step-by-step, providing a solid foundation for both beginners and advanced practitioners in the field.

Introduction to the Problem

The quadratic form (x^T P x) is a common expression in linear algebra, where (x) is an (n)-dimensional vector and (P) is an (n times n) matrix. The goal is to find the gradient of this expression with respect to the components of the vector (x).

Understanding the Derivative

The derivative is approached through the concept of partial derivatives. The gradient is a vector composed of the partial derivatives of the quadratic form with respect to each component of (x).

Step 1: Expressing the Quadratic Form

Let's start by expressing the quadratic form in a more manageable way:

$$x^T P x left[begin{matrix} x_1 cdots x_n end{matrix}right] left[begin{matrix} p_{11} cdots p_{1n} vdots ddots p_{n1} cdots p_{nn} end{matrix}right] left[begin{matrix} x_1 vdots x_n end{matrix}right]$$

Step 2: Calculating the Partial Derivative

For a specific component (x_s), the partial derivative of (x^T P x) with respect to (x_s) can be calculated as follows:

$$frac{partial}{partial x_s} left[x^T P xright] frac{partial}{partial x_s} left[sum_{i1}^{n} sum_{j1}^{n} x_i p_{ij} x_jright]$$

This process involves differentiating the expression with respect to (x_s). Let’s break it down step by step:

Substep 2.1: Individual Partial Derivative Calculation

The expression inside the sum simplifies to:

$$frac{partial}{partial x_s} left[x_i p_{ij} x_jright] left(frac{partial}{partial x_s} [x_i]right) p_{ij} x_j x_i p_{ij} left(frac{partial}{partial x_s} [x_j]right)$$

Since the only non-zero terms occur when (i s) or (j s), we get:

$$frac{partial}{partial x_s} left[x_i p_{ij} x_jright] p_{sj} x_j x_i p_{is}$$

Substep 2.2: Aggregating the Results

Summing up the terms for all (i) and (j), the expression simplifies to:

$$sum_{i1}^{n} sum_{j1}^{n} frac{partial}{partial x_s} [x_i p_{ij} x_j] sum_{j1}^{n} p_{sj} x_j sum_{i1}^{n} x_i p_{is}$$

Step 3: Simplifying the Expression

This can be rewritten as:

$$frac{partial}{partial x_s} left[x^T P xright] left(sum_{j1}^{n} p_{sj} x_jright) left(sum_{i1}^{n} x_i p_{is}right)$$

Notice that the first term is the (s)-th component of the vector (P x) and the second term is the (s)-th entry of the vector (x^T P).

Step 4: Forming the Gradient Vector

The gradient is the vector of all partial derivatives. Therefore, the gradient of (x^T P x) is:

$$ abla_{x} (x^T P x) left[begin{matrix} sum_{j1}^{n} p_{1j} x_j vdots sum_{j1}^{n} p_{nj} x_j end{matrix}right] left[begin{matrix} sum_{i1}^{n} x_i p_{i1} vdots sum_{i1}^{n} x_i p_{in} end{matrix}right] (P x) (x^T P)$$

This can be further simplified to:

$$ abla_{x} (x^T P x) P x x^T P $$

Key Points to Remember

The gradient of (x^T P x) with respect to (x) is (P x x^T P). Understanding the transpose property is crucial: (x^T P (P^T x)^T). Matrix multiplication rules, particularly (PX cdot (x^T P)^T (P^T x) cdot P^T x), play a significant role. The gradient operator ( abla_x) indicates that the derivative is taken with respect to (x), treating other variables as constants.

Conclusion

Understanding the gradient of quadratic forms is fundamental for effective optimization and machine learning tasks. By breaking down the problem step by step, we have derived the gradient, which serves as a cornerstone for further advanced topics in these fields.