Are there any relationships between PLS and Krylov subspace methods?

----A question posted by Jinbo Bi

A drug mining problem --- docking problem

Leads Receptors

DATA:

	y1 y2 ... yl	x1 x2 x3 x4 x5 … xn
Lead1 Lead2 Lead3 * * * LeadN	1.2 1.3 … 1.4 2.1 2.2 … 2.4 3.1 3.2 … 3.4 * * * 3.1 3.2 3.4	0.23 0.14 0.4 0.31 0.2 … 0.176 0.33 0.12 0.4 0.32 0.5 … 0.177 0.23 0.14 0.4 0.31 0.2 … 0.176 * * * 0.33 0.12 0.4 0.32 0.5 … 0.177
	Y	X

*

*

Ordinary Least Squares

Principal Component Analysis (PCA)

This is a singular value decomposition

Each p (loading) vector is actually a projection operator, and the t (score) vector is the coordinate of each point along these projection directions p's.

What are those p vectors? What are those q vectors? And what are in the diagonal of the diagonal matrix S?

So p’s are the eigenvectors of matrix , and q’s are eigenvectors of matrix , diagonal elements of S are sqrt of eigenvalues.

Principal Component Regression

Cut off some eigenvectors corresponding to smaller eigenvalues, i.e., use fewer rank-one matrices to approximate original data X.

Partial Least Squares

Algorithm:

1. do PCA for matrix X, find score T_x,

2. do PCA for matrix Y, find score T_y,

3. fit regression model between T_x and T_y, like

A real-life algorithm: (iteratively finding t’s and p’s for X and Y)

For each h=1,…,c, where A₀=X'Y, M₀=X'X, C₀=I, and given,

1. compute q_h, the dominant eigenvector of A_h^TA_h

2. w_h=C_hA_hq_h, w_h=w_h/||w_h||, and store w_h into W as a column

3. p_h=M_hw_h, c_h=w_h^TM_hw_h, p_h=p_h/c_h, and store p_h into P as a column

4. q_h=A_h^Tw_h/c_h, and store q_h into Q as a column

5. A_h+1=A_h - c_hp_hq_h^T and B_h+1=M_h - c_hp_hp_h^T

6. C_h+1=C_h - w_hp_h^T

A theorem is:

Consider one dependent y,

where

Krylov Subspace Methods

It is an equation solver, i.e., an iterative algorithm to solve an equation system like

A kth Krylov subspace generated by a vector c is like

The method is to find solutions of the following problem within the kth Krylov subspace generated by y at the kth iteration

Gram-Schmidt scheme to construct orthonormal basis for

Krylov method is shown can converge in super-linear rate under certain assumptions.

For a real-life algorithm, please check the paper

“The idea behind Krylov methods” by Ilse C.F. Ipsen and Carl D. Meyer

The real-life algorithms for PLS and Krylov method are pretty much similar to each other, so the questions are

1) if PLS is really a kind of Krylov method if you consider fitting regression is somehow kind of solving equation system?

2) if PLS is not Krylov method, can it be tailored to be able to apply any of Krylov methods, which be able to make solving PLS faster?