f(x1,x2,x3)
= 5x1^2+5x2^2+3x3^2-2x1x2+6x1x3-6x2x3
= 5(x1-1/5x2+3/5x3)^2+24/5x2^2+6/5x3^2-24/5x2x3
= 5(x1-1/5x2+3/5x3)^2+24/5(x2-1/2x3)^2
= 5y1^2+24/5y2^2
y1=x1-1/5x2+3/5x3
y2=x2-1/2x3
y3=x3
即
x1=y1+1/5y2-1/5y3
x2=y2+1/2y3
x3=y3
C=
1 1/5 -1/5
0 1 1/2
0 0 1
X=CY.