|A-λE| = (5-λ)(1+λ)^2.
所以A的特征值为 5,-1,-1
(A-5E)X = 0 的基础解系为:a1 = (1,1,1)'
(A+E)X = 0 的基础解系为:a2 = (1,-1,0)',a3 = (1,0,-1)'
将 a2,a3 正交化得 b2 = (1,-1,0)',b3 = (1/2,1/2,-1)'
单位化得
c1 = (1/√3,1/√3,1/√3)',
c2 = (1/√2,-1/√2,0)',
c3 = (1/√6,1/√6,-2/√6)'
令矩阵P = (c1,c2,c3),则P为正交矩阵,且 P^-1AP = diag(5,-1,-1).