f(x)=sin²x+2√3sin(x+π/4)sin[π/2-(x-π/4)]-cos²x-√3
=sin²x+2√3sin(x+π/4)sin(3π/4-x)-cos²x-√3
=sin²x+2√3sin(x+π/4)sin[π-(3π/4-x)]-cos²x-√3
=sin²x+2√3sin²(x+π/4)-cos²x-√3
=2√3[1-cos(2x+π/2)]/2-(cos²x-sin²x)-√3
=√3sin2x-cos2x
=2sin(2x-π/6)