f(x)=a((sinx)^2+1-(cosx)^2)-√3asin2x+b+1=a(1-cos2x)-√3sin2x+b+1=a-2asin(2x+π/6)+b+1,x∈[0,π/2],∴1/2≤sin(x+π/6)≤1,∴b+1≤f(x)≤b-a+1,∴b+1=-5,b-a+1=0,得a=-5,b=-6