f(f(f(x))) = x/(1+3x^2)^0.5
f(f(x))=f(x/(1+x^2)^0.5)
=x/(1+x^2)^0.5/(1+(x/(1+x^2)^0.5)^2)^0.5
=x/(1+2x^2)^0.5