Y= (cos²ax³)' = 2cos ax³ *(cos ax³)' = 2cos ax³ *(-sin ax³)*(ax³)' = 2cos ax³ *(-sin ax³)*3ax²
=-6 a x^2 Cos[a x^3] Sin[a x^3]