5(3a^2b-ab^2)-(ab^2+3a^2b)
=15a^2b-5ab^2-ab^2-3a^2b
=(15-3)a^2b-(5+1)ab^2
=12a^2b-6ab^2.