5ab^2-{2a^2b-[3ab^2-(4ab^2-2a^2b]}
=5ab^2-2a^2b+[3ab^2-(4ab^2-2a^2b]
=5ab^2-2a^2b+3ab^2-(4ab^2-2a^2b)
=8ab^2-2a^2b-4ab^2+2a^2b
=4ab^2