5abc-2a²b-[3abc-3(4ab²+a²b)]
=5abc-2a²b-(3abc-12ab²-3a²b)
=5abc-2a²b-3abc+12ab²+3a²b
=2abc+a²b+12ab²