(x+2y)^2-(x+y)(3x-y)-5y^2
=(x^2+4xy+4y^2)-(3x^2+2xy-y^2)-5y^2
=-2x^2+2xy