(1)
BP=3PA
OP-OB = 3(OA-OP)
4OP = 3OA+OB
OP = (3/4)OA + (1/4)OB
=> x = 3/4 and y =1/4
(2)
OP.AB
=((3/4)OA + (1/4)OB).(OB-OA)
= -(3/4)|OA|^2 +(1/4)|OB|^2+(1/2)|OA||OB|cos60°
= -12+1+2
=-9