联立y=ax+b和双曲线3x²-y²=1,得：
3x²-(ax+b)²=1,即(3-a²)x²-2abx-b²-1=0
设A、B坐标分别为(x1,y1)(x2,y2)
则x1+x2=-2ab/(a²-3),x1x2=(b²+1)/(a²-3)
则y1y2=(ax1+b)(ax2+b)=a²x1x2+ab(x1+x2)+b²=a²(b²+1)/(a²-3)-2a²b²/(a²-3)+b²=(a²-3b²)/(a²-3)
∵O在圆上,AB是直径,∴∠AOB=90°
∴OA·OB=0,即x1x2+y1y2=0
∴(b²+1)/(a²-3)+(a²-3b²)/(a²-3)=0,解得：2b²-a²=1
即P的轨迹方程为2y²-x²=1