You might want to review the linear algebra text book of freshman's course in most college.
First of all, let's make some vectors based on your given points:
b=B-A
c=C-A
d=D-A
, In other words, we simply let A be the original point.Vectors b, c, and d are connecting from A to B, to C, and to D, respectively.We are going to answer your questions based on these vectors.
To check if A, B, and C are not in a straight line, we simply need to check if vector b and c are independent.We use the rank of matrix with verctor b and c to determine:
r = rank([b c])
Here we assume b and c are column vectors.(If b and c are row vectors, r = rank([b;c]) in Matlab)Now, If r=2, vector b and c are independent, in other words, A, B and C are not in one straight line, they can determine a plane.
We assume b and c are independent now.We use the following steps to obtain a basis:
(1) take b as the first vector: v1 = b
(2) get the normal vector of the plane determined by b and c:
v3 = cross(b,c)
(3) get the third vector which is orthogonal to both v1 and v3:
v2 = cross(v3, v1)
*Note: v2 must be on the plane defined by b and c, or equavilently by A, B and C
(4) normalize v1, v2 and v3, we get:
e1 = v1 / norm(v1)
e2 = v2 / norm(v2)
e3 = v3 / norm(v3)
Then, e1, e2 and e3 form a basis of whole 3D spance.
To get the projection of D onto plane ABC, you can get the inner product of d and each vector of the basis :
p1 = dot(d,e1)
p2 = dot(d,e2)
p3 = dot(d,e3)
Then:
projectionOfDOntoABC = p1*e1+p2*e2+p3*e3
Finally, e1 and e2 form a basis of the plane ABC. You can use (p1,p2) as the coordinates with respect to the basis e1,e2 to refer a point on the plane.