The woman to whom you were talking is his mother.
等于
The woman whom you were talking TO is his mother.
和谁谈话是talk TO sb.
不能说talk sb.
所以要有to