1.to have been...是不能用于独立主格结构的.
要使用不定式的话,一般只能是：to be...
2.但是如果该句改成：That to be my first visit to it,I didn't know where I should go first.
则此句错误.
3.因为不定式用在独立主格结构中,表示该动作尚未发生.
如：That to be my first visit to it,I decided to buy a map first.
4.但是该题根据I didn't know where I should go first可以看出,my first visit所表示的动作已经发生,所以用不定式是错误的.