They should have gone earlier为虚拟语气中的主句,表示与过去事实相反的情况,从句谓语动词形式Had+过去分词（过去完成时）,主句谓语动词形式Should/would/could/might+have+过去分词（现在完成将来时）
eg.If I had got there earlier,I should/could have met her.如果我早到那儿,我就会见到她.（事实：去晚了）
If he had taken my advice,he would not have made such a mistake.如果他听我的劝告的话,就不会犯这样的错误了.（事实：没有听我的话）
与过去相反,所以反问也用过去时.