had to 是have to (不得不)的过去形式,不是情态动词,所以提取助动词did
Mr Black had to go back,didn't he?
have to 是固定用法,有不得不、必须的意思,把它看成一般的行为动词就好了,有过去形式