个人认为,两个都可以,因为在我看过的很多英语文章,两者都可以通用,be patient of 对…容忍,如,He is patient of cold and hunger 他能够容忍饥饿 或 Tom has been quite patient of the noise outside the window for two days
而be patient with 是能够对sth有耐心,如Teachers are always patient with us.老师总是对我们很有耐心
一个偏向于容忍（相对有些贬义）,一个偏向于耐心.