I have a difficult time getting along with him.
have a difficult time :很难的(时间)
get along with sb.:和某人相处