把句子的顺序调整过来就是
He had so much trouble doing the experiment that he gave up.
句子的结构以及运用的短语就很明显了.
have trouble in doing sth.做某事有困难或很麻烦
其中in可以省略.