请注意比较：
I had a really hard time with science.
==I had a really hard time (in) learning / studying science.能用doing ,为什么to do不可以呢？你好。have a hard time (in) doing sth已经成了一个习惯 / 固定用法了。 (in) doing sth 做 have a hard time 的伴随动作。它不做have a hard time 的定语。另外，在你的语境中，in == while :==I had a really hard time WHILE( I WAS )learning / studyingscience.