两个都可以,
Encouraged by the teacher,he detemined to study harder. 这其中determine 是在作动词
Encouraged by the teacher,he was detemined to study.
这句也可以,因为determine 在这里是作形容词,即be determined to do sth ,下定决心要干某事.
而实际使用的时候,be determined to do sth要用得多一些.