the one指代the library,后面跟着的是定语从句,the one在定语从句中作地点状语,即you always borrow books from the library,所以要用where来引导定语从句,故答案是the one where.

如果要选the one that,那么the one在定语从句中必须作主语或宾语,如：
This room is the one that they want to decorate.
This room is the one that needs to be cleaned.