My deskmate doesn't like reading, so he rarely goes to the library.
My deskmate doesn't like reading, so he seldom goes to the library.
都可以的
rarely和seldom都是副词,表示“很少地,不经常地”