Mr Smith came back late by (when all the guest had left).
by引导的时间状语,这个时间状语是when 引导的从句
Smith先生回来得很晚,那时所有的客人都离开了.