I had intended to have dinner with you.
口语化的表达有：
Having dinner with you was my original plan.