AT THE OFFICE would mean to call someone while they are at work; TO THE OFFICE would be to send for someone to come to your office. But you can't have both TO and AT in the same phrase.