I suggest just having a wander around and looking at the colleges. The bits to the south of the High Street are probably prettiest/least busy, and if the weather's nice you could take a turn round Christ Church Meadows too - you could make a circular walk of that and take in Broad Street with the Bod and the Sheldonian too.

For coffee and cake I recommend the Vaults cafe, which is in the vaults of the University Church (entrance in Radcliffe Square) - lovely setting and gorgeous cakes!