I'd look more at the terrain you'll be hunting rather than the size of the deer.

Hard to do better than the .308 for most hunting conditions, especially if you load your own.