go with the 243 and shoot 55 grain bullets. Should be on par with 223 recoil wise. Then you can work him up to 80 or 100 grain bullets later.
I don't think shooting Varmint bullets at deer is a good idea unless the little guy is good at head shots. That ain't what 55 grainers are made for
