I am guilty of misjudging deer pretty often. I have killed some very nice looking three year olds. and Honestly, I do not know a way around it.
In the places I hunt(99% public land) it is VERY thick, I can see a rack coming through the palmettos and 9 times out of 10, the buck will only get in a hole where I can see his entire body for a second or two. I can either spend that time sizing him up or shooting at him.
I have made the mistake in both directions. they have got in a hole and I've taken the shot only to later regret it, AND conversely I have let them get by me and thought "oh dang that was a nice one I should've taken him. I really don't know what to do differently. because both mistakes feel equally bad