Good comments.

But one thing... a doe passes her genes to 5-10 offspring in a lifetime.
(One to two offspring for 5 reproductive years)

A buck may pass his to many many more. Say 30 offspring.
(3 does per season at two offspring per doe for 5 years)

So that's a pretty significant difference. But that also assumes you catch this "cull" at a young age.

All that said, you do have to give credit to the OP quote concerning home territory etc. If you let a true cull camp out on your property, there's the potential he's running off better bucks with better genetics. He's also consuming some value of your properties best resources. The significance of these two things on your property can vary from person to person, but I think it's still a valid point.