To answer your question, no, personally, I wouldn't kill it. I have had bucks in my yard during open season and they have always gotten a pass.
I can certainly understand your desire to kill him if this buck is bigger than anything you've killed. It is usually not easy to kill a buck of this caliber.
I have had regrets about not shooting deer that either were never seen by me again or killed by someone else. I have also had regrets about killing certain deer. I am now at a point, where I kill what makes me happy and try not to worry too much about what all the goobers out there think.
No matter what anyone else says, whether they would kill the deer or not kill it, ultimately it is a decision that you will live with. It sounds like you have made your decision, so go forward without any regrets.
