I understand what you mean. Here's some o my experiences with the topic.
The first nice buck I ever killed was on a green field from a shooting house. I was hunting as a guest on a club in marshall county. It was raining that evening so we hunted from the shooting houses. It was a nice deer and I was very proud of it, but looking back, it took absolutley no hunting skills to take it. I just had to kill it. Anyone could have done the same.
On the other hand, I killed a 3.5 yr old 8pt last year on skyline WMA. Wasn't a huge deer, but I had to work for it. I put some hard miles on the ol rocky boots one friday scouting an area deep in the mountains that I had never hunted. I found a good spot that I thought would produce a buck. Didn't have a clue how big of a buck he was, but I knew a decent one was in the area due to the sign he was leaving. I got up early enough to beat the crowd on saturday morning. With 30lbs of treestand and gear on my back I had to find my way in the dark to the tree I had picked out the evening before. About 8:00, sure enough, he popped up out of nowhere from under a STEEP holler below to check his scrapeline on the flat I was on. That was his last mistake.
That was a deer I was proud of. Not because of the size of the rack, but because I felt like I had hunted it. To me, hunting is a more about a personal challenge than just shooting deer.