Alright, I know I am late to the party, but I just had to chime in. It has been a while since I had any of my wildlife classes at Auburn,but from what I can remember, the average home range of a mature whitetail buck is upwards of 2000 acres. The range of a doe is a few hundred acres. I know that all of this has been said before, but all the deer on the 324 acres in question are just passing through whether they are on the 320 or the 4. While it is possible to keep deer on a small parcel for a good deal of time, they will eventually wander on to someone else's land an possibly get killed.
The only thing you can do as a land manager on a smaller piece of property is to provide the deer with everything they could want or need and give them reason to stay on your property as much as possible. The key is to draw the deer into your property with something that is better than what everybody else has. It sounds like the 4 acre guy is better at this and the 320 acre guy is jealous. However if the 4 acre guy starts killing everything he sees, the deer will wise up and stay off the smaller piece of property.
I don't see where the 4 acre guy can do a whole lot to really and truly screw up the herd. Deer respond to hunting pressure and will go to where they feel safe during the day or become nocturnal. If the 320 acre guy is doing a good job with his habitat improvement and managing his hunting pressure, he is still going to see and kill deer regardless of what the 4 acre guy does.
I speak about all of this from personal experience, as I am the little guy where I hunt in Pickens County. I hunt 240 acres surrounded by 750 acres of Corps. of Engineers land, and a hunting club that totals about 1500 acres. Since I hunt and work the land primarily alone and I cannot afford a ton of improvements, my management plan revolves around managing hunting pressure. I know that the corps. land will be hunted heavily and once the shooting starts the deer like to spend a lot of their time on my place. The hunting club hammers does at the beginning of the season, and I like to spread my harvest out over the course of the season. Since the club puts a lot of pressure on the deer and I don't, I consistently see deer throughout the season. Its all about how you hunt.
Sorry I ran long,but I couldn't turn down the opportunity to get in on this.