Kip Adams with QDMA lends some insight on the holes in antlers, and it's not genetic:

So what actually causes them? It is unlikely they are caused by a “boring fly” or other insect, but rather the result of an injury to the growing antler. I asked some of the best minds in deer research if they were aware of any studies identifying the cause of holes in antlers. Neither Dr. Bubenik nor John Ozoga nor Drs. Grant Woods or Karl Miller were aware of a single study. I have observed deer at captive research facilities with blood on the surface of an injured velvet antler. This blood often attracted flies and may be the cause of speculation about the role of flies in this scenario. In the few cases I observed, the bucks injured their antlers on a fence and the flies arrived afterward. The above researchers suggested the likelihood of the holes being injury induced.

https://www.realtree.com/all/articles/antler-trivia