The only problem I have with a heavy baselayer is sweating on the way to the stand. Then an hour later im freezing. I havent quite figured out a system to stop this. If I make it 1/4 mile to my tree without sweating, I always sweat getting up the tree.
Only have the base layer on your upperbody when you leave the truck. Carry everything else you wear on top. Leave a few minutes early and take a rest/cool down stop or two on the way in.