You're lucky he didn't have thumbs or he would have shot you with your own bow.
I had a similar experience on a public land 6 point. I was hunting at Skeegee and had JUST sat down in my stand when I see a nice 18" wide mature 6 point walking straight towards me about 50 yards away. I start trying to slowly pull up my bow and he saw it moving, watched me pull it up to the stand, turned and walked away.
I know he would have blasted out of there if had moved