Personally, I'll take in the flesh deer sightings over deer sign if I am flipping a coin on a spot. Rubs and scrapes are exciting, but I have much better luck hunting what I know are travel routes or food areas, rather than setting up on a specific sign.
I'd study the topo map close to where you saw those rubs and look for a terrain feature that may determine his movement, and set up on that.