Bucks react to scent, movement and human activity negatively. If all that happened was a loud 'BOOM" with dirt flying close by, then you should be fine. Deer don't equate a shot from a rifle with humans, unless they see the human.