I haven't ever worn hearing protection while deer hunting. I have for dove, but not for anything else. If all you're going to do is shoot 1 or 2 shots in a day, it is not going to screw up your hearing. It's the repeated hammering of something like duck or dove hunting that can really hurt.
Having said that, you bring up an interesting point: what about out of a wooden box blind. All I can say is ZOWIE! Back in 2005 my #2 son took a shot out of our blind at a doe with his 30-30. He was shooting out the front. I was spotting with binos out the side window. The rifle was angled about 45 degrees to the front of the blind and it put the muzzle about in line with my ears. The concussion was terrific! I got knocked off my stool. This was THROUGH 1/2" plywood. I was deaf in my left ear for two days.
On the other hand, if you're shooting perpendicular to the blind-- straight out, the blast from even a 30-06 can be quite uneventful if it's just you, or if you're the shooter. In fact, I can never remember a time where my hearing was affected by a shot while deer hunting. It's the guy next to you that will have the larger blast-- if it's just you alone, there's a cone of attenuation directly behind the hunter.
I have two sons that hunt, so I end up hunting next to someone most days of my season. In a box blind, a popup or an open buddy stand, whoever is taking the shots is supposed to give the other guy a chance to hold his ears. We usually count it off " 3,2, 1,. . ." in a low whisper.