What you are probably seeing is a seasonal shift in the patterns of the deer rather than a response to your blind. The other possibility is that somebody is coming in behind you and hunting that blind when you are not there.
When you look at all the stuff I do in the woods, you would wonder why I have any deer coming near any of my stands. This year, I took down two ladder stands, put up a third and then took the first two, repainted them and located them to new spots. With the ladders that were already in place, I went out and sprayed rust inhibitor on all the new surface rust-- all this before Oct 1. In almost every case, I was driving a truck to within 50 yards of the stand, schlepping the 84 lb stand in or out, and using absolutely no scent control whatsoever. I also had to re-shingle one of my ground blinds due to Hurricane Ike.
Normally I would have had this all done by Labor Day, but we had a death in the family, and the funeral arrangements pushed back all my hunting preparations.
The results? The funeral went off about as well as could be anticipated. We had to do all the arranging long distance, because Granny was up here in Cincinnati, and the funeral was down in. . .
. . . oh, you meant the stands. Sorry.
So far we've had KY's Yute Season, and KY's Early Muzzleloader Season, and my two sons both bagged doe. I had probably a dozen or so chances to score, but I am holding off for rifle season. Of the 3 dozen sightings of deer we've had since Oct 1, most were within bow range.
One other datum to throw in here: On the way back from scouting the other night, I walked past the entrance to the new stand sight I set up for my son. The property's resident monster buck (we call him Chandelier) broke from cover and ran off straight down the track I'd run with the truck and directly towards my son's stand. He's also been sighted directly under one of the newly painted stands.
My point in bringing this all up is that you can do a lot stuff in the woods and not scare off deer. Erecting a ground blind and then letting it lie fallow until season probably means nothing to them.