What are the laws in your area? Do they define how close someone can be to your boundary? If not, I would say that as long as they are on the public land they are not in violation. It might be annoying to you but that is about the worst thing I can say about it. Some might try to call it unethical but that would be a misuse of that word. Why do they do it? Well, don't we all try to find the sanctuaries and hunt the trails in and out? Your 50 acres is a sanctuary of lower hunting preasue, either real or imagined.
Here is the part you will love. Assume a guy is set up right on your border. If the biggest buck you have seen steps accross that line and that guys shoots him he has done nothing wrong. Many states have laws that say a hunter has a right to claim his kill if it runs back on your property after being shot. That would just be a hoot wouldn't it.
You might want a buffer zone around your land but you really have no right to that. Your control ends at the property line and you pretty much just have to live with whatever happens on your side.
No then, if someone DOES come on to your land it should be met with swift and maximum reaction. Call the sherrif and report the trespassing. If you find stand on your land, guess what. You just got yourself a new stand. It would be in the back of my truck in a heartbeat if it was me. I would have zero sympathy for anyone that was trespassing unless I truely believed that did it by mistake.
You know you could always just dump a bunch of corn in the middle of your land. We all know that piles of corn are irresistable to deer and once you put it down they will never leave your property. (Heh, heh, sorry could not resist with that one but you gotta admit it was pretty clever how I worked that topic into this thread