i agree here, it might be real easy to push him. sounds like he has a lot of food and likely tons of cover. he probably doesnt bed too far away, so you might be closer to his bedding area than you think. i have a lot of deer bedding right in the corn for parts of the day, if he's using corn he might even do that. one option i would try is if you have corn available to hunt near, check the edge of the field for tracks. if you find an area where some big fat tracks come and go from the field, try hunting that edge of the corn. you might catch him leaving or entering the corn, and that could happen any time of day. im getting trail camera pics of deer entering and leaving the corn fields at all times of the day, and surprisingly, mostly during day light hours. no huge bucks, but some small ones. mid day activity all the same.
i would definitely try to locate a bedding area because that is good information to know, just try not to push him out of it. its frustrating having deer move in your area the middle of the night. sometimes all you can do is wait until the rut and hope he passes through while youre there. he will pass through, just is a matter of whether you happen to be hunting that day or not. or, you could dig a giant hole in the orchard and cover it with sticks and c'mere deer.