Sounds like something that happened to me. Was out after a deer for my antlerless tag since I already filled my buck tag with a decent 8pt 3 weeks earlier with a bow. A 6 pt shows up and then the biggest buck of my life shows up offering me shot after shot. In my well-brushed in blind I was invisible and had to yell just to get them to leave! Then I nicked the side of the blind when a doe showed up so my arrow went in a tree. To add insult to injury, one more spike shows up and walks to within 8 ft of me (in full view) and I scare it away. Then, when I am wrapping up, the spike shows up again! That is how it goes sometimes!
You did the right thing - if someone is gracious enough to allow you to hunt, you shoot what they specify. If someone allows me to hunt on their land, I will always ask what they want me to shoot (or not shoot).