I've never beaten myself up over a deer I've shot and I've killed some small racked bucks, broken rack bucks, a button, a shed, etc. Hell the only thing I've never actually killed is a true spike but I still collect all the spike racks I find in the woods. All my bucks are like 200" deer to me. They all are prizes and they're all unique for reasons beyond just being big.
I have, however, had some bit of remorse with does I've shot. I've never wanted to take the shot back, but emotionally I struggle a little bit with killing does. This has created many either-sex days where the safety was off, the crosshairs were placed, and I just wouldn't apply the necessary trigger pressure. I was willing to go home empty handed and happy. I've never felt this way with the smallest of bucks. I will say I feel better once it's in the skinning shed because then it's all work and I don't have the same feelings when a partner shoots a doe. Weird, I know.
I will say I've only regretted bucks I didn't shoot. Quality opportunities and whitetail bucks are not terms that go together often. Normally at the end of the year I look back on one buck or one situation and say "man, I should've done the deed." But as long as this is my regret after the season I feel I'm doing alright. I do have an advantage over some in that I'm not limited to one buck a year. Out of three buck tags I will fill one or two a year but rarely all three.
One other thing about this holding out year after year. I believe success breeds success. The longer you wait for that one opportunity the less experience and confidence you will have in getting the job done when it's time. There's something to be said about becoming proficient in the act of taking a deer from the decisiveness, the brisk pace, the shot placement, the tracking/recovery, and on and on. I'd rather fill my tag yearly with a nice buck than wait 3 years for that one chance at a Joe Nailer that could likely result in a botched go anyway.
My point is to just be happy with the deer you take and if you can't get over the remorse you can wait it out next year, but don't be remorseful while eating tag soup as well. I guess only then you'll know which remorse is worse. Also, it helps to stay realistic and grounded so don't let your standards be twice the size of the biggest buck in your herd. It's pretty hard to hunt a deer that doesn't exist. Personally, this is how I kind of define a shooter on my land. I take the biggest deer I know exists which I determine one of three ways. 1-trail cam pic 2-personal sighting 3-best surviving deer known from last year with a realistic extra year of growth. Then from that particular deer anything that looks 3/4 his size is a possible shooter. Anything his size or larger is a no-doubter. I find this keeps hunting very fun and I have nice resident bucks worth hunting year to year.