If whitetails were an exact science, they would be boring.
You have had a very rare interaction. First, you showed stealth. by staying low, and (un-human) they didn't know what to make of you. You did not appear to be a threat. once they understood you were a possible threat, they hightailed it.
WoodsWalker hunts in a gillie-suit, and slow stalks like what you describe. And he will likely tell you, that a man on his knees, or seated, is not a man shape for deer.
Yes, one arrow was dumb. I carry three, with spares back in the truck if needed. Yo should also carry a drag rope, to haul it back out, if you get one.
Yes, deer see differently than we do. First, they do not have the rods and cones in the eyes to see reds, though they see more into the ultraviolet than we do. hunter's orange looks more like tan or yellow to them, and even the best camo glows slightly blue-grey for them. (that is why hunter's soap, should not have ultra-violet enhancers in it, "To keep colors bright") you want to deaden the ultra-violet enhancers that are already in the inks/ dyes used to print the camo pattern.
Add to that, their eyes are spaced to the sides of their heads, this allows for vision nearly everywhere around them, except directly behind, and above. (for details on this, use the search box, we have threads on this.)
However, while they see much better than we do in low light, they do not see clearly as we do, they see movement very well, but not detail. Even if you COULD be completely scent free, movement will get you busted faster than anything else. (You didn't move, so you could have been a bush.) They charged you, and stomped and blowed , to see if they cold get you to move, and to identify you.
I would not say they are very intelligent, perhaps a little less than a dog, bt he/she has the tools to decide if you are a threat, and the speed to out run most threats, but will not run any further than it takes to feel safe. for one deer that might be miles, another, 10 yards. but with such a small brain, WILL IT listen to it's nose, or use it's confused eyes? they don't agree.
The shakiness is most likely "Buck Fever" nearly everyone gets it to some extent. The key is to focus on taking the perfect shot, and not thinking of the prize, just the shot. Stop thinking about getting the deer, just making the perfect shot like at the range. the rest will follow on it's own.
Lastly, you might consider a crossbow, if legal where you are. I hunt with one by choice. the range is a little longer, the impact is harder, and I blow two holes in each deer I hit, instead of one. (That makes for better tracking, and recovery) they bleed out very quickly, instead of bleeding internally only. It would be easier for you to hold for that perfect shot,
The only real difference between a good tracker and a bad tracker is observation. All the same data is present for both. The rest is understanding what you are seeing.