I always hunted with it off, but I always had a good way to keep the quiver handy for a quick second shot. I've hunted with a Quickie Quiver for all my days, and for the most part, I've had a second Quickie bracket attached to a screw that went into the side of the tree or a nearby limb. For the past few years, I've attached to the shooting rail of my buddy stands. The arrows are right there at my feet, just like when I'm practicing.
My reasoning was this: with all that extra mass flopping around on the off side of the bow, I figured by taking the quiver off the bow I had a more predictable flight. The other reason I had was that all that extra hardware on the side had a tendency to catch branches and stuff . Your mileage may vary of course, but I found it gave me a tighter group. It certainly made the bow lighter to hold.