Is it possible that you are not seating the round all the way? Muzzleloaders can be fussy at times, but swabbing after each shot should help keep the accuracy. Hopefully somebody will come up with an idea to help you. Good luck
Keep life simple...if you hear hoofbeats, think horses, not zebras.