The caliber would definitely help, but there are a few things that I can help with right off the bat.
First, I would recommend bumping up your powder to about 100 grains. Most guys use about that much as a starting point, and then adjust a few grains from there. Also, I would change bullets. If your bullet is falling out of the barrel, you're losing a lot of accuracy there. I would recommend trying the Barnes T-MZ sabot, because it is built for a tighter fit that some guns require.
Let us know more about your gun, and we can all give you better information on how to move forward.