Are you letting it cool completely between these shots?
If you are, and this is what's happening off a cool barrel, there are a bunch of avenues.
1) A change in ammo. I've switched between 150 and 165 grain bullets in a 30-06 and had problems like this go away
2) Lapping the barrel with one of those kits like this:
I haven't done it, but I know folks that have.
3) Polishing the barrel the Hillbilly way: Use FLITZ on a tight-fitting patch and drag it up and down the barrel. I had great results improving a Mosin Nagant and an old Muzzleloader this way-- two patches X50 times each, and just watch out for the crown (!). Then clean it spotlessly to get the FLITZ out.
4) Hoppes Moly Spray on a spotlessly cleaned barrel (see #3 above)