I also suspect genetics. If the deer are otherwise healthy and have normal width and mass for your area then I cannot see how nutrition or stress can be blamed.
Can a deer have forks one year and develop brow tines later? I would say yes. From what I have seen, deer begin to grow mass, sticker points and increase in brow tine length after 3 years.
The biggest thing that I would NOT do, from a QDM standpoint, is try to cull these deer from your herd as yearlings or 2.5's. My advice, if it is a concern for you, is to give them 3 years.