my feelings are the dominant one would stay in the area. However I wouldnt just say the younger deer got the area for your land. The other 2 may be around you just dont know it. Also I would be pleased to have deer like that in my area. Keep in mind that with the rut approaching, they will be back.
"If I pull the hammer and shoot this young buck, he's dead. But if I pass on him, the next hunter might not shoot so straight."