Civs and Mil factories

So i'm still a newbie at HOI4 and am playing as Germany and wonder should i be building civ factories then build mil factories in preparation for war or should i just build mil factories then rely on the civ factories for nations I annex?