In metallic alloys, inherent defects can significantly reduce resistance to high cycle fatigue, making them more vulnerable to failure. Several approaches exist to account for the effects of these defects, including methods based on Linear Elastic Fracture Mechanics (LEFM) or the use of fatigue criteria. Although these approaches are generally robust, accurately determining the defect population remains a challenge. A common non-destructive method for this is X-ray micro-computed tomography (µCT), which provides detailed information about defect distributions throughout the entire volume of the material. However, the use of µCT can be time-consuming, particularly for large volumes, and controlling thick samples is challenging due to X-ray attenuation. The present study investigates whether generative neural networks, such as diffusion models or GANs, can be used to build representative defect populations. This approach is applied to materials manufactured either by casting or by additive manufacturing: cast AlSi7Mg aluminium alloy and Laser Powder Bed Fused 316L steel. Generative neural networks are trained on µCT data from both materials, and the generated defect populations are analyzed for representativity. Finally, we discuss the relevance of the generated populations and suggest perspectives to enhance the robustness of this approach. The aim here is to build synthetic microstructure for fatigue predictions purpose.