Fatigue design is crucial for reliable and lightweight steel structures. Especially, welded joints reduce the fatigue strength of those structures. This is attributed to the notched weld surface geometry and other phenomena like residual stresses or increased hardness value which are related to the welding process. It is well known that the weld surface geometry varies along the weld seam and due to that also the stress concentration. Measuring techniques like 3D laser scanning offer the possibility to capture these precisely and holistically along the weld seam. However, most commonly the nominal stress approach is used for fatigue strength assessment, which only considers the weld seam surface geometry globally. High safety margins are therefore required to ensure reliability, which on the other hand means that the full potential of lightweight construction cannot be fully exploited. Numerical methods based on the FEM can be used for a much more accurate fatigue strength assessment, but are computationally time-consuming and therefore practically not applicable for a holistic assessment of the weld seam. One possibility to overcome the lack of complexity of the nominal stress approach and the high computing times of FEM, is by using data driven methods based on Machine Learning (ML) algorithms. The success of data driven methods depends on the quality and the representation of the data. This study employs ML to predict fatigue strength based on XGBoost models pretrained on flux cord arc welded (FCAW) and submerged arc welded (SAW) butt welds. Extending the dataset to include Laser Hybrid (LH) welds and specimens with weld imperfections, the research explores ML model transferability across different welding methods and imperfection types defined by ISO standards. Results highlight challenges in predicting fatigue life accurately when weld surface geometry varies or significant imperfections are present, emphasizing the need for robust anomaly detection in 3D laser scanning data. The study points out the impact of locally occurring weld defects and imperfections on fatigue life predictions and highlights the limitations of current data driven fatigue assessment methods. Based on this, it is discussed how the representation of the data is influenced by the use of different measures of location for the statistical distribution of weld surface geometry parameters.