In the field of semiconductor manufacturing, Scanning Electron Microscope (SEM) is employed for critical dimension (CD) measurements, overlay measurements, and defect inspections to ensure the quality and reliability of semiconductor devices. Nevertheless, SEM images inherently carry a significant level of noise, leading to inaccurate metrology and false defect inspections. Therefore, it is crucial to develop denoising techniques. One widely used method is frame averaging, which reduces cumulative noise by averaging multiple scans. While increasing scan times enhances SEM image quality, it comes with drawbacks such as surface charging, pattern shrinkage, and reduced throughput. Deep learning (DL) techniques, including supervised and unsupervised approaches, have shown remarkable progress in the field of SEM image denoising. However, supervised methods are notably affected by phenomena such as pattern shrinkage and surface charging, which occur during the capture of reference images. On the other hand, unsupervised methods are typically more effective with lower noise levels. In this paper, we introduced a flexible DL method for denoising SEM images that operates without the requirement for paired data. To demonstrate its effectiveness, we analyzed and evaluated its performance in two metrology tasks. Experimental results validated the efficacy of our method in reducing noise, demonstrating its applicability to both ADI and AEI.