Today, recognizing offline handwritten character images is still hard challenge. This is because there are the obstacles, ‘noise’ produced in scanning process. Noise makes handwritten character distorted, murky, and blurred. As a result, it become hard to read and recognize these images for human. In this study, we tried to get rid of various noises using CNN architecture named “U-Net” to analyze 607,200 sample images consisting of 3,036 Japanese characters. Finally, our results indicate that the “U-Net” has efficient ability to remove noise and enhance the parts of strokes even through there are a huge variety of handwritten styles which includes various noises.