To achieve high perceptual quality of compressed images, many objective image quality metrics for compression artifacts evaluation and reduction have been developed based on characterization of local image features. However, it is the end user who is judging the image quality in various applications, so the validation of how well these metrics predict human perception is important and necessary. In this paper, we present a preliminary psychophysics experiment method to capture human perception of local ringing artifacts in JPEG images with different severity levels. Observers are asked to annotate the compressed image where they perceive artifacts along the edges, directly on the screen using an interactive tablet display. They are asked to catalog the severity of artifacts into one of the three levels: Strong, Medium, and Light. We process the hand-marked data into a ringing visibility edge map showing a ringing severity mean opinion score (MOS) at every edge pixel. The perceptual information captured in this experiment, enables us to study the correlation between human perception and local image features, which is an important step towards the goal of developing a non-reference (NR) objective metric to predict the visibility of JPEG ringing artifacts in alignment with the assessments of human observers.