During routine use of flatbed scanners for grading lentils, unexpected short-term variability in scanner performance was noticed. This variability was detected as scan-to-scan differences, or repeatability. This study aims at developing an objective measure of scanner repeatability to facilitate the selection of scanners by establishing performance criteria for a scanner-based vision system. Four measures of scanner repeatability were compared, using seven scanners for two types of target objects. Scanner selection for grain grading applications is discussed. For grain grading, scanner repeatability is best characterized by color variations as measured by Eq. (4), which performed better than the other three measures of repeatability. The selection of the method of determining scanner repeatability, and the tolerance thereof, are dependent upon the constraints of the imaging targets.