Most state of the art video-based gait recognition algorithms start from binary silhouettes. These silhouettes, defined as foreground regions, are usually detected by background subtraction methods, which results in holes or missed parts due to similarity of foreground and background color, and boundary errors due to video compression artifacts. Errors in low-level representation make it hard to understand the effect of certain conditions, such as surface and time, on gait recognition. In this paper, we present a part-level, manual silhouette database consisting of 71 subjects, over one gait cycle, with differences in surface, shoe-type, carrying condition, and time. We have a total of about 11,000 manual silhouette frames. The purpose of this manual silhouette database is twofold. First, this is a resource that we make available at http://www.GaitChallenge.org for use by the gait community to test and design better silhouette detection algorithms. These silhouettes can also be used to learn gait dynamics. Second, using the baseline gait recognition algorithm, which was specified along with the HumanID Gait Challenge problem, we show that performance from manual silhouettes is similar and only sometimes better than that from automated silhouettes detected by statistical background subtraction. Low performances when comparing sequences with differences in walking surfaces and time-variation are not fully explained by silhouette quality. We also study the recognition power in each body part and show that recognition based on just the legs is equal to that from the whole silhouette. There is also significant recognition power in the head and torso shape.