Generally, the designs of digital image processing algorithms and image gathering devices remain separate. However, experiments show that the image gathering process profoundly impacts the performance of digital image processing and the quality of the resulting images. We proposed an end-to-end information theory based system to assess linear shift-invariant edge detection algorithms, where the different parts, such as scene, image gathering, and processing, are assessed in an integrated manner using Shannon’s information theory. We evaluated the performance of the different algorithms as a function of the characteristics of the scene and the parameters, such as sampling, additive noise etc., that define the image gathering system. The edge detection algorithm is regarded as having high performance only if the information rate from the scene to the edge image approaches its maximum possible. This goal can be achieved only by jointly optimizing all processes. To validate our information theoretical conclusions, a series of experiments simulated the whole image acquisition process are conducted. After comparison and discussion between theoretic analysis and simulation analysis, we can draw a conclusion that the proposed information-theoretic assessment provides a new tool which allows us to compare different linear shift-invariant edge detectors in a common environment.