Metrology measurement and defect inspection steps in routes are more pervasive than many people realize and the number continues to grow. Digging deeper, it turns out that E-beam metrology and defect inspection tools typically occupy the most overall steps and therefore are extremely critical tools for semiconductor development and manufacturing. The Critical Dimension Scanning Electron Microscope (CDSEM) is an E-beam tool responsible for image-based structural metrology measurements while the E-beam review (EBR) and E-beam inspection (EBI) tools are responsible for defect inspection. The CDSEM faces significant future challenges in a world where device architectures are changing (as with the FinFET device), increasing the need for more structural measurement parameters such as sidewall angle, height and undercut, than the CDSEM can deliver. These applications are now migrating to scattering-based tools, also referred to as model- or computational-based structural metrology techniques. We explore the history of the CDSEM, the key fundamental limits (primarily resolution) of the CDSEM preventing it from capturing these applications and assess if an image-based structural metrology tool is still needed given the niche that scattering tools are filling. The answer is yes. Next we discuss whether the CDSEM will eventually meet those needs or if other alternative solutions are needed. Lastly, we discuss an industry survey on which image-based technique holds the most promise to solve these challenges. Regarding defect inspection, resolution is also a concern for the applications that need to detect extremely small defects which current optical-based Brightfield tools cannot address. While resolution is a primary concern in structural metrology, current EBI and EBR tools possess the resolution needed for defect inspection and have extendibility through at least the next advanced technology node. We also explore the niche E-beam is filling in defect inspection, evaluate the idea that a better synergistic solution exists today between the CDSEM and the EBI tools and propose a future E-beam landscape where E-beam tool variety in future fabs is significantly reduced to enhance productivity.