Overlay is a one of the most critical design specifications in semiconductor device manufacturing. Any state-of- the-art production facility has overlay metrology in place to monitor overlay performance during manufacturing and to use the measurements for overlay control. Especially since the introduction of multi-patterning, with its tight overlay requirements and increased number of process steps, there has been an increased need for additional metrology. Overlay metrology brings cost-added value to semiconductor device manufacturing and it should be reduced to a minimum to keep costs at acceptable levels, which can be a challenge in the multi-patterning era. Replacing some real overlay measurements with predicted values, referred to as virtual overlay metrology, could be a viable solution to address this challenge. In this work, we develop virtual overlay metrology and aim at predicting the overlay for a series of implant layers. To this end, we apply machine learning algorithms, and neural networks in particular, to build a complex non-linear model directly from data. Our model takes a set of features that are designed based on the physical concepts of overlay and outputs the overlay map of a target layer. The features include overlay of another implant layer of the same wafer, exposure tool fingerprints, scanner logging, and process data. We evaluate our model using production data and we show the prediction performance for the raw overlay, as well as for the correctable and non-correctable overlay errors.