Many digital signal processing and image coding systems implement the linear predictor with rounding. Usually, people will obtain the linear predictors by solving the Yule-Walker equations or doing something equivalent. The predictors obtained in this way will not necessarily be the true minimum mean square error predictor considering the effect of rounding. In this paper, we address the issue of finding the true optimum mean square error rounded linear predictor. Experiment results show that when the prediction results are rounded, this true MMSE linear predictor could outperform the conventional one without considering the effect of rounding very significantly for data of low prediction errors.
Fu Yu Tsai,
"Minimum mean square error linear predictor with rounding", Proc. SPIE 2501, Visual Communications and Image Processing '95, (21 April 1995); doi: 10.1117/12.206750; https://doi.org/10.1117/12.206750