In practical analog/digital hybrid optical transmission, carriers are modulated by video signals or digital data, and the amplitude of a multiplexed signal composed of these modulation signals is more compressed than that of the carriers. This causes a decrease in the frequency of clipping of the multiplexed signal at the laser threshold. Consequently, the BER of the M-QAM signal in a practical hybrid transmission is lower than that of the experimental results for same optical modulation index (OMI). However, it is difficult to prepare many practical modulation signals for experiments in a laboratory. Therefore, there is demand for a BER analysis method for a multiplexed signal that includes modulation signals needs to evaluate the BER and determine optimum OMI in a practical hybrid transmission. In this paper, we describe such a BER analysis method that can effectively estimate the BER in a practical hybrid transmission by using representative profiles of the modulation signals. In practical systems, a black pictures gives the largest averaged amplitude for the AM-VSB signal, and the most severe conditions for clipping occurrences. However, in such systems, the BER was greatly improved over the BER of a multiplexed signal of carriers. Furthermore, BER degradations due to clipping can be neglected for the AM-VSB signals in setting a practical OMI range.