As speech based operation becomes a main hand-free interaction solution between human and mobile devices (i.e., smartphones,
Google Glass), privacy preserving speaker verification receives much attention nowadays. Privacy preserving
speaker verification can be achieved through many different ways, such as fuzzy vault and encryption. Encryption based
solutions are promising as cryptography is based on solid mathematic foundations and the security properties can be easily
analyzed in a well established framework. Most current asymmetric encryption schemes work on finite algebraic structures,
such as finite group and finite fields. However, the encryption scheme for privacy preserving speaker verification
must handle floating point numbers. This gap must be filled to make the overall scheme practical. In this paper, we propose
a number system that meets the requirements of both speaker verification and the encryption scheme used in the process.
It also supports addition homomorphic property of Pailliers encryption, which is crucial for privacy preserving speaker
verification. As asymmetric encryption is expensive, we propose a method of packing several numbers into one plain-text
and the computation overhead is greatly reduced. To evaluate the performance of this method, we implement Pailliers
encryption scheme over proposed number system and the packing technique. Our findings show that the proposed solution
can fulfill the gap between speaker verification and encryption scheme very well, and the packing technique improves
the overall performance. Furthermore, our solution is a building block of encryption based privacy preserving speaker
verification, the privacy protection and accuracy rate are not affected.