Efficient compression is pertinent for the convenient storage, transmission, and processing of modern high-resolution hyperspectral images (HSI). We propose a high-performance HSI compression method using library-based spectral unmixing and tensor decomposition. Unlike the existing approaches, our proposed work incorporates unmixing in the compression framework and achieves significantly higher compression performance with negligible loss. The proposed library-based unmixing method includes an index for accurate endmember number estimation, followed by exact library pruning and a sparsity regularized formulation with norm-smoothing to compute the abundance maps. As the spectral library is available at the reconstruction (decoder) side also; compressing the abundance maps is as good as compressing the original HSI data. Since the abundance constraints used for the unmixing indicate the correlation of the abundance maps, compressing all abundance maps seems to cause redundant computation. A metric using the image smoothness and information measures is used here to identify the abundance map hardest to compress and the remaining part is left uncompressed. Subsequently, the work compresses the abundance map tensor using orthogonal PARAFAC decomposition with optimal rank determination. The orthogonalization process ensures that the factors span independent subspaces and reduces redundancy, whereas the rank selection prevents noisy or insignificant components. Extensive experiments are carried out to demonstrate that the unmixing workflow leads to negligible loss due to accurate endmember number estimation, exact library pruning, and accurate physically meaningful sparse inversion. Comparative assessments of compression efficacy suggest that the proposed work corresponds to better compression performance and higher classification accuracy.