Dose accumulation across External Beam Radiotherapy (EBRT) and Brachytherapy (BT) treatment fractions in cervical cancer is extremely challenging due to structural dissimilarities and large inter-fractional anatomic deformations between the EBRT and BT images. The brachytherapy applicator and the bladder balloon, present only in the BT images, introduce missing structural correspondences for the underlying registration problem. Complex anatomical deformations caused by the applicator and the balloon, different rectum and bladder filling and tumor shrinkage compound the registration difficulties. Conventional free-form registration methods struggle to handle such topological differences. In this paper, we propose a registration pipeline that first transforms the original images to their distance maps based on segmentations of critical organs and then performs non-linear registration of the distance maps. The resulting dense deformation field is then used to transform the original anatomical image. The registration accuracy is evaluated on 27 image pairs from stage 2B-4A cervical cancer patients. The algorithm reaches a Hausdorff distance of close to 0:5 mm for the uterus, 2:2 mm for the bladder and 1:7 mm for the rectum when applied to (EBRT,BT) pairs, taken at time points more than three months apart. This generalized model-free framework can be used to register any combination of EBRT and BT images as opposed to methods in the literature that are tuned for either only (BT,BT) pair, or only (EBRT,EBRT) pair or only (BT,EBRT) pair. A unified framework for 3D dose accumulation across multiple EBRT and BT fractions is proposed to facilitate adaptive personalized radiation therapy.