High accuracy and low-cost indoor localization has received much attention for the past few years. Visible light (VL) localization, which reuses indoor LED luminaire infrastructure as artificial visual landmarks, has emerged as a promising technology. Using only lights as localization beacons, it can provide highly accurate indoor localization without reconstructing the whole environmental scenario. We propose a localization system that utilizes luminaire distribution patterns on the ceiling as visual landmarks without the costly luminaire circuits as in previous methods. For pattern detection, a unique descriptor for each luminaire cluster is derived through image processing followed by perspective rectification. Then, each pattern’s ID in a lookup table is retrieved with earth mover’s distance under rotation transformation. Using all individual luminaires in each pattern, our system achieves accurate and robust localization under noise and interference. Furthermore, our system can automatically generate pattern dictionaries of hundreds to thousands of unique IDs resulting in great scalability. Evaluations are conducted through real-world experiments. Compared with previous systems, the proposed method achieves superior performance in terms of detection range, localization, and orientation accuracy. Specifically, our system achieves high average three-dimensional localization accuracy of 2.47 cm and orientation accuracy of 0.87 deg under different heights for the best dictionary setting. Finally, an implementation on a mobile phone demonstrates the potential to run on resource-constrained platforms in real time.