The conventional printing technologies use the standard four ink (yellow, magenta, cyan, and black) process. In some special cases, an unconventional ink set printing process can be selected. The unconventional inks set printing process uses a particular set of inks, selected according to the image to be reproduced. The inks are deposited on the substrate material in a certain order, through individual ink masks. Designing the unconventional printing process requires us to find the ink color set, named primary color palette, the sequence of ink mask printing, and the individual ink masks (color separation). In addition to the primary color palette, the secondary color palette is defined as the set of all the color combinations resulting by overlapping the ink masks after the complete printing process. A single primary color palette may conduct after printing process to a number of different secondary color palettes. This paper provides a color separation method which converts the input color components to the ink mask values that determine the amount of ink deposited on the substrate. The separation process depends essentially on the color calibration process that determines the colors of the secondary palette. The color calibration is accomplished using the printed samples. The calibration process conducts to a hierarchical structure of color combinations of the secondary color palette. The simulation of the printed colors is described, based on the generalized Neugebauer model for an arbitrary number of inks and an arbitrary set of inks. The color hierarchy established during the calibration process is used to derive the coefficients of the Neugebauer model.