Heritage sites often comprise buildings and monuments constructed from stone materials. When these heritage sites require restoration, selecting replacement materials that match originals is crucial for preserving cultural authenticity. Nevertheless, sampling is generally prohibited at heritage sites, preventing destructive analyses for quantitatively comparing originals and replacements. To address this issue, we established a non-destructive image analysis protocol to derive proportions of constituent minerals for quantitatively comparing the appearance of stone materials. Among the construction stones, granite was selected for developing this color differentiation algorithm because it is a ubiquitous construction material and it comprises minerals with distinctive colors. Mineral proportions are, therefore, prime parameters for describing the images of granites. By appearance, granites were sorted into the gray-white group and the brown-pink group. The corresponding relationship between the colors in the image and the main minerals is confirmed by a Raman spectrometer. In image analysis, the brown and pink pixels were distinct from the gray, white (plagioclase), and black (biotite) ones for higher R (red) values in the red, green, and blue (RGB) color model. Moreover, the dark colored pixels had a standard deviation from the mean of RGB values [δ (RGB)] extending to higher values. Granite pixels were, therefore, first separated using a 2R/(G+B) versus δ (RGB) plot. The brown and pink pixels formed a high R array with a positive 2R/(G+B) – δ (RGB) slope, whereas the gray, white, and black pixels defined a low R array showing an inverse 2R/(G+B)–δ(RGB) correlation. The combination of δ (RGB) and (R+G+B) further separated light and dark colored pixels in a 2R/(G+B) – δ (RGB) array. At last, grayscale was applied to resolve the white and gray pixels in a low R array after isolating the black pixels. The numbers of pixels for each color were then converted to mineral proportions. More precise mineral proportions were acquired for comparison through composition 2 analyses that required destructive procedures. The comparator values were derived by solving a matrix equation that described a bulk composition as the sum of the products of the constituent mineral compositions and their corresponding proportions. The mineral proportions from image analysis and composition analyses were within 10%. Hence, our new protocol was validated as a method for selecting proper replacement materials for restoring granite-based cultural heritage sites. Finally, the image analysis procedures were applied to the restoration of two monument in southern Taiwan.