Analysis and Comparison of Color Features for Content Based Image Retrieval

: Creation of a content-based image retrieval system implies solving a number of difficult problems, including analysis of low-level image features and construction of feature vectors, multidimensional indexing, design of user interface, and data visualization. Color is one of the important features used inCBIRsystems. The methods of characterizing color fallinto two major categories: Histograms and Statistical. Anexperimental comparison of a number of different color features for content-based image retrieval presented in these paper. The primary goal is to determine which color featureis most efficient inrepresenting the spatial distribution of images. In this paper,we analyze and evaluate both Statistical and Structuralcolor features. For the experiments, publicly availableimage databases are used. Analysis and comparison of individual color features arepresented.

Such retrieval is possible only if all images of the collectionhave annotations describing their content. In this case, the image retrieval problem reduces to the classical text search. Discussion of the description based image retrieval methods, as well as algorithms of automated annotation, is beyond the scope of this survey.
Three main directions of CBIR-related studies are as follows [1].

Feature extraction
Exploration of image content representation and matching techniques. In the framework of this direction, new feature vectors and methods to extract them are suggested, and new metrics are introduced on the corresponding vector spaces.

TRADITIONAL ARCHITECTURE OF
Development of multidimensionalindexing algorithms to facilitate fastsearch in large-scale collections of high dimensiondataRetrieval system design. An important requirementto any system is user-friendliness. For CBIR systems, this requirement is of special importance because of great complexity of such systems. How to simultaneously show to the user many images selected by thesystem as a response to a query? How to provide the user with the possibility of the retrieval quality evaluation in order to further take this evaluation into account when refining the retrieval results? How to organize the dialog of the user with the system? Finding answers to these questions is the task of the researchers designing retrieval systems.
Feature extraction algorithms affect quality of theretrieval. Multidimensional indexing makes search faster. Humanengineered interface of a retrieval system helps the user to correctly formulate queries and refine them on subsequent stages of communication with the system and facilitates work with the retrieval system on the whole.
In this survey, only the first direction of studies is discussed, and well-known algorithms for feature extraction are considered. Afeature vector (or simply features) is a set of numeric parameters describing animage. The majority of such vectors represent one image feature, such as color, texture, or shape of the object. Feature vectors built by the same algorithm form a space of feature vectors (for short, a feature space). Having defined metrics on such a space, one can compare images by calculating distances between the corresponding vectors.
Many researchers distinguish several levels ofimage content and of the corresponding features [2].
Text annotations describe image semantics and are classified as high-level features. Visual features, such as color and texture, are low-level features. Some researchers consider shape of objects on the image, which can be obtained by analyzing texture, as a low level feature too. Others consider shape as a middle level feature. Color, texture, and shape of objects are classified as general characteristics. They are used in the majority of CBIR systems and are convenient for retrieval from heterogeneous image collections. In the case of a certain subject domain, it is often possible to identify domain-specific features, such as, for example, shape of an eye, nose, or mouth and face oval in face recognition or fingerprint curves. Such features can be considered as an application dependent particular case of the above-listed general features.
For every feature, there are many methods for its representation with feature vector. Each method has its own w w w . i j c t o n l i n e . c o m advantages and disadvantages, and development of new algorithms and improvement of the existing ones is currently underway.
Further in this survey, we discuss the well-known algorithms for feature extraction for color.

CBIR SYSTEMS
Traditional architecture of CBIR systems ( Fig. 1)has much in common with that of classical retrievalsystems. It also includes two basic modules: indexingand retrieval module. The former is responsiblefor data processing and constructing indices, which considerably speed up the search. The latter takes care of the retrieval itself by the user request. One of the key components of CBIR systems is thatfor image parameterization, i.e., calculationof feature vectors. The input of this component is supplied with a matrix of values of image pixels. The image is analyzed, and the corresponding feature vectors are calculated and sent to the indexing module. It is by these feature vectors of the image that the index is constructed. The component that calculates the features takes part in the search if the retrieval is based on a sample image loaded by the user (which is not available in the collection sought at the beginning of the search process).
It is required to extract features for the query image using the same algorithms that were used for feature extraction while indexing image collection. Then the calculated feature vectors of the query image are used to perform the retrieval.The traditional approach to the content-basedretrieval is to search independently by different image features. Each of them is represented by a point in the corresponding feature space. Some systems use several feature spaces to represent the same feature in order to improve retrieval accuracy. In this case, search in each feature space is also performed independently, which is followed by applying data fusion methods to merge the retrieved sets (intermediate outputs) into one common output. Anoutputhere is meant to be a ranked set of retrieved objects.

General schema of Content Based image Retrieval
The block diagram consists of following main blocksdigitizer, feature extraction, image database, feature database, and matching and multidimensional indexing. Function of each block is as follows. Digitizer: To add new images in image database or query images which are acquired from CCD Camera, X-ray imaging system, microdensitometer's, image dissectors, vision cameras etc. are needed to be digitized, so that computer can process those images.
Image Database: The Comparison between Query image and images from image database can be done directly pixel by pixel which will give precise match but on the other hand, recognizing objects entirely at query time will limit the retrieval speed of the system, due to the high expense of such computing. Generally this crude method of comparison is not used, but image database, which contains raw images, is required for visual display purpose.
Feature Extraction: To avoid above problem of pixel-by-pixel comparison next abstraction level for representing images is the feature level. Every image is characterized by a set of features such as Texture, Color, Shape and others. Extract these features at the time of injecting new image in image database. Then summarize these features in a reduced set of k indexes and store it in Image feature database. The query image is processed in the same way as images in the database.
Matching is carried out on the feature database.
Image matching and Multidimensional Indexing: Extracted features of query image are compared with features, which are stored in image feature database. To achieve fast retrieval speed and make the retrieval system truly scalable to large size image collections an effective multidimensional indexing is indispensable part of the whole system. The system selects the Nimages having the greatest overall similarities to the query image.

GENERAL CLASSIFICATION OF APPROACHES
Feature extraction algorithms and similarity measures used for image comparison underlie any CBIR system. All contentbased retrieval methods can be classified into classes depending on the features they use: search by color. Each class, in turn, is divided into subclasses by the type of the algorithm used for constructing the feature vector. Some researchers classify spatial features of images into a separate class [4,5]. Spatial features are those reflecting spatial layout of homogeneous image regions in terms of one or another feature: for example, regionof the same color or a particular object. In other words, these are features of one of the classes (color) with additional informationon spatial layout. Therefore, in the author'sopinion, separation of spatial features in a special class seems to be not quite correct. In what follows, we consider common algorithms for color

COLOR
Color feature is the most significant one in searching collections of color images of arbitrary subject matter. Color plays very important role in the human visual perception mechanism. Besides, image color is easy-to analyze, and it is invariant with respect to the size of the image and orientation of objects on it. This explains why the color feature is most frequently used in image retrieval, as well as the fact that the number of fundamentally different approaches is not too great.
All methods for representing color feature of an image can be classified into two groups: color histograms and statistical methods of color representation (Fig. 4).Quality of color feature vectors greatly dependsonthecolor space selection. Reliable features are those comparison of which allows one to derive a correct conclusion regarding similarity of the corresponding images.

Color Spaces
A color space (also referred to as a color modelorcolor system) [6,7] is a specification of a coordinatesystem and a subspace within this system where each color is represented by asingle point. Thus, each color in a color space has itscolor coordinates.The most frequently used color spaces are as follows. RGB, red, green, and blue, (color monitors and cameras); CMY, cyan, magenta, and yellow, and CMYK, cyan, magenta, yellow, and black, (color printers); HSV, hue, saturation, and value;Lab (CIE L*a*b*, CIELAB), lightness, a and b aretwo color dimensions, from green to red and from blue to yellow. The Lab space relies on the international standard of color measurement developed by the International Commission on Illumination CIE (Commission International de l' Éclair age). The HSV space is similar to spaces HSI, HSL, and HSB. The majority of spaces for representing full-color images are threedimensional.
A traditional space for representing digital images is RGB. However, this space is not perceptually uniform: the fact that the distances of points A and B to point O in the RGB space are equal does nor mean that, for a human being, the colors corresponding to points A and B look equally similar (or not similar) to the color corresponding to point O. The Lab space and HSV family better correspond to human color perception. These spaces have additional advantage in that they decouplecolor and gray-scale information, which is convenientin image processing. Therefore, the majority of theirsearchers construct color feature vectors in one ofthese spaces. The HSV space is used more frequentlybecause the RGB HSV transformation is simplerfrom the computational standpoint compared to the RGB Lab transformation.

Color Histograms
The simplest and most frequently used way to represent color is color histograms [8][9][10]. For each pointof the considered color space, the number of image pixelsof a given color is calculated. For a space of Ncolors,the color histogram is anNdimensional vector ( h1,h2,--hn) where h i is the portion of pixels of colori on the image. Fig. 4.Four different images for which color histograms areidentical [8].
For the distance function for such histograms, Swain and Ballard [10] proposed to use metrics defined by normL 1 [10]: WhereQis a query image,Iis the image from the collection by which the desired image is retrieved, h j Q and h j I and are elements of the feature vectors of the corresponding images.
Such representation of information on color is simpleand natural; however, it has one considerable disadvantage: the distance between two images that have similar but not identical colors is large. In addition, such histograms are very sparse and, thus, sensitive to noise.
A more frequently used distance function on thespace of color histograms is coordinated with the L 2 norm [11]: If the color space is selected appropriately, the useof this metrics reduces the number of the Type II errors (-false negatives‖) inherent in metrics (1). However, this metrics is also noise-sensitive. A more detailed discussion of disadvantages of using histograms together with metrics (1) and (2) can be found in [9]. Strickerand Orengo used cumulative color histograms [9]. Such a representation of color is less sensitiveto noise and also reduces the number of the Type IIerrors if adjacent elements of histograms correspond to similar colors. In addition to metrics (1) and (2), they used metrics associated with L ∞ norm [11]: Experiments showed that the use of cumulative histograms together with the L-metrics yields insignificantgain compared to the ordinary color histograms [9].
Smith and Chang [12,13] notice importance of quantization of the original color space in construction of color histograms. The color space is quantized into color clusters (similar colors fall into one cluster), and each bin of the histogram w w w . i j c t o n l i n e . c o m corresponds to one color cluster (the number of pixels of the corresponding color cluster is counted). Hardly distinguishable colors should belong to the same cluster, whereas colors from different clusters should easily be distinguishable.
Since the RGB space is not perceptually uniform, a quantization in accordance with human color perception for the HSV and CIE LAB spaces is a simpler task. The HSV space is used more frequently in view of simplicity of the transformation from RGB to HSV andback. Construction of color histograms by quantized color space makes it possible to reduce dimensionality of the feature vector, which speeds up the retrieval. Another approach to take into account the similarity of different colors is presented in [14]. In this work, the following metrics defined on the space of color vectors (histograms) is proposed: Here, an element a ij of matrixAis the coefficient ofsimilarity between colorsIandj.
The color histogram itself does not store information on spatial layout of colors on the image. For example, color histograms for images presented in Fig. 4 are identical.A solution to this problem was suggested in [15].After constructing a color histogram where only main colors of an image are taken into account, for everyNonzero element of the histogram, the coordinates ofthe center of mass of the corresponding color region is calculated. This information is used to measure the similarity between the images together with the number ofpixels belonging to this color region. This solution makes it possible, in a sense, to take into account spatial layout of colors, but it possesses one significant disadvantage.
If the image contains several compound components of the same color, this fact will not be reflected in the feature vector of the image. Instead, a common center of mass for all components will be calculated. Hence, for the images presented in Fig. 4, the featurevector for image (a) is different from others; the feature vectors for (c) and (b) and (c) and (d) differ insignificantly from one another; and the vectors corresponding to (b) and (d) are identical.
Another simple way to take spatial layout of colors into account is to partition images into fixed layout regions and calculate histograms for each region separately. However, under this approach, it is not easy to determine a universal region size. Small layout regions are appropriate for images with many details, whereas, for images consisting of few onecolor regions, such a representation is redundant. The growth of the number of the layout regions increases memory required for storing them and slows down the retrieval, since dimension of the feature vectors grows. The discussion of more complicated techniques of image segmentation is beyond the scope of this survey.

Statistical Model of Color Representation
An alternative to the histograms is a statistical model of color representation suggested by StrickerandOrengo [9]. This model is based on probability distribution of individual color channels. The feature vector consists of dominant distribution features for each channel. These are the first three central moments: average, variance, and skewness. Thus, the description of a color feature requires only nine numbers. For the distance function, a weighted sum of differences of the corresponding color moments for the two images is used. Based on test results, the authors report that this model of color representation significantly outperforms all kinds of color histograms.
A modification of this model was suggested byStricker and co-authors in [16]: distributions of separate color channels are considered as a part of a three dimensional distribution rather than as independent distributions. For the feature vector, average values for each color channel and covariance matrix of the channel distributions are used.
However, like classical histograms, this model does not take into account spatial layout of colors. To overcome this, the approach described in the previous section can be applied, which consists in partitioning the image into layout regions of fixed size (or more complicated image segmentation) and calculating features of color distribution for each of them. It is suggested in [16] to divide images into -fuzzy regions.‖ The following five regions are introduced: central ellipsoidal region and four surrounding regions (Fig.  5). All regions are defined by membership functions presented in [9].
According to these membership functions, pixelslocated strictly in the center of the image completely belong to the central region and thus affect the feature vector of the central region only. The closer the pixel to the region border, the lesser its influence to the region's feature vector. Pixels located on a border separating two regions affect the feature vectors of both regions. Experiments show that such an approach makes it possibleto improve retrieval results in the case of morecomplicated queries, when it is required to take into account spatial layout of objects on the image.

DCD Color Feature Representation
In general, color is one of the most dominant and distinguishable low-level visual features in describing image. Many CBIR systems employ color to retrieve images, such as QBIC system and Visual SEEK. In theory, it will lead to minimum error by extracting color feature for retrieval using real color image directly, but the problem is that the computation cost and storage required will expand rapidly. So it goes against practical application. In fact, for a given color image, the number of actual colors only occupies a small proportion of the total number of colors in the whole color space, and further observation shows that some dominant colors majority of pixels. Consequently, it won't influence the understanding of image content though reducing thequality of image if we use these dominant colors to represent image.
In the MPEG-7 Final Committee Draft, several color descriptors have been approved including number of histogram descriptors and a dominant color descriptor (DCD) w w w . i j c t o n l i n e . c o m [4,6]. DCD contains two main components: representative colors and the percentage of each color. DCD can provide an effective, compact, and intuitive salient color representation, and describe the color distribution in an image or a region of interesting. But, for the DCD in MPEG-7, the representative colors depend on the color distribution, and the greater part of representative colors will be located in the higher color distribution range with smaller color distance. It is may be not consistent with human perception because human eyes cannot exactly distinguish the colors with close distance. Moreover, DCD similarity matching does not fit human perception very well, and it will cause incorrect ranks for images with similar color distribution. We will adopt a new and efficient dominant color extraction scheme to address the above problems [7,8].
According to numerous experiments, the selection of color space is not a critical issue for DCD extraction. Therefore, for simplicity and without loss of generality, the RGB color space is used. Firstly the image is uniformly divided into 8 coarse partitions, as shown in Fig. 2. If there are several colors located on the same partitioned block, they are assumed to be similar. After the above coarse partition, the centroid of each partition is selected as its quantized color. Let X=(XR, XG,XB) represent color components of a pixel with color components Red, Green, and Blue, and Ci be the quantized color for partition i.
The procedure to extract dominant color of an image is as follows: According to numerous experiments, the selection of color space is not a critical issue for DCD extraction. Therefore, for simplicity and without loss of generality, the RGB color space is used. Firstly, the RGB color space is uniformly divided into 8 coarse partitions, as shown in Fig. 6. If there are several colors located on the same partitioned block, they are assumed to be similar. After the above coarse partition, the centroid of each partition (-color Bin‖ in MPEG-7) is selected as its quantized color. Let X=(XR, XG,XB) represent color components of a pixel with color components Red, Green, and Blue, and Ci be the quantized color for partition i.
In this way, Dominant color of an image will be obtained.

Comparison of Color Features
Experiments carried out by the authors of the statistical model of color representation showed great advantage of this approach over the color histograms [9]. Numbers of positions of images relevant to the query in the output were used to evaluate performance of the method. In the paper, a table of numbers of such positions for one and the same query for different retrieval methods (based on histograms and on statistical models) was presented.
However, our experiments on comparison of methods based on histograms supplemented by information on spatial layout of colors (ColorHist) [15] and on the statistical model with -fuzzy regions‖ (ColorMoments) [16] did not demonstrate significant advantage of thestatistical method over the former.
In our experiments,a subset consisting of 285 images from the Corel Photo Set collection was employed. Average recall and precision of the methods were estimated. Table 1 showsresults of the experiments, which were carried out based on the weak relevance judgments. The majority of known CBIR systems, such as VisualSEEk [17], QBIC [18], Mars [19,20], and Netra [21], use color histograms in retrieval by color. DCD color methods similarity matching does not fit human perception very well, and it will cause incorrect ranks for images with similar color distributionThe above allows us to assume that, under appropriate selection of the color space, its quantization, and metrics on the feature space, DCD color can be quite effective in retrieval by color.
The performance of a retrieval system can be measured in terms of its recall (or sensitivity) and precision (or specificity). Recall measures the ability of the system to retrieve all models that are relevant, while precision measures the ability of the system to retrieve only models that are relevant. They are defined as

Conclusions
An experimental comparison of a number of different color features for content-based image retrieval was carried out. The Histograms model and statistical model were considered for retrieval. The retrieval efficiency of the color features was investigated by means of relevance. According to the results obtained it is difficult to claim that any individual feature is superior to others. The performance depends on the spatial distribution of images. The test results indicated that DCD performs well compared to other features when images are homogeneous. In most of the image categories, w w w . i j c t o n l i n e . c o m Autocorrelation feature also shows similar performance. It is also noted that the Histograms features are more effective than the statistical model features. In case of combination features, combinations recorded better retrieval rate compared to the performances of those individual color features.