Dunn index

There are many ways to define the size or diameter of a cluster. It could be the distance between the farthest two points inside a cluster, it could be the mean of all the pairwise distances between data points inside the cluster, or it could as well be the distance of each data point from the cluster centroid. Each of these formulations are mathematically shown below:

Let C_i be a cluster of vectors. Let x and y be any two n dimensional feature vectors assigned to the same cluster C_i.

${\displaystyle \Delta _{i}={\underset {x,y\in C_{i}}{\text{max}}}d(x,y)}$ , which calculates the maximum distance (the version proposed by Dunn).

${\displaystyle \Delta _{i}={\dfrac {2}{|C_{i}|(|C_{i}|-1)}}{\underset {x,y\in C_{i},x\neq y}{\sum }}d(x,y)}$ , which calculates the mean distance between all pairs.

${\displaystyle \Delta _{i}={\dfrac {{\underset {x\in C_{i}}{\sum }}d(x,\mu )}{|C_{i}|}},\mu ={\dfrac {{\underset {x\in C_{i}}{\sum }}x}{|C_{i}|}}}$ , calculates distance of all the points from the mean.

This can also be said about the intercluster distance, where similar formulations can be made, using either the closest two data points (used by Dunn), one in each cluster, or the farthest two, or the distance between the centroids and so on. The definition of the index includes any such formulation, and the family of indices so formed are called Dunn-like Indices. Let ${\displaystyle \delta (C_{i},C_{j})}$ be this intercluster distance metric, between clusters C_i and C_j.

With the above notation, if there are m clusters, then the Dunn Index for the set is defined as:

${\displaystyle {\mathit {DI}}_{m}={\frac {{\underset {1\leqslant i<j\leqslant m}{\text{min}}}\left.\delta (C_{i},C_{j})\right.}{{\underset {1\leqslant k\leqslant m}{\text{max}}}\left.\Delta _{k}\right.}}}$.

Being defined in this way, the DI depends on m, the number of clusters in the set. If the number of clusters is not known apriori, the m for which the DI is the highest can be chosen as the number of clusters. There is also some flexibility when it comes to the definition of d(x,y) where any of the well known metrics can be used, like Manhattan distance or Euclidean distance based on the geometry of the clustering problem. This formulation has a peculiar problem, in that if one of the clusters is badly behaved, where the others are tightly packed, since the denominator contains a 'max' term instead of an average term, the Dunn Index for that set of clusters will be uncharacteristically low. This is thus a worst case indicator, and has to be kept in mind. There are ready implementations of the Dunn index in some vector based programming languages like MATLAB, R and Apache Mahout.[3][4][5]

• Pakhira, Malay K.; Bandyopadhyay, Sanghamitra; Maulik, Ujjwal (2004). "Validity index for crisp and fuzzy clusters". Pattern Recognition. 37 (3): 487–501. Bibcode:2004PatRe..37..487P. doi:10.1016/j.patcog.2003.06.005.
• Bezdek, J.C.; Pal, N.R. (1995). "Cluster validation with generalized Dunn's indices". Proceedings 1995 Second New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems. IEEE Xplore. pp. 190–193. doi:10.1109/ANNES.1995.499469. ISBN 0-8186-7174-2. S2CID 7816379.