Experiments

In this section, the experimental settings are introduced first in Section 5.1, then the performance of the proposed method on synthetic datasets are presented in Section 5.2. We further present the effectiveness test on our method against distributional shift between domains on real-world datasets in Section 5.3. In addition, we verify the effectiveness of framework components through ablation studies in Section 5.4. We also measure the parameter sensitivity in Section 5.6.

Experimental Settings

Synthetic datasets.

In order to assess the generalization capability of our proposed method towards unseen domain data, we conduct an evaluation using synthetic datasets. A source domain is first generated by sampling K equal-sized data clusters. The data features are sampled from multi-Gaussian distributions with randomized centers and covariance matrices, which is similar to previous works [4, 11]. Subsequently, a corresponding target domain is created by randomly perturbing the centers of the source domain clusters. This ensures the presence of distributional drift between the train and test set data. To provide comprehensive results, we vary the value of K and generate 10 distinct datasets for each value of K. We train the clustering model on source domain and test on the target domain. Our experimental results are reported as an average of 5 runs on each dataset, with different random seeds employed to ensure robustness.

Real-world datasets.

To further evaluate the generalization capability of our proposed method under real-world senarios, commonly used real-world benchmark datasets are included. (1) Digits which includes MNIST and USPS, is a standard digit recognition benchmark that commonly used by previous studies [16, 17, 33, 35]. Follow previous works [16, 17], we train the model on the source domain training set and test the model on the target domain test set. All input images are resized to 32 × 32. (2) CIFAR-10 [13] is commonly used image benchmark datasets in evaluating deep clustering models. We treat the training set as source domain and test set as target domain. We introduce CenterCrop to the test set to create distribution drift.

Comparison methods.

We evaluate the proposed method on both synthetic and real-world benchmark datasets and compare it with both traditional clustering and state-of-the-art deep clustering techniques such as k-means, GMM, DAE [30], DAEGMM [31], DEC [33], DCN [35], JULE [36], CC [15] and IDFD [34].

Evaluation metrics.

In our evaluation of clustering performance, we employ widely recognized metrics, namely normalized mutual information (NMI) [3], adjusted rand index (ARI) [38], and clustering accruracy (ACC) [3]. By combining NMI, ARI, and ACC, we can comprehensively demonstrate the efficacy of our clustering results.

Implementation details.

Our proposed model serves as a general framework, allowing for the integration of various commonly used deep representation learning techniques as the encoder part. To ensure a fair comparison with previous works, we enforce the use of the same encoder for feature extraction across all models. Specifically, for synthetic data, we utilize a three-layer multilayer perceptron (MLP) as the encoder. For the Digits dataset, we employ the classical LeNet-5 network [14] as the encoder. Furthermore, for the CIFAR-10 datasets, we utilize the ResNet-18 network [9] as the encoder. We use L=4 layers of blocks to update the synthetic datasets and L=5 for the real-world datasets. The temperature τ is set as 1.0 throughout the whole experiments. We use an linearly increasing series of values for the weights α for penalizing each block in loss function, where the final layer has the largest weight. We train the whole network through back-propagation and utilize Adam [12] as the optimizer. The initial learning rate is set as 5e-3 for the synthetic datasets and 5e-4 for the real-world datasets, and the weight decay rate is set as 5e-4. The total number of training epochs is 500 for the synthetic datasets and 2, 000 for the real-world datasets. The batch size is set as 256 for synthetic and DIGITS datasets, and 128 for CIFAR-10 dataset. Data augmentation techniques are added like previous papers [15, 34] for the purpose of training discriminative representations for all the image datasets. The experiments are carried out on NVIDIA A6000 GPUs, which takes around 30 gpu-hours to train the model on CIFAR-10 dataset.

Synthetic Data Results

Table 1 presents the clustering performance on both the trained source domain and test target domain of synthetic datasets. The results show the remarkable effectiveness of our proposed TDCM framework in achieving superior generalization performance when transferring the trained model from source to target sets across all synthetic scenarios. Specifically, TDCM consistently outperforms all the comparison methods, exhibiting an average improvement of 0.215, 0.243, and 0.157 on NMI, ARI, and ACC metrics, respectively. Notably, the performance of the TDCM model on the test set exhibits only a marginal average decrease of 0.033, 0.044, and 0.024 on NMI, ARI, and ACC metrics, respectively, compared to the training set. These results provide strong evidence that our proposed method significantly enhances the transferability of the clustering model, demonstrating its superior performance and robustness. On the other hand, although the comparison methods can achieve competitive performance on the trained training set, their performance drops significantly when transfer from source to target domains, which proves that their fixed set of optimized centroids can not handle the distribution drift between domains.

Real-world Data Results

We report the clustering results of the real-world datasets in Table 2. The results demonstrate the strength of our proposed TDCM framework by consistently achieving the best performance when test on test sets across all datasets. Specifically, TDCM consistently outperforms all the comparison methods, exhibiting an average improvement of 0.206,0.342, 0.439 on MNIST, USPS, and CIFAR-10 data test sets, respectively. Our results strongly demonstrate the enhanced transferability of our proposed method for the clustering model, highlighting its superior performance. It worth noting that the improvement of our model on CIFAR-10 dataset is more significant than the other two digits dataset. A possible reason is CIFAR-10 datasets are more complex than the other two datasets, which may prove that our model can handle complex data with high dimensional features.

We have carried out additional experiments using the COIL100 dataset [23], which consists of 100 distinct classes. In these experiments, the first 50 classes are used as the training source domain, while the remaining 50 classes serve as the test target domain. The Normalized Mutual Information (NMI) clustering results for our method and comparative techniques are presented in the Table 4. We can observe that our methods can outperform all the comparison methods on the target domain with only minor performance drop from source domain (0.002). Although state-of-the-art comparison methods such as JULE, CC anad IDFD can achieve comparable results on source domain, they are notably affected by the domain shift between source and target, leading to a noticeable decline in performance.

Ablation Studies

Here we investigate the impact of the proposed components of TDCM. We first consider variants of removing the real-symmetric constraints and orthogonal constriants in our model, named variant-R and variant-O. In addition, we also remove the entropy loss in our overall loss function, named variant-E. We present the results on two synthetic datasets (K=2,5) and CIFAR-10 real-world dataset in Table 3, where we can observe a significant performance drop consistently for all variants. Especically, we observe that the standard deviation of all variants are larger than the full model, especially for the variant-R that removes real-symmetric constraint. Such behavior may demonstrate the importance of these proposed constraints in guaranteeing a stable training process, which is highly consistent with our theoretical analysis.

Visualization of Centroids Updating Process

In order to illustrate the learned centroids updating behavior by our designed module, here we visualize the layer-wise updated centroids in K=2 synthetic dataset test set in Figure 3. From the visualization we can observe that by forward passing the updating blocks, the centroids are adapted to a more clear cluster structure by the learned similarity metrics. It worth noting that the final adapted centroids are not necessarily at the 'center' of the clusters, which demonstrate that the designed module can automatically find the underlying similarity metric between samples.

Sensitivity Analysis

In this section, we include synsitivity analysis on the critical parameters that involved in our designed model, which is the temperature parameter τ in Equation 2. In a intuitive manner, the temperature parameter significantly influences the determination of the computed distance metrics between samples. Specifically, an increased value of τ tends to reduce the disparity between the estimated similarity scores between samples and centroids. Therefore, we vary the value of τ from 0.1 to 5.0 to examine the sensitivity of choice on temperature. We present the results of sensitivity analysis on K=5 synthetic datasets in Figure 4. From the figure we can observe that the performance of our proposed method usually peaks at the value of τ=1.0 for both training and test set. It also worth noting that our proposed method is robustness since the performance will stably change with the value of temperature parameter τ. Besides, an interesting observation is that when the value of τ is becoming larger (e.g. 5.0), the gap of performance between source and target domain decreased. A possible explanation is that a larger temperature value can smooth the discrepancy between samples, which results in a better generalizable model.

We also include additional sensitivity analysis on the hyperparameters, e.g. number of updating blocks L, weights of penalization terms on each layers α, and β to balance the clustering loss and entropy loss. Here we use the synthetic dataset (K=2) for conducting the performance for the purpose of efficiency. The results are presented in Table 5.

(1) It is noticeable that the selection of hyperparameters can influence the training capacity within the source domain. However, the ability to generalize from the source domain to the target domain is hardly affected to these changes.

(2) We vary the number of updating blocks L from 2 to 16. We can observe from the results that an appropriate choice of number of updating blocks can affect the training performance on the source domain. Choosing either a too small value (such as L=2) or a too large one (such as L=16) could hinder the model's ability to optimize towards the underlying optimal clustering.

(3) We compare two choices of α to tune the balance of loss between different blocks. We adopt αlinear, which is to equally penalize the loss on each block, and αlast, which only penalize on the last block. We observe that the performance only slightly different between these two, which indicates our model is not sensitive to the choice of α.

(4) We additionally evaluate the impact of the regularization term through the hyperparameter β. Our observations indicate that performance can significantly drop if the entropy regularization is eliminated (β=0) or if the regularization is excessively strong (β=10). This decline in performance is particularly pronounced in the case where β=10.