Autoencoder-Based Nonlinear Dimension Reduction for Single-Cell RNA-Seq Data: A Comparative Study of t-SNE and UMAP

Abstract

This paper proposes using an Autoencoder (AE) prior to t-SNE or UMAP visualization for scRNA-seq data. Direct application of t-SNE/UMAP to the raw, sparse expression matrix often yields unstable, poorly separated clusters. To address this, the framework first employs an AE to learn a denoised, compact latent representation. Subsequent t-SNE or UMAP embedding of this latent space produces more robust visualizations with enhanced cluster consistency and structural separability. A real-data-based comparison shows that, when using the same AE-derived latent space, UMAP outperforms t-SNE. It achieves better cluster cohesion, stronger global structure preservation, greater robustness to initialization and data perturbation, and lower computational cost. Statistical validation via a projection F-test confirms that clusters in the AE latent space exhibit significant between-group mean differences, quantifying the observed visual improvement. The study concludes that AE-based representation learning creates an effective input space for nonlinear embedding, with the AE-UMAP pipeline emerging as a particularly stable and efficient choice for scRNA-seq exploratory analysis.

Purpose: This study aims to investigate the effectiveness of AE based latent representations in enhancing nonlinear dimension reduction methods, namely t-SNE and UMAP, for single-cell gene expression data analysis. The performance of AE-based UMAP and AE-based t-SNE is systematically evaluated from multiple perspectives, including visualization quality, clustering consistency, structural preservation, and robustness.

Methods: This paper constructs a two-step dimension reduction framework for single-cell gene expression data analysis. First, an AE is employed to compress high-dimensional, sparse, and noisy gene expression data into a low-dimensional latent representation. Subsequently, t-SNE and UMAP are applied to the learned AE latent space for nonlinear embedding and visualization. The performance of different methods is systematically evaluated under multiple experimental conditions using clustering consistency metrics, structure preservation measures, and a projected F-test.

Results: Experimental results indicate that directly applying t-SNE or UMAP to the original expression data fails to stably recover meaningful clustering structures, whereas nonlinear dimension reduction performed on AE latent representations substantially improves visualization quality and clustering stability. Within the same latent space, t-SNE and UMAP exhibit comparable performance in terms of clustering accuracy; however, UMAP demonstrates superior performance with respect to cluster compactness, global structure preservation, stability across repeated experiments, and computational efficiency. Statistical testing further confirms the significance of between cluster differences in the AE latent space.

Contribution: This study systematically reveals the critical role of AE latent representations in stabilizing nonlinear dimension reduction for single cell data and provides a quantitative comparison between t-SNE and UMAP within a unified latent space. The results demonstrate that UMAP applied to AE latent representations achieves superior performance in terms of visualization stability and computational efficiency, offering a more robust two step dimension reduction strategy for exploratory analysis of high dimensional single cell data.