Clustering refers to automatic (unsupervised) procedures that group observations by relatedness. While numerous clustering procedures have been proposed for various kinds of exploratory data analysis tasks, here we focus on problems where the clusters are of scientific interest and it is important that their inference proceed in a principled way, acknowledging any uncertainty in the final answer. Bayesian approaches to clustering are promising since important assumptions for the inference of clusters can be encoded into the prior (and the loss function) while the posterior describes the clustering uncertainty. However, it is known that the popular Bayesian approach to clustering based on mixture models is brittle to even small imperfections in the distribution (commonly Gaussian) used to model the data within each cluster. In this talk, I will discuss two approaches to fix this brittleness. First is a general information-theoretic approach to fit models that we know are imperfect, and the second is a decision-theoretic approach that changes the loss function to target a flexible notion of clustering defined directly in terms of the population density. We illustrate applications to the clustering of galaxies based on​ astronomical survey data, and of cells based on RNA-sequencing data.