Summary
A bioRxiv preprint finds that randomly dividing single-cell data can leak information between training and test sets when related cells share labels. The authors propose eakcheck, a tool that estimates this leakage before a predictive model is trained.
Machine-learning benchmarks in single-cell biology can produce misleadingly high scores when related cells are randomly divided between training and test sets, according to a bioRxiv preprint by Sha Sun and Hu Cang. The authors report that cells from the same clone, patient or experimental batch can carry shared label information into the training data, allowing a model to recognise relatives of test cells rather than learn a rule that transfers to genuinely independent samples.
The preprint, posted on September 15, 2026, examines this issue in settings involving lineage-barcoded cells and patient data. It introduces eakcheck, a tool designed to estimate the risk before the outcome-prediction model is fitted.
Why cell-level randomisation can mislead
Single-cell experiments measure many individual cells, but those cells are not always independent observations. Cells can share ancestry, come from the same patient or be processed in the same batch. If a dataset is split at the level of individual cells, members of the same biological or technical group can appear on both sides of the benchmark.
That creates a route for label information to cross from the training set into the test set. A model forecasting differentiation, reprogramming or therapeutic response may score well partly because it has encountered a close relative of the test cell, rather than because it has identified a transferable relationship between an early transcriptomic profile and the later outcome.
The alternative is grouped validation: keeping cells from the same clone, patient or other relevant group together when creating training and test sets. This better tests performance on independent groups, but it also leaves fewer independent units supporting each estimate of model error.
Measuring the leakage before modelling
The authors describe two properties that can be calculated from the data before training the prediction model.
Exposure is the fraction of test cells that have relatives in the training set. It indicates whether a random split has opened a potential path for shared group information to enter the test result.
Retrievability measures how often a nearest-neighbour search finds such a relative instead of an unrelated cell. It captures whether the related cells remain close enough in their measured gene-expression profiles for a model to use that connection.
The preprint reports that exposure determines whether the leakage channel exists, while retrievability determines how strongly it can affect the score. The effect was described as negligible when cell state had become decoupled from ancestry, but substantially larger when clonal sister cells remained close in expression space. In a patient cohort, the authors report that the resulting inflation was large enough to overturn a clinical conclusion.
Eakcheck computes both properties in seconds, allowing researchers to assess the structure of a benchmark before fitting the outcome model. The approach is intended as an early diagnostic for deciding whether a cell-level random split is suitable or whether validation should instead be grouped around patients, clones or another independent unit.
The work is presented as a bioRxiv preprint rather than a peer-reviewed journal article. The supplied abstract reports the direction and settings of the findings but does not give cohort sizes or numerical changes in benchmark scores, so the magnitude of the problem will depend on the dataset structure and the biological question being tested.