Summary
A bioRxiv preprint reports that RELAX produced different likelihood estimates in repeated runs, with some changes reversing statistical conclusions. The authors link the instability to site-class models and show that a seed alone did not ensure repeatable results in their tests.
A bioRxiv preprint by MinSeo Kim and Jae-Ho Shin reports that RELAX, a method for testing changes in the intensity of natural selection, returned different likelihood estimates when the same analyses were repeated under default settings. In some of the researchers’ tests, the changes were large enough to reverse the statistical verdict.
RELAX uses a parameter called K to estimate selection intensity across evolutionary branches, alongside a likelihood-ratio test. In the method’s standard interpretation, K below 1 indicates relaxed selection and K above 1 indicates intensified selection relative to the comparison set. The new report focuses on whether the software can reproduce its own fits reliably.
Repeated runs produced different estimates
The researchers refitted 27 enzyme entries under two optimiser configurations. None reproduced its log-likelihood to within 0.01 units; the median change was 103 units, and the largest exceeded 3,400. Four analyses changed their statistical verdict.
The researchers also report that a byte-identical command returned a different likelihood on every repetition, even when run single-threaded. This occurred across three software releases. With alignments simulated under the fitted model, 3.3% of replicates reproduced the result, according to the abstract.
The authors tested whether this instability was specific to RELAX or associated with the model’s treatment of variation among sites. In PAML, a two-ratio model without site classes reproduced its log-likelihood for all 288 genes tested. Models with site classes agreed in 27% to 67% of cases. Site classes allow a model to represent different categories of sites with different evolutionary behaviour; the comparison points to that mixture as a source of instability in these tests.
Output and seed handling can affect interpretation
The preprint reports that some diagnostic information was not apparent from the usual output. Among 410 fits, 46 ended with a negative likelihood-ratio statistic, which the authors say should not occur if the models have converged. In 123 fits, the reported K had been re-estimated under a domain restriction rather than taken from the unconstrained maximum.
The authors also found that assigning the documented random-number seed on the command line did not pass it to the generator, even though the program read back the supplied value. In their tests, a seed combined with a single-thread setting reproduced 60 of 60 runs on 20 genes under two releases. The seed alone reproduced none of five runs.
The authors report that none of 234 published studies using RELAX stated a seed. They recommend repeating fits and reporting the dispersion across runs, so readers can assess how stable an estimate or statistical conclusion is. For research that compares selection intensity across evolutionary groups, reproducibility of the underlying fit is important: a changed likelihood or verdict can alter the interpretation of the analysis.
The findings are presented in a bioRxiv preprint, and the reported results concern the software versions, analyses and settings described by its authors.