The Mandatory Field's First Month Produced One Thousand One Hundred Entries, And Four Hundred And Six Of Them Are The Number Printed In The Example

Compliance is one hundred percent. The modal answer is 5.0. The help text's worked example uses 5.0, and for about a third of those trials 5.0 is defensible, which is the problem.

Science By Anita Farnsworth Filed from the science desk with one thousand one hundred entries in a spreadsheet 6 min read
A registry form field for the minimum clinically important difference holding the value 5.0 beside a red required asterisk, with help text below it whose worked example reads 5.0 in a gold box and a note that an example with no number in it is not an example, beside a histogram of one thousand one hundred and six entries in which the 5.0 bar runs red across the card at four hundred and six and every other value is a short grey stub, under a line recording that one hundred and thirty-one of the four hundred and six are correct and that a defensible mode cannot be told apart from a copied one, over a red banner reading compliance one hundred percent and the mode is the example and a gold band reading 406 of 1,106 entries read 5.0.
The field is full. A third of it is the number printed in the help text.

On the nineteenth of September this desk reported that the registry had made the minimum clinically important difference compulsory, that the form would not accept a blank, and that two hundred and ninety-four of the three hundred entries then on file had been stamped after the trial's result was posted. That was a story about timestamps. This is a story about the number itself, and it is worse, and it is not the registry's fault in any way this desk can locate.

The field has now been compulsory for one month. One thousand one hundred and six trials have registered in that month and one thousand one hundred and six have supplied a minimum clinically important difference, because the form will not submit without one. Compliance is one hundred percent and has been since the first day, which is what a required field does and is not a finding. The finding is the distribution. Four hundred and six of the one thousand one hundred and six entries read 5.0.

Where 5.0 Comes From

The help text beside the field is one hundred and forty words long and it is good help text. It explains what the quantity is, it explains why it has to be specified in advance, and because it is good help text it finishes with a worked example, which is the thing that makes help text usable. The worked example is a hypothetical trial of a hypothetical analgesic, and in it the registrant has entered a minimum clinically important difference of 5.0 on a hundred-point scale.

This desk would like to be clear that nobody did anything wrong in writing that. An example without a number in it is not an example. The registry's documentation team has produced the best help text in this registry by a distance and the reason the example exists is that the field's first four months as a voluntary entry produced eleven responses, nine of which were the word unknown.

"You are going to tell me to take the number out of the example. If I take the number out of the example it is not an example, it is a definition, and we had a definition for four months and it produced nine unknowns and two blanks." The registry's documentation lead, October 6

The Worst Part Is That 5.0 Is Often Right

This desk took the four hundred and six and read them, which took nine days, and sorted them by whether 5.0 is a defensible threshold for that trial's primary outcome. One hundred and thirty-one of them are measuring something on a hundred-point patient-reported scale where five points is a long-standing and well-published threshold with its own literature. For those trials 5.0 is not a copied number, it is the correct number, and it is the number a careful registrant would have written with no example in front of her.

A further ninety-four are on scales where five points is arguable. One hundred and eighty-one are on outcomes where it is not arguable at all: time to event in days, counts of exacerbations, millimetres of mercury, one bone density measure where 5.0 is roughly four times any effect anybody has ever reported. Those one hundred and eighty-one are the copied ones and this desk is comfortable saying so.

But the one hundred and thirty-one are the methodological problem, and it is the problem this desk has not been able to solve in two weeks of trying. A defensible modal answer cannot be distinguished from a copied one. If this field had produced four hundred and six entries of 5.0 on outcomes where 5.0 is absurd, the registry could flag them and the problem would be a data quality problem with a data quality solution. What it has produced is a pool in which a third of the suspicious answers are correct, which means no automated check and no reviewer can tell, for any individual trial, whether the specified threshold is a scientific judgement or a default, and the whole value of specifying it in advance rests on it being a judgement.

The Registry Ran The Experiment Nobody Designed

On the twenty-fourth of September, before this desk had published anything and for its own reasons, the registry changed the worked example. It did not remove the number. It replaced it with a number chosen to be obviously unusable as a real answer, and it did not announce the change.

The mode moved. Within nine days, 5.0 had fallen from thirty-seven percent of new entries to nine percent, and the new example's value had gone from nothing at all to twenty-two percent of new entries, which is one hundred and three trials specifying a threshold that the registry chose specifically because no trial should have it. This desk is calling that the cleanest experiment this registry has ever run, and nobody designed it, nobody registered it, there was no protocol, and the documentation lead who made the change did it on a Thursday afternoon because the number was bothering her.

It establishes the thing the first part of this story could only suggest, which is that the example is not advice, it is an anchor, and it does not matter whether the anchor is plausible. One hundred and three registrants copied a number that was placed in front of them precisely because it could not be copied in good conscience, and they copied it in nine days.

The Request This Desk Is Granting

The documentation lead asked this desk not to print the new example's value. Her reasoning took four sentences and this desk did not argue with any of them: the value is currently in twenty-two percent of new entries because registrants are reading the help text, and if it appears in a newspaper beside the sentence a third of registrants copy the example, it becomes a number that people who have not read the help text also know, and the registry loses the one diagnostic it now has for telling a copied entry from a judged one.

So this story does not contain that number, and this desk has to concede on the record that publishing the whole of this finding would damage the instrument the finding is about. That is an uncomfortable thing for a methodology column to write and this desk has written it three times in two days to see whether it stops being true. It does not stop being true. The suppression is a single number, the finding is intact without it, and the registry has agreed to publish the value itself in twelve months with a note explaining why it waited, and this desk will be checking in twelve months.

Desk note: the timestamp finding from the nineteenth is unaffected and is now less interesting than this one, which this desk did not expect. One hundred percent compliance and a modal answer taken from the example are the same sentence read twice. The registry's position, which this desk endorses, is that the field is still worth having and is still better than the blank: a copied threshold specified in advance is at least fixed in advance, and a trial that copies 5.0 and then reports a two-point effect has still written the number down before it knew, which is the whole of what the field was for. The useful rate this desk reported in September at two percent is now eleven percent and rising, by the registry's own count, and nobody is claiming that is good. The one hundred and thirty-one defensible entries are being left alone and will never be separable from the copies, and that is the permanent cost of putting a number in an example, and there was no way to avoid putting a number in the example.