Journal Club: when the headline and findings diverge

This is Joseph.

This study has been making the rounds:

Now I want to be very careful here. What was the actual category (listed as #2 in the article:

The datasets are available from the corresponding author on reasonable request.

Now look at what happens:

In summary, of 1792 e-mails sent, we did not receive any response for 1538 articles because messages were not delivered (N=77; 4.3%) or the author did not reply (N=1461; 81%). Responses were received from 254 (14%) contacted authors 

So, first of all, I am not sure that we should consider the 4% who did not have a message delivered considered to be "non-compliant". That is important as the headline number of  93% did not respond includes those who were not actually successfully contacted. Of those responding, about 1/2 (122 vs 132) shared their data. 

Then we have the most common reasons:

1. The authors asked for more information about our study, but after our detailed response and clarification, we did not receive further response from them 
2. Their informed patient consent did not include sharing data with other researchers, or the ethical committee prohibited external data sharing and use 
3. They cannot access the data, either because they are no longer in the institution that conducted the research, or they are no longer active on the project 
4. They do not want to share the data or in any way participate in our study without a specific explanation

Some of these are declining to be in the study and some seem like reasonable excuses. Data often belongs to the institution and a closed down lab might have no way to get data. Still, of the responders the success rate was 48%, which seems like a reasonable level of outcome given that you are openly declaring that this is for a study and not for the research project itself.

That reminds me, what about the non-responders:

The Ethics Committee of the University of Split School of Medicine approved the study protocol. Scanned approval of the Ethics Committee was available to the study participants on request. Study participants were informed in the invitation that their response to the emailed questions is considered as their written informed consent.

So, responding to the email is considered to be consenting to the study. Which means that people who decided not to participate are being told that sending a response is considered written and informed consent. Now the sample letter they used is not yet available online, but is participating in a study mandatory? 

So what does Canada think about this? 

. . . the core ethics principle of respect for persons implies that individuals who participate in research should do so voluntarily, understanding the purpose of the research, and its risks and potential benefits, as fully as reasonably possible

What about the US NIH:

Informed Consent

The Belmont principle of respect for persons is primarily applied by requiring that all human

subjects research participants provide voluntary informed consent to participate in research.

The three fundamental aspects of informed consent are:

1. Voluntariness

2. Comprehension

3. Disclosure

And:

Potential participants must understand that enrolling in the research is voluntary and that they may withdraw from the study at any time without penalty or loss of benefits

So there are two reasons that a person could chose not to have provided data:

1. They were not being especially honest in providing data
2. They chose not to consent to a research study

What do the authors conclude:

In conclusion, authors of research articles are frequently not willing to share their data, even if they wrote in their manuscript DAS that they would do so. Our findings can enable the creation of new guidelines and practices in the research community to foster the availability of raw research data

It is worth also noting that they are not making the raw data available for this study, which doesn't seem to foster the availability of raw data.

We did not publish our raw data along with the manuscript because it could be understood that we are publicly shaming authors who did not want to share their data. As for the raw data that were received during this study, we informed our study participants that those raw data will be deleted after being examined and that all data and communication will be treated with strict confidence

 But putting that aside, they really can't tell the difference between choosing not to participate and declining to share data.  But they had a 48% success rate among the responders and even that denominator is shaky.

Consider items 1 and 4 of the most common reasons. How different is that from withdrawing consent from the study based on being informed? Reason #3 (lost access) seems like a pretty odd thing to frame as "not compliant" (the author's title). Reason #2 is odd but could be an issue for sharing data outside of the context of replication or exploration (what is reasonable is a very important nuance).

So all we could be discovering is that busy researchers often decline to participate in mass email research studies that involve a fair amount of labor (in this case uncompensated). Among those who had the ability to respond, I think they got better than half as reasons like the ones below may not really make sense to include in the denominator:

• A conceptual article, no data to share
• The author wanted ethics approval translated into English
• They cannot access the data, either because they are no longer in the institution that conducted the research, or they are no longer active on the project
• The corresponding author instructed us to use a web service to request the data

Is it "not compliant" if the author lost access to the data (institution being especially clear) or they wanted a web service to be used? What is really missing here is a clear statement of the estimand, which is the propensity to participate in an email study and not the propensity to share data with peers. 

In any case, this is a long discussion of an article. But the headline is getting a lot of engagement and I am not sure it can be clearly interpreted that way. A data request, as a part of research, becomes voluntary based on clear ethical principles and it seems very odd to characterize a decision not to engage as "not compliant" when you state even replying to the email is taken as informed consent. Nobody is entitled to informed consent (it's consent not obligation) and so it is utterly unclear what the rate would be in the 85% who either never got the email or chose not to consent to a voluntary research study.