One thing that never really comes up when people talk about “Questionable Research Practices,” is what to do when you’re a junior in the field and someone your senior suggests that you partake. [snip] It can be daunting to be the only one on who thinks we shouldn’t drop 2 outliers to get our p-value from .08 to .01, or who thinks we shouldn’t go collect 5 more subjects to make it “work.” When it is 1 vs 4 and you’re at the bottom of the totem pole, it rarely works out the way you want. It is hard not to get defensive, and you desperately want everyone to just come around to your thinking- but it doesn’t happen. What can the little guy say to the behemoths staring him down?
I’ve recently been put in this situation, and I am finding it to be a challenge that I don’t know how to overcome. It is difficult to explain to someone that what they are suggesting you do is [questionable] (At least not without sounding accusatory). I can explain the problems with letting our post hoc p-value guide interpretation, or the problems for replicability when the analysis plan isn’t predetermined, or the problems with cherry picking outliers, but it’s really an ethical issue at its core. I don’t want to engage in what I know is a [questionable] practice, but I don’t have a choice. I can’t afford to burn bridges when those same bridges are the only things that get me over the water and into a job.
I’ve realized that this amazing movement in the field of psychology has left me feeling somewhat helpless. When push comes to shove, the one running the lab wins and I have to yield- even against my better judgment. After six five months of data collection, am I supposed to just step away and not put my name on the work? There’s something to that, I suppose. A bit of poetic justice. But justice doesn’t get you into grad school, or get you a PhD, or get you a faculty job, or get you a grant, or get you tenure. The pressure is real for the ones at the bottom. I think more attention needs to be paid to this aspect of the psychology movement. I can’t be the only one who feels like I know what I should (and shouldn’t) be doing but don’t have a choice.
Edit: See another great point of view on this issue here http://jonathanramsay.com/questionable-research-practices-the-grad-student-perspective/
edit3: Changed some language
The Etz-Files 
Well, you could say “Wouldn’t it be simpler just to make up the data?”. And when everyone looks at you in shock, you then say, “Well, how is removing these two outliers — without writing up in the article /a/ that we did it and /b/ why we did it — actually any different from making up the data?”
We won’t fix anything until we accept that the only difference between QRPs and fraud is that first-year undergraduates don’t know what QRPs are, whereas anyone can probably work out that inventing your data is wrong. But once you’ve been exposed to the fact that data peeking and running 10 more participants, or not reporting half the DVs, are going to change your results in *exactly the same way* as changing half a dozen 4s into 5s on the questionnaire, you have no excuse any more.
Yeah, I wish I had the balls to say that! I even got a response [to the tune of], “You’re concerned about the ethics right? We’ve seen the whole replication movement and understand the file-drawer problem [goes on to accurately summarize the importance of reporting all conditions and variables to save other people time and money]. But listen. The times haven’t changed and null effects still don’t have a chance, and a reviewer is going to kill us on this. So we aren’t reporting that we even tested [X]. Maybe mention it as a future direction.” What can anyone say when someone is clearly playing the game like that?
At that point, when I was going against 5 other people who just want get the paper out, I realized the point was lost and I just gave in.
Having found myself in a similar situation myself previously, I completely empathise. Good job on making your concerns known. I admire your integrity. You still have a couple of options here other than giving in. If your seniors are completely against publishing the results in the manuscript in full, you could as a compromise suggest including them in the supplementary material. If no compromise can be reached, I would consider removing your name from the paper entirely. It’s a bold move, and won’t win you friends, but it will make your point very clearly, will avoid compromising your principles, and may even win you respect. Publish what you *believe* in.
Thanks so much for the comment, JJ.
/ I would consider removing your name from the paper entirely./
I’ve started to seriously consider that option. Part of me wants to just give in and write the paper the way they want it, the same part that has been collecting data for 5 months. It was essentially my own project that I’ve been spearheading, and I was even working extra hours to get it done. But the other part of me knows that what we are doing is less than forthright, and I seem to be cursed with a moral compass.
/Publish what you *believe* in/
I think I’m at the point in my career where this value is being formed. I want to show everyone the great work that came out of our efforts from the last 5 months; the data isn’t perfect but I’ve never seen any paper that is.
I also think I have an ethical obligation to the subjects who volunteered their time to actually *use their data*. If I participated in a study for someone and then found out later that my data was excluded for no good reason other than it made the reviewers happy, I would feel betrayed.
I agree that, once you have clearly made your point about the scientific and ethical issues, there is no honorable attitude but to refuse to sign the paper.
I had to do this once, about twenty years ago. Although it involved a serious conflict with my boss of the time and earned me a reputation of methodological ayatollah, I am still glad I made this decision.
You can go work for someone else. I don’t know who you are or much about you (other than what I can see on your blog), but I would be proud to have someone with that kind of integrity working in my lab. And I know there are plenty of other people out there who do too.
I’m reminded of a story. A young, but already up-and-coming researcher was writing a number of papers that challenged the famous researcher X. The young researcher was told by senior colleagues “Oh you shouldn’t say that. You are going to make famous researcher X mad at you.” The young researcher thought about it, but ultimately decided to do what he thought was right (which was to publish as truthfully as possible). Famous researcher X was indeed mad at the young researcher. But, perhaps to his surprise, the young researcher found that just by virtue of having high integrity, a number of other well-known researchers respected (and even liked!) him. And in the end, it turns out that these were the kinds of people the young researcher preferred to have respect from (and friendships with) anyhow.
Thanks, Ryne, I really appreciate these comments.
/You can go work for someone else./
Actually, one of the reasons I felt I could voice my concern is that I *am* leaving for another job. I worked for months to get this project done, and I wanted to stay on to write the paper in my free time this summer- while I started my new job. If I were still working for the lab I think I would be even more hesitant to bring it up.
/ And in the end, it turns out that these were the kinds of people the young researcher preferred to have respect from (and friendships with) anyhow./
Thanks for the pep talk. Like I said to JJ, these are formative years for me and I think the decisions I make now are going to shape my career. I feel really lucky to be starting a career in a time where I can have people I’ve never met before reach out to me with support. So thank you for that.
Please let us be balanced. It’s not that only senior people push QRP’s and that there are no senior people promoting openness. I have also seen it the other way around.
Sure, I hope the post didn’t come off sounding that way. But if a junior wants to do questionable work and a senior knocks some sense into him, I’d call that good mentoring. If a senior wants to do questionable work and a junior comes out against it, what then? I’m just saying the problem isn’t so tough when the senior is the one doing the right thing.
Well said. Hold your ground and saying less is more in my opinion.
“I’m not comfortable with that and will add replicates.”
End of story. Any thing else leaves it open to a discussion and then you are on a slippery slope.
As a junior person, it can help to refer to some kind of authority rather then your own opinion or values, since senior people may think they know better than a junior person’s opinion/values! In this respect the latest Declaration of Helsinki (which many journals and ethics committees hold research to) may come in handy:
“Researchers have a duty to make publicly available the results of their research on human subjects”
“Negative and inconclusive as well as positive results must be published or otherwise made publicly available”
although they may have been referring to medical trials rather than your topic..
Alex
Thanks for the advice, Alex. I think I will try to collect some comments like that and make my case again.
Thank you for sharing. I completely understand your concerns and have been in a similar situation. It becomes especially daunting if you are asked to run more complex models the researchers in higher positions could not do themselves and you know that you are fishing for p-values there. They couldn’t do it without you and you know it’s at least questionable, especially if the usage of different models will not be announced in a publication. You might also be the only one who actually understands the problem of multiple comparisons in your team. You also know that this would be really important for your future, it might lead to serious problems if you call it out and the researcher also might need this study and to publish something you will de-facto need a significant effect.
What I will try to do – with thanks to Andrew Gelman for the advice – is not approaching it as an ethical issue but strengthening the idea of including a footnote mentioning all the analyses done and letting reviewers an readers decide about how to judge the results. As this can be seen as an qualitative improvement of the paper, this might be a helpful way to approach such situations. It’s still not great, though, and people like us have very limited possibilities and powers.
Thank you for the comment, Kival. I think you’re right. If I can’t get my ethical concerns across then it may be more practical to make an argument for a more informative and complete analysis report.
[…] https://nicebrain.wordpress.com/2014/06/15/not-quite-fraud-but-close/ […]
[…] Sad account of what it’s like to be a student in a psychology lab that openly pushes for terrible practices […]
[…] Sad account of what it’s like to be a student in a psychology lab that openly pushes for terrible practices […]
Have you thought about asking permission to use the same data later, in a re-analysis? Specify one or two alternatives that are more about your skill development — say, a bootstrap analysis of confidence intervals, or (if you have the chops) a Bayesian approach. State that you would not submit to a journal or working-paper archive for X years (I’d recommend 3).
[…] : que faire quand on est étudiant en psychologie et que l'équipe de chercheurs qui vous encadre vous demande de flirter avec la fraude scientifique, de modifier un peu l'échantillon de personnes testées pour obtenir un résultat positif (et […]
The only right thing to do is to insist. Pulling out of the paper entirely is not enough. Is there a research ethics comitee in your university ? It is their job to help you there. There should be an internal enquiry on other works by the same research team.
A lamer CYA solution would be to exclude yourself from the analysis in the “Who done what” section of the paper.
Minh, Ph.D., senior researcher
Thanks for the comment, Minh. I still don’t know what I’m going to do going forward. I’ve switched labs though, so I can decide what to do more objectively now.
Thanks for your post. It is good to see people caring about doing the right thing.
This may not be helpful (because I’m not a psychologist; I’m a retired mathematician who taught graduate statistics courses for a number of years before retiring), but in case it can help you (or anyone else reading this blog) make a good argument for something, or give references to back up your point, I’ve got a fair amount of information on the web about some of the types of statistical issues you are (rightfully) concerned with. These are:
1. Most recently, a series of blog posts (with more forthcoming) on statistical issues I noticed in some of the replications in the recent special issue of Social Psychology (http://www.ma.utexas.edu/blogs/mks/). I have tired to take a non-blaming approach and to emphasize that what have written about specific studies should not be taken as singling out the authors or the studies for criticism; that the problems I have found in those papers cited are common, and that I have chosen the particular papers simply because I have some prior experience with the field.
2. A website on “Common Mistakes in Using Statistics” (http://www.ma.utexas.edu/users/mks/statmistakes/StatisticsMistakes.html)
3. Slides from a continuing education course of the same name are online at http://www.ma.utexas.edu/users/mks/CommonMistakes2014/commonmistakeshome2014.html
Thanks for the links, Martha. I think I’m going to put together one last attempt to make my case with links and quotes, and these will be quite useful.
I’m trying to stress the importance of teaching issues with multiple testing in the basis applied statistics courses and I hope your ressources might prove useful for my argumentation.
I don’t know how much you know what you are doing and hence with which level of certainty you can assess that this is *wrong-borderline-fraud*. Even if you are 99% sure what they suggest is wrong I would recommand contacting asap other researchers of the field to ask them advices on the analysis approach. It is hard / impossible to really judge the soudness without all the information, and I think it would help you to reach a decision if you share with your peers.
Albatus, sadly “other researchers of the field” might *not* be the best to ask. There are some fields of research where there is close to nobody who really knows what he’s doing with statistical analysis and some failures are widespread. That muliiple testing is an issue which invalidates p-values is *not* something which is well known.
Thanks, Albatus. I don’t think it would qualify as actual fraud, but some things they’re suggesting make me uncomfortable. I’ve shared this now with many many peers (around my status) and they’ve almost unanimously shared similar experiences. So that gave me some validation.
[…] of resistance–bacteria win again Not-quite-fraud-but-close What your science teacher told you about sex chromosomes is wrong METATRANSCRIPTOME OF GUT […]
Shut up and do what your PI tells you. If the fraud comes out, keep you mouth shut and if necessary take the hit for the team.
When you have your own lab you will have your own chain of underlings to do your dirty work, so you will never have to sully your hands again, and the circle of life will be complete.
Not sure If I get your sarcasm right, but the issue is not restricted to highly authoritarian labs but even if you have a great working atmosphere and quite horizontal hierarchies in fact if not by law, junior researchers do still encounter the problem described.
Obviously I don’t know the details of the situation, but allow me to mention another potential side to this: My experience is that younger people like to find situations where they can make an ethical stand and that showcase how “sold” and rotten the old farts are. I know I felt like that several times during my studies.
With time I have come to see more things coming into play. Some cases were very clearly still wrong, but often, what I used to see as black/white actually had many shades of gray.
Again, I don’t know the situation, and there may very well have been something terribly wrong going on. But if I had a dime for each opinion I have had to change from the time I was a youngling to now, I would be able to retire. I take this change as a good thing: experience. It could also be that I am sold and rotten myself.
As for advice, talk to the people doing this. The one who mentioned the ethics side. How come they don’t see it as a problem? When asking don’t just point the judge finger, but truly ask. There may be a good lesson there too. If you ask properly it is unlikely they will get offended, and may even respect you more. Don’t burn this bridge.
Juan,
From the evidence that nicebrain has given (e.g., in his original post, “It can be daunting to be the only one on who thinks we shouldn’t drop 2 outliers to get our p-value from .08 to .01, or who thinks we shouldn’t go collect 5 more subjects to make it “work.” “), he is talking about things that are indeed questionable (to put it extremely politely) research practices.
The practices he is concerned about are practices that someone who doesn’t really understand statistics might think are OK, so from a young vs old perspective you would expect it to be the young person who thinks the questionable practices are OK, not the more senior people who are arguing for them.
In fact, some of the questionable research practices are ones that involve seeing things in black and white terms rather than seeing the shades of gray that are inherent in using statistics.
By the way, I’m not a young person — I’m retired and in my seventies.