Thoughts from a nobody on the whole 'replication bullying' thing

May 25, 2014

Hello. My name is Christian, and I am going to weigh in on the 'replication bullying' thing that's been buzzing around this year's APS meeting. It's important to know right now that I'm more or less a nobody - I'm a postdoc in a cognitive neuroscience lab and am certainly not a big figure in my field. But that's kind of the point of this post, because as a nobody I feel like I can write a blog post that:

So I have something to say, and I'm not too sure how well it's going to be received, but I think it's important, so here goes:

I have always had a hard time taking psychology seriously, despite the fact that I have a BSc. and (gasp) an MA in it. One factor in my choosing to get my PhD in neuroscience instead of psychology was so that "real scientists" and engineers would stop rolling their eyes at me at parties when I told them what I did. My thesis project would have been the same but the courses offered for each program were different - the neuro program allowed me to learn about cell & molecular bio and take more interesting stats and compsci courses, which was also a big motivator.

So no, I didn't switch from psych to neuro because of the criticisms from the rest of science. I'm a big boy and I can take my licks. Rather, I switched because the criticisms were completely legitimate. Modern psychology has disappointingly low standards relative to other fields. Biologists, chemists, and physicists are always expected to reproduce their experiments. This difference isn't just because of publishing standards - problems in psychology begin way before anyone writes their first journal article. Undergraduate chemists and biologists spend a whole lot of time in either lab-based or stats-based courses as they come up in the academic game, and that kind of education is crucial. I started off my undergrad in toxicology and we were in lab about half the time and enrolled in stats since year one. Not the same in psych - our research methods courses were as much about writing in APA style than they were about running an experiment, and you only had to take that class once. ONCE. One lab-based course for a degree that claims to be a science. Tragic. Of course I loved this as a drunken undergrad because it was so easy to coast through the program, but by the time I got to TAing my attitude shifted. It became obvious why all the other STEM kids used my field as a punching bag - we don't learn how to do good research until relatively late in the game, if ever.

This is a problem. I do a lot of fMRI work and it's critically important that the tasks I have people do while in the magnet have strong and repeatable effects - scanner time is really, really expensive. Right now I don't trust most of what's being published because of things like p-hacking and just straight-up falsification, so I get by on piloting any behavioral paradigms beforehand and also basing my work off the output of friends and colleagues who I do trust. But this is limiting and inefficient, and I'd like to see it change.

So this one goes out to the nobodys like me, especially the new grads who are just coming up in the game, because at the end of the day, we're the ones who get screwed by questionable research practices, and we need to change that. Change takes time and the high-profile profs who are pushing back against higher standards for psychological research are more or less a lost cause, but I have hope for the future. Science is iterative, and the way to go from being a nobody to a somebody is going to depend on your ability to build theories based on previous research. This is particularly true early in your career - if you're doing a one or two-year degree MA or an honours thesis you do not have time to screw around with stuff that won't turn out (although if you are a multitasker, these things can make good side projects). If you want to build or extend on a finding, make sure it's been replicated, or do a replication study yourself (this is particularly good for an undergrad thesis). Read up on questionable research practices and learn to recognize the signs - that PhD student down the hall from you who's 6 years into a 5-year degree and 'finally got his experiment to work' is probably doing it right now. Talk to other people at your level at poster sessions - you'd be surprised how many people wind up doing something they know was wrong because they were told to by a superior. If this is happening to you, remember that you're probably not alone. You might not be able to do much about it, but at least you might be able to find some people to commiserate with.

Psychology is clearly going through some growing pains right now and it's going to be awkward for all of us, particularly those of us who are still trying to find our places in it. While there are going to be some, shall we say, adolescent, reactions to things like replication initiatives, ultimately raising our standards is an important part of pushing our field into a more mature state so we can learn about the things that fascinate us. In fact, consider yourself lucky to be working at such a time when things appear to be getting better. I know I do.

If you'd like to comment on this piece, feel free to harass me on twitter.