Article Design: Something Every Social Psychologist Learns But Few Are Taught
I am a fourth-year doctoral student in social psychology. Later this year I will propose my dissertation prospectus and, hopefully, become a doctoral candidate. If you’ll forgive me for going a little niche, I’d like to talk about one aspect of the hidden curriculum that’s been on my mind as I prepare to enter the black hole of candidacy. I’m talking about article design. Before you laugh and tell me to switch careers, I’ll try to convince us both that I’m hitting on an actual problem here.
First, I’m not talking about designing individual studies. When I say “article design,” I’m talking about research design at the level of the paper: putting together sequences of studies that build on each other in a compelling way to test one or more hypotheses. This is where it gets niche, because nowhere else in psychology have I seen such an emphasis on cramming experimentation, replication, and extension into a single article as I have in social psychology. In my relatively short time in this field, I’ve seen articles in social psychology grow longer and more sophisticated, with some recent ones containing ten or more studies.
This wasn’t unheard of before the replication crisis - Asch’s classic 1946 paper on impression formation contained ten experiments. But the number of studies in social psychological articles has certainly been rising since the replication crisis, especially for top journals like Journal of Personality and Social Psychology (JPSP). Every social psychologist is aware of this trend, but it was empirically documented in a 2019 study by Kai Sassenberg. In 2011, the average paper in JPSP (the top social psych journal) contained around 4 studies, while the average second-tier journal article (JESP, PSPB) contained closer to 2. JPSP had reached an average of 6 studies by 2016 and the second-tier journals were averaging right around 4 studies in 2018. The replication crisis has clearly kicked social psychology into gear in the last ten years.
While it’s true that the open science movement has encouraged more complex papers across psychology broadly, I would contrast evolving norms in social psychology with those of other areas in the field. My sense is that, whereas researchers in other areas of psychology focus more on exploring many research questions in fewer studies, social psychologists tend to iterate on a few hypotheses in chains of compact experiments. This seems natural given that other areas of psychology have a healthier balance between correlational and experimental designs. For example, a health psychologist might draw on a large dataset to examine how psychological factors relate to immune system function. An entire paper might consist of a single study exploring correlations among predictors, regressions testing several psychological variables as predictors of immune system function, and a set of additional analyses exploring moderation by participant demographics like race and gender. The result would be a comprehensive examination of a research question and several of its offshoots. Most graduate students in psychology could jump right into a project like this after completing their research design coursework.
Although a social psychologist might report a similar number of statistical analyses within a paper, these might be distributed across a larger number of studies, each of which is designed to test the same hypothesis. I can use my own research as an example here. If I wanted to test the hypothesis that judgments about cultural appropriation are influenced by perceptions of the appropriated group’s status, I might begin by testing this hypothesis correlationally, then conduct a series of experiments designed to establish the causal role of group status, then explore some moderators of the effect, and end by testing potential mediators. Some experiments in the sequence might be exact replications of earlier ones. But part of the work’s scientific value comes from demonstrating that my hypothesis survives attacks from many angles and outcompetes alternative explanations. So realistically, I would develop a few manipulations and sets of stimuli for the paper. The final study package could easily contain five to ten studies, each with a different design. This becomes ten times harder when there isn’t much existing work on your topic to guide you, because designing each new study feels like inventing the wheel.
Crucially, this social psychological model of “study packages” involves a kind of rhetorical skill that is not taught in most programs that I am familiar with. Social psychology has notoriously been branded as theatrical and flashy, and that stereotype applies perfectly to this model of publication. Every paper is a story about how the authors’ underdog hypothesis trumped all of its competitors, but only under the conditions that follow from their pet theory, and there’s even a neat answer to the question of, “but what’s the mechanism?” The more studies one can chain together to bolster this story, the better. But this way of constructing publications is simply not the norm in areas of psychology that are more concerned with application and external validity. Depending on who designs the methodological curriculum in your department, you may or may not have structured opportunities to learn the skill of article design. You would almost need a creative writing class.
To be clear, I think that higher standards for publication are good for psychological science, including its replicability and generalizability. But I’m concerned that graduate students in social psychology are not receiving the formal training that would help them move with the times. I mean it when I say that article building has been the single largest stumbling block for me in grad school. Early on I would ask myself, is there a formula, or a natural progression that I’m supposed to pick up? What makes for a good Study 1, 2, 3, or beyond? If I have some data already, what makes for a natural next study? What sequences of studies can I leverage to accomplish different scientific goals? I was required to take graduate courses on study design, so why was there no course on article design?
The existence of answers to these questions was clearly communicated to me in journals and rejection letters, and of course by my advisor, but no one ever sat me down and said, “Listen, a good formula for an article is a correlational study, a few 2 x 2 experiments, and some mediation.” I learned this gradually, through hours of reading and tidbits of wisdom I gleaned from conversations with my advisor. But I could tell that I was failing to grasp basic strategies and heuristics that everyone in the field was using to guide their work. My first years of grad school were cluttered with dead-end studies that went nowhere because I neither designed them with a larger package in mind nor knew how to build one after I collected the data. I started to worry that I would leave grad school with nothing to show for myself. Later I learned that some institutions actually do offer formal opportunities to learn article design, either through coursework or in professional development seminars. Recently I was mentored by a post-doc who graciously shared his notes from such a seminar. It was the first time I had ever seen a collection of study package heuristics boiled down into a few bullet points, and I was in my third year of graduate school. To this day that document serves as my research bible.
It feels vulnerable to admit having struggled with something that feels like it should be intuitive, but I don’t think I’m the only one. Graduate students in most areas of psychology receive formal training through their coursework that prepares them for the model of publication in which they will engage as mature researchers, and the same should apply to students in social psychology. But it’s hard because social psychology, more than any other area, feels like an art form. The same way creative writers are taught the key beats of a romance novel, or the way composers are trained in the essential features of a concerto, students of social psychology need structured opportunities to learn the elements of a compelling empirical article.
About the Author
Shiri Spitz Siddiqi is a doctoral student at UC Irvine, where she studies how people make moral judgments about divisive social issues. In her free time, she enjoys thrifting outrageous retro clothing, trying out new recipes, and knitting.
