Author: Drew Roan
At Egalitarian Feminism, we strongly believe in using evidence based claims and using good quality research to back our assertions. We believe poor quality research can have the unfortunate effect of increasing peoples’ fears and creating hostility where it is not warranted. In striving for equality for all regardless of gender, we must make sure that any research we use is as fair and reasonable as possible. We are also strongly aware that sometimes, research can be misrepresented in the media and how this can affect what assumptions people make.
Therefore, Egalitarian Feminism will be starting an “EgaFem Analysis” series where research is scrutinized in detail and any findings are posted here for anyone to read.
But we want to do more than just analyse data. We want to help people learn what to look for, how to understand academic pieces and facilitate themselves in being able to interpret research and come to their own conclusions, not to solely rely on the reports of others.
Here are some points I have found to be highly useful when examining research papers. Hopefully, you will too:
Do you know where the claim has come from?
Statistics and claims are often thrown around during discussions online and in the media. Ask yourself “do you know where this statistic has come from”? Have you seen any source material for it? If you have no idea where the claim has come from, can you honestly say you understand the claim and can trust it?
As a rule of thumb, if a claim is made but no evidence can be or has been provided for it, it’s best to assume that it may not be accurate or fully trustworthy.
Don’t rely on the summary. Skim the introduction. Study the method and the results.
A trap many people fall into is to read the summary and assume it accurately reflects the research material in the study to such a level they often feel safe using it as a source. This is a bad idea. The summary at most only gives you a snap shot of what the researchers want you to take away from their study. It tells you nothing about their methods, results, conclusions, errors or disclaimers that might complicate the picture.
Introductions are usually filler text to explain the background and the necessity for the study in the first place. That said, it is always worth skimming through an introduction as it can yield a lot of information about the researchers’ point of view before conducting the study. Researchers who make bold or extraordinary claims, especially if those claims lack citations, may be more inclined to offer up incomplete or misleading research.
It cannot be stressed enough that reading the method and results are essential to understanding any research piece.
Always check the sample size and neutrality.
Sample size and bias can massively alter the quality of the results. A study with a small sample size might produce disproportionately large or small results that do not accurately reflect the wider population. Participants who have volunteered often offer a higher response bias than participants who are randomly sampled.
On rare occasions, participants in a sample may have been “coached” to elicit certain responses before participating in the research. Again, this can massively distort the results and potentially make the research extremely unreliable.
Who funded and conducted the research?
A commonly overlooked point to consider is the question of where the money for the research came from. It is not unheard of for researchers to be funded by specialist or advocacy groups, in order to conduct a study on a particular topic. Whilst it should be expected that researchers will always remain impartial, this is not always the case. You might find a study on the “harms of sugar in the brain” being funded by diabetes research groups, or a study on the benefits of alcohol being privately funded by alcohol manufacturers. This does not necessarily invalidate a piece of research, but it may warrant taking the findings with a pinch of salt.
Of equal interest is the question “who conducted the research"? Was it conducted by a survey group paid to find data on a particular topic, or from professors with a known history of possessing a particular ideological bias? All people are capable of being influenced by their own biases and this can be reflected in their research. Once again, it does not necessarily invalidate a piece of research, but it may give a reason to be cautious when repeating the findings.
Are you sure you understand their definitions?
Another common trap to fall into is to assume the definitions the researchers are using match up to legal or common use definitions. Equally, some terms that are frequently used in regular conversations may have a different meaning when applied in a certain context. Some examples might be:
- Confusing “wage” (pay per hour) with “salary” (pay over the course of a year).
- Mixing up “sexual offences” with “sexual assault”.
A researcher who may be using a non-legal definition of “discrimination” to include examples that may not match the legal definition.
Always make sure that you are clear on exactly what the researchers mean when using particular phrases.
If the research includes a survey, are those questions clear cut and straight forward to respond to?
Researchers have known for a long time that asking a question directly often does not yield particularly fruitful responses. Instead, questions with more ambiguous wording often yield a higher response rate, though these sometimes come with a risk of inaccurate reporting and artificially inflated response rates.
For example, consider the following two questions:
- Have you ever been raped whilst drunk?
- Have you ever had sexual intercourse when you did not want to whilst under the influence of alcohol?
On the surface, these two may appear to be the same in nature. In reality, the second version may yield a greater response yet, but may also include occasions that were not rape (such as drunken one-night stands, incidences of cheating and so forth).
Have the researchers used all the available data in their conclusions?
Always check to see if all relevant and available data has been used in the conclusions of the research. If data has been left out, why? Would it have affected the results? Was that data potentially relevant to the overall piece?
Whilst it may not necessarily invalidate the research in question, if researchers have chosen to leave out certain results or data sources, it may lead to others misinterpreting the information or claims being made which lack context. It may even lead the researchers to drawing a potentially unreasonable conclusion.
A little critical thinking is a great thing.
One of the best things you can do with any piece of research is to critically think about the findings, method and conclusions. Be mindful of potential flaws in the study. What would you have done differently? Would you have come to the same conclusions? Have you double checked their mathematics, to see if the numbers add up properly? Not all criticisms will be reasonable, but you might be surprised what issues with a research piece you can find if you examine it in more detail.
I hope you have found the above points useful in some way and that in turn, you feel more confident in reading research papers for yourself. My next piece will be a basic glossary of technical terms you may frequently encounter to help further deepen your understanding.