choose an option
- Update: What is Statistical Significance?
- How to calculate statistical significance
- A 5-step model for testing the significance of null hypotheses
- 1. State the null and alternative hypotheses
- 2. Set a threshold for statistical significance
- 3. Get a sample and collect data
- 4. Determine if your data is statistically significant
- 5. Interpret the results
Part II: How to Determine If My Survey Is Statistically Significant
Update: What is Statistical Significance?
What does it mean when search results "statistically significant?”
Simply put, statistical significance is a way for researchers to quantify the likelihood that their results are due to chance. Statistically significant results are those where the researcher is confident that the results are real and reliable because there is little chance of getting the results just by chance.
Statistical significance tests involve several abstract concepts. So let's try to break things down with an example of how you might perform a statistical significance test.
How to calculate statistical significance
Statistical significance tests in psychological research
Everyone knows what it's like to feel regret. But few people care enough about the feeling of regret to design experiments that study other people's behavior.avoidthe feeling. Jane Risen and Thomas Gilovich are two of those people.
In a 2008 publication titled "Why people are reluctant to tempt fateRisen and Gilovich explored how people's fear of feeling remorseful can lead them to engage in irrational behavior.
In his first experiment, people read about Jon - a young man who had just applied to graduate school. Jon has been dying to visit Stanford, and knowing this, Jon's mother sends him a Stanford T-shirt. In the scenario, people read that Jon either a) stuffed the shirt in a drawer or b) tempted fate by wearing the shirtbeforehear from Stanford.
After reading the scenario, study participants were asked: What is the probability that Jon will be admitted to Stanford (0 –not likelybis 10 –Probably)?
As the authors expected, people who read Jon wearing the shirt were less likely to believe he would be accepted to Stanford (M= 5,19,SD= 1.35) than those who read that he put his shirt in the drawer (M= 6,13,SD= 1.02). However, the question is whether the difference between groups is large enough to be statistically significant.
Authors began to discoverNull hypothesis significance test.
A 5-step model for testing the significance of null hypotheses
1. State the null and alternative hypotheses
The first step in testing for statistical significance is to accept a null hypothesis. The null hypothesis is skeptical of the researchers' data and assumes that what the researcher is studying does not really exist.
In the case of Risen and Gilovich's experiment, the null hypothesis was that temptation—wearing the T-shirt before receiving news from Stanford—would not affect people's beliefs about whether Jon would be accepted to Stanford.
The alternative hypothesis, on the other hand, is that there is an impact of the tragedy on people's belief that Jon will be accepted at Stanford.
Two other points about the null and alternative hypotheses deserve mention.
First, both hypotheses are often more useful in theory than in practice. Researchers rarely write down their null and alternative hypotheses. Rather, hypotheses are implied or accepted as part of the statistical analysis.
Second, the null and alternative hypotheses for the population the researcher is studying do not apply to the sample. Although researchers rely on the samples they collect for data, the hypotheses they test and the models they rely on are assumed to occur at the population level (that is, among people outside the sample).
2. Set a threshold for statistical significance
The second step in performing a significance test is to define a significance threshold. Traditionally, the gold standard in academic research has been a level of significance (orpvalue) of 0.05. This means that researchers are only willing to accept their results as statistically significant if there is less than a 5% chance that they will get the same results if the null hypothesis is true (that is, if there really is no effect).
Although the 0.05 significance level is common in academia, it has no value in itself. In fact, for over 30 years,two respected statisticiansemphasized how arbitrary the 0.05 significance level is, writing: "...surely God loves 0.06 almost as much as 0.05".0,005.
Setting a threshold for statistical significance should be done in the context of your study objectives. If you are conducting a study for companies, the significance level you choose should depend on how you intend to use the data. How important is the decision you are trying to make? What are the consequences of a wrong decision? How valuable is the planned course of action if you are right? The answers to these questions could lead you to adopt a very conservative or more liberal significance level, perhaps even above 0.10 or 0.20.
Your level of importance should balance your desire to be confident in your results with the practical impact of the decision you want to make.
3. Get a sample and collect data
The third step is data collection.
As it is often impractical to collect data from everyone in the population of interest, researchers are doing surveyscollect a sample🇧🇷 Sample data is used to make inferences about the population.
For example, in Risen and Gilovich's study, they were interested in the general feeling of regret. Your interest group can reasonably be described as adults in the US or people from western industrialized countries. However, the sample collected consisted of 62 undergraduate students from Cornell University.
For many researchers, online data collection is the most efficient method today. CloudResearch can help you collect data from large and diverse groups of people quickly and easily. learn howUnsere Sampling-Toolscan help you find the sample you need and ensure you have enough power for your statistical tests today.
4. Determine if your data is statistically significant
After collecting data, the next step is to perform statistical tests. There are many different tests researchers can perform depending on the type of data they have. In our example, Risen and Gilovich performed a simple twogroupt-Test. Other common analyzes include linear regression, chi-square tests, ANOVA, and Mann-Whitney U tests.
All statistical tests follow a formula. This onet-The test formula below converts the difference between two groups into a proportion:
Once researchers have a right, they compare it to a probability distribution like the one below. If thetIf the statistic exceeds the significance level, researchers reject the null hypothesis and conclude that its effect is statistically significant.
For example, in the study by Risen and Gilovich, researchers rejected the null hypothesis and concluded that the difference between the two groups was statistically significant due to the likelihood of receiving atA value equal to or greater than 3.01 was 1% (t(60) = 3,01,p= 0,01,d= 0,78).
Although statistical tests can often be calculated by hand, most researchers use an analysis program. Software packages such as Tableau, SPSS and Stata are commercially available and facilitate these calculations. But even without such software, researchers can often use programs such as Microsoft Excel to facilitate statistical analyses.
5. Interpret the results
When the analyzes are complete and you have a statistically significant finding, what happens? How should the data be interpreted?
Perhaps the first step in interpretation is to remember that your results don't "prove" anything, they just provide support for your ideas. After all, even if the relationship you're studying doesn't exist (the null hypothesis is true), you'd get results below the 0.05 significance level five times out of a hundred if you ran the same study over and over again.
The second step in interpreting your data goes beyond that.pValue and evaluation of the practical importance of the result. In some studies, the practical implications become obvious, as the results can be interpreted as differences in dollars spent, visits to your website or new contacts made. In other studies, however, the practical significance is more difficult to determine.
In the study by Risen and Gilovich, for example, it is not immediately clear what the practical significance of the finding is. The study suggests that people believe negative consequences can befall someone who tempts fate, but it doesn't say much more. In situations like this, researchers often turn to effect size estimation to understand the strength of their results. Effect sizes are a quantitative measure of the strength or extent of a finding. The effect size in the Risen and Gilovich study was adof 0.78, which is quite large.
To make informed decisions, it is important that the survey be statistically significant. CloudResearch can help you with your research by providing quick access to millions of participants. With our platform, you can be confident that you will find enough participants to conduct well-researched studies. And if you don't have the resources or aren't sure how to design or analyze your study, we're here to help. We have experienced social scientists who can help you plan and manage any study you wish to undertake. Once the data is collected, we can perform advanced statistical analysis and interpret the data for you, saving your staff time and resources.