- (a) a value for the 'test statistic' we need to compute, and
- (b) the probability for this value to appear
- 'reject' the null hypothesis and support an 'alternative' hypothesis that better interprets the data, or
- 'fail to reject' the null hypothesis in which case we stay with it, until -of course- new data may lead us to reject it (note that, in general, we avoid the expression "to accept the null hypothesis").
Suppose you are an educational technology researcher and you are interested in the following research topic:
- Does background music in a multimedia learning environment have a positive/negative impact on students who use this environment to learn?
Furthermore, suppose that you have no reason to believe that b.m. might have any impact on student learning whatsoever. Then you may state a null hypothesis as follows:
- "Background music has no impact whatsoever on students' learning"
The above is your "null hypothesis" (H0). As said, based on the calculated probability you may either:
- 'reject' and come up with an alternative (Ha) hypothesis that better explains what happens, for example "Background music has statistically significant impact on students' learning".
- 'fail to reject': and declare that your experiment failed to reject the null hypothesis.
'non-directional' hypothesis: does not predict the direction of impact (positive/negative).
- "Background music has no impact whatsoever on students' learning" is a non-directional hypothesis
'directional' hypothesis: it predicts the direction of impact (positive/negative).
- "Background music has a positive impact on students' learning" is a directional hypothesis
We usually state an alternative hypothesis as directional in which case the respective null hypothesis should be stated so that when (H0) and (Ha) are taken together they are mutually exclusive and exhaustive (they leave no other alternative)
- "Background music has not a positive impact on students' learning" is the null hypothesis for the above directional Ha
'Signal to noise' ratio: essentially all statistical hypothesis tests try to elicit order out of chaos by estimating a 'signal to noise' ratio. In the core of this conceptualization lies data variability.
In statistics we usually do measurements in sample groups. For example, we measure the performance of a group of students after study, the prices of a group of stocks at a certain moment, the answers that a group of users give in a user-satisfaction survey, etc. These group measurements exhibit a certain variability (not all students in a group get the same perfomance grade, grades vary). Also, group measurements vary across groups (the mean grade most probably is differnet across student groups studying with or without background music).
'Variability' refers to the property of data sets exhibiting variations (random or not) around some value of reference (for example, the mean value of the set). Considering this unavoidable variation we introduce the 'noise' and 'signal' concepts as follows:
- 'Noise': Data in a set vary from the reference value (the mean value) due to various uncontrollable factors that may affect any single measurement. Variability in this case is considered as a natural and unavoidable 'noise' that interferes in data measurement.
- 'Signal': when comparing different data sets, mean values will almost always appear different. However, if this difference is big enough this might signal that important variability exists due to some factor totally different from the factors producing the effect of the anticipated random noise.
So, now we have a way of estimating the 'noise to signal ratio':
- 'Noise' (denominator) is the expected variation within the groups. As explained, this represents the variability in measurement which appears in our sample groups (within groups variability)
- 'Signal' (nominator) is a possibly exceptional variability identified when comparing two or more groups. Thus, the signal is the variability between the two (or more) sample groups (between groups variability)
- when to 'reject': if we believe that this is quite a small probability then it's reasonable to assume that the measure (signal/noise ratio) did not get its exceptionally value by pure chance. Some other factor might had a significant impact during the measurement. Typically, in a situation like this, we 'reject' the original null hypothesis (H0) and look for an alternative Ha. In educational studies we are satisfied with a probability cutoff value of 0.05 (the a level) so the above p=0.035 value would lead us to this path of thinking.
- when to 'fail to reject': if, however, we still think that this is a high probability (as in critical medical studies where the 'a cutoff level' is usually set at 0.001) then we may continue arguing that we 'failed to reject' the H0.
- a) Defines a "signal/noise measure" and provides a means (algorithm) for its computation
- b) Uses a "measure-relevant distribution" to provide the probability that a measure lies within a range of values
The rest is the researcher's decision