|
<< Click to Display Table of Contents >> Navigation: Reference Manual > Tables > Significance testing |
See Significance formats for the relevant format options.
See Column identifiers for details on how to identify columns for comparison which is done by adding a letter in parentheses at the end of the column label.
Question: Is this figure significant?
Answer: Compared to what?
When discussing 95% or 99% significance it is important to bear in mind that a figure can only be significantly different when compared with some other figure or a known fixed figure. Companion has many ways of marking a figure as 95% significant but what does this mean?
Let us assume that we have a product rating mean score for two subsets of the data. Men have a mean score of 1.56 and Women have a mean of 1.34. When compared, Men are flagged as having a significantly higher mean at the 95% level than Women. What this means is that there is only a 1 in 20 chance that a difference this large (0.22) would be obtained if the two subsets had been drawn from the same universe. Put another way we can be 95% confident that there is a real difference between Men and Women when rating the product.
The use of the word “significance” in this context is unfortunate here because it is not the same as the normal use of the word. On a rating scale of 1 to 5 we can have a mean score difference of 0.1 that is “significant” and a difference of 2.0 that is not “significant”. The word “confidence” is much more descriptive.
So when a figure is marked at 99% it means that it is very unlikely (1 in 100) that there is no difference between the two samples being looked at. The reverse is not true; we cannot say that if a figure is not marked then there is no difference - it may just be that the sample sizes are not large enough for us to be confident. The more interviews we do the more differences will be found.
The levels used are set by formats SLA and SLB. For more than 2 levels you can also use SLC and SLD.
There are two types of significance testing:
Using format SIG individual cells in table are compared with other cells in the same row or in same row in the total column.
The figures tested are the percentages (or proportions) of records in the cells based on the total row (vertical percentages).
Any "significant" differences are indicated with markers below the cell or alongside the cell in TXT output or on request, see Spreadsheet options.
Mean scores (averages) are compared with mean scores in other columns or in the total column.
Any "significant" differences are indicated with markers below the cell or alongside the cell in TXT output or on request, see Spreadsheet options.
You can turn off the marking using formats SLA101/SLB101.
When the rows of a table are the difference between two paired variables the "expected" value is zero (if there is no difference between them), then format MSE will gived the paired t test value and format PRO2 can be used to see the percentage "significance" of that value.
Figures can be tested against figures in the total column using SHG0 (default), SHG11, or SHG12.
As an example we could have six columns (total plus 5 responses):
Total, Male, Female, Young, Middle, Old
This tests whether any particular column is different from the rest of the data in the row being tested. In this way Companion will highlight any “interesting” columns for further investigation. In our example, assuming everyone has answered both the Gender and Age questions:
Males are tested against Females
Females are tested against Males
Young are tested against Middle and Old together,
Middle are tested against Young and Old together
Old are tested against Young and Middle together.
The actual calculation is done by first subtracting the column being tested from the total column to get "the rest" and then testing the column against "the rest". It is important to do it this way to avoid overlapping samples.
This method of testing uses the total sample in every test and is therefore more likely to detect differences than other types of comparison unless the sample sizes are very large.
This method will only work on tables with a total column.
Using default format settings, Cells are marked with plus or minus signs for 95% and two plus or minus signs for 99%. Plus means higher than "the rest" and minus means lower than "the rest".
Cells in the spreadsheet output will be coloured green for higher and red for lower.
Figures can be tested against other nearby figures using SHG1, SHG2, SHG11, or SHG12.
Usually the columns that each column is tested against is the other columns under the same header.
Individual pairs of columns are compared; the range of the comparisons depends upon format SHG.
See Column identifiers for details on how to identify columns for comparison which is done by adding a letter in parentheses at the end of the column label.
Other columns not under the same header can also be tested by enclosing the cadditional column identifier(s) between + signs, see Column identifiers for more information.
Columns are marked if they are higher (percentage or mean) than the figure in the column they are being compared to.
NOTE: comparing against total column shows higher and lower; this method only shows higher.
NOTE: testing overlapping samples will not mark differences that do not exist, but it might miss real differences.
If all the identifiers used are lower case then lower case markers represent SLA significance and upper case letters represent SLB significance. You can turn off SLB testing by setting SLB101.
If a respondent rates two or more products, you can produce a banked table with the products as the breakdown and use column identifiers to compare them.
To get a paired comparison test you can subtract the score for one product from the score for the other. For example, with scores 1 to 5 the paired score will be between –4 and +4. This paired score can then be tabulated and format MSE will give a t test value comparing the mean score with the expected fixed value of 0.0. If the expected value for a mean is zero, and if format MSE is >1.96 this is 95% and >2.57 is 99%.
Data can be output to other software for further tests:
Export to Excel by exporting a CSV file
Export to SPSS