When I talk about weighting survey data, I’m often met with facial expressions of the type ‘Huh?’ in the person I’m talking to. This article will attempt to uncover the mystery surrounding survey weightings by explaining what they are and when they can be useful.
The basics: Sample, Universe and Inference
Behind every survey or market study is the desire to find out certain parameters (demographic variables, perceptions,customer satisfaction, knowledge, etc.) for a given group of people. In statistics, we call this group the Universe. When surveys are used, the intention is to get answers to our questions so that we can characterize our universe.
Ideally, we would like answers from the entire universe, but this is rarely the case. Sometimes, there are too many people in the universe for it to be feasible to survey them all; other times, it will not be possible to reach certain groups in the universe. To keep things simple in this article, we will ignore the fact that there are almost always people who do not want to be surveyed.
If we cannot access the whole universe, then we survey a part of it. We call this part the Sample, and the aim is to Infer what the universe is like based on the data we obtain from our sample.
Looking at it like this, the definition of the sample will obviously be critical for ensuring that our characterization of the universe by inference is as close as possible to the real universe.
Errors due to Sample composition
The two basic points to consider when defining our sample are Representativeness (or proportionality) and Size:
Representativeness/Proportionality: The sample must be representative of the universe that we want to characterize. Although it seems obvious when we put it like this, this is often ignored when it comes to start surveying.
If we want to characterize certain preferences of the ‘Average Citizen’, we can’t just survey people from the three biggest cities, or only young people or women.
Size: In surveys, size matters… at least to some extent. If we only ask 10 people out of a group of 100, how reliable will our inference be? But what if we ask half of them? And if the group contains a million people?
Although this article will not cover how to define sample size, we will leave you with two ideas to think about: (1) The smaller the sample, the greater the percentage of people that we will need to ask, and (2) as of a certain size of universe, it is no longer necessary to increase the number of surveys to carry out.
Purpose of weightings
The use of weightings is related to the Representativeness/Proportionality part of the sample.
Ideally, we should give each person in the universe the same probability of being surveyed. Only then will we obtain a sample that truly represents the universe we want to characterize.
When this is not the case, either because it is not possible or because other considerations enter in the design of the sample, we can correct these deviations with weightings. To weight means to give each surveyed person a specific weight in the survey, which can vary from person to person. When we do not use weightings, what we are in fact doing is giving each person the same weight (usually 1).
Let’s see some examples. We’ll take the case of a multiplex cinema theater with several screens, each with different numbers of public (varying according to the capacity of the auditorium and the number of showings in them). Our goal is to find out the satisfaction of the customers of the multiplex. Let’s also suppose that the satisfaction surveys are done as the customers leave the cinema, so the people leaving the auditoriums will do them as they exit.
1-. Average score of the theater
If we analyze the satisfaction surveys without weightings, we obtain the average score from the respondents. However, the sample of respondents may not correspond to the universe of people who were in the auditoriums. This may be the case if certain movies end at the same time, when the surveys are only done in certain time slots, etc.
In this case, we can correct these deviations by using the number of people who have visited each auditorium as a weighting criterion. The weight to apply to each respondent who was in auditorium Y would be:
(Number of people who were in auditorium Y) / (Number of surveys answered by people from auditorium Y)
2-. Average score with quotas per auditorium
Other times, the sample is not representative because of its design. If, for example, we want a more in-depth analysis of a certain auditorium, then we are going to need a bigger sample. Budget restraints may deter us from streamlining the scenario by increasing the sample in proportion to the sample of the other auditoriums.
In this case, if we did not use weightings, the auditorium with the biggest sample would be overrepresented in the aggregate value. The correction to apply is the same as for the previous point, although the reason for doing it is different.
3-. Average auditorium score
Now let’s imagine that the management of the Multiplex is not concerned about the average visitor score and instead wants to know whether the metrics it has set as an indicator is the average of the auditorium scores.
In this case, we could forget about weightings, calculate the satisfaction per auditorium and then obtain the arithmetic mean… or apply the next weighting to each person who was in auditorium Y:
1 / (Number of surveys of people from auditorium Y)
4-. Average score based on people entering each auditorium
Lastly, let’s imagine that the management wants the scores to reflect the fact that visitors to auditoriums showing 3D movies have paid more (and are therefore more important for turnover) than people who go to see movie premieres and re-releases.
In this case, a possible criterion for the weightings of each person who was in auditorium Y would be:
(Ticket price for auditorium Y) x (Number of people who were in auditorium Y) / (Number of surveys on people from auditorium Y)
This would give us the average level of customer satisfaction per Euro of turnover.
This article has explained weighting as a correction mechanism for surveys. To do so, we described the relationship between Universe and Sample before outlining possible corrections that we can make by applying weightings.
More information on Electronic Surveys.
