The choice of units, target groups and cases for the impact assessment depends substantially on the design and/or the comparisons.
Ideally, all relevant units, target groups and cases are taken into account during data collection for the impact assessment. Such cases are called a total population survey. In practice, total population surveys are not always possible for specific reasons or due to the cost. There must therefore be a decision as to which cases should be taken into account for the impact assessment. With quantitative methods this is known as a sample.
The disadvantage of samples compared to total population surveys is that information is only ever collected for a portion of the interesting observations. As a result, it must be considered whether the results of the sampling hold for the whole of the unit of analysis. If this is not the case, then the sampling has not been carried out correctly and/or the cases have not been correctly selected.
Selection criteria for qualitative methods
It is not only for quantitative methods that the sample selection is an issue. One must also consider when using qualitative methods which cases or units should be studied. The number of units to be considered is generally automatically determined by the selection criteria. One would generally seek to consider one or two units per selection criterion. From a theoretical point of view, the number of cases or units is sufficient when the principle of saturation sets in. A selection or sample is said to be saturated when additional cases bring no new data and knowledge gains are saturated with the material already collected. One can use a three-step approach to work out the correct selection of cases:
- The first recommended step is to specify what facts are required from specific groups;
- The second step is to make sure that every possible form and feature of the unit of analysis have been considered during the selection process;
- The third step involves verifying again after data collection which constellations and features do not feature in the data already collected. That also means that the choice of the cases should not be carried out in one step as is the case with quantitative methods.
Selection criteria for quantitative methods
To avoid mistakes or biases due to choosing the wrong cases, it is necessary to clarify who or what is included in the population under investigation. Special attention is to be paid to taking adequate account of groups that are difficult to reach (e.g. geographically) and marginalised groups such as religious minorities or women during sampling. There is also a need to determine the size of the sample, the main criterion being how accurate the results need to be. The size of the population – at least for fairly large populations – has little influence on the minimum sample size (also cf. quantitative methods). Of course, in practice, the time available and the costs also play an important role.
There are various samples to choose from when using quantitative methods. Fundamentally, one must differentiate between “random sampling” and “non-random sampling”, which are put together according to specific criteria. If the sample is to be composed randomly, everyone in the population has the same likelihood of being “picked” for the sample. Some of the main selection methods are described in the next section.
- Simple random samples
Each unit in the population has the same likelihood of being picked (e.g. names drawn from a pot or every nth house).
- Layered random samples
The units of analysis are subdivided into groups (layers) according to a particular feature (e.g. villages, courses). Samples are then taken randomly from these sub-populations.
- Graduated random samples
First, the graduation criteria (e.g. Regions A-D) are determined. The population is then divided up and a random selection made (e.g. Regions B and D) and limited to a certain number of primary units, which are then investigated (e.g. 10 wells per region). The remaining sub-populations are ignored. From the randomly selected primary units (e.g. 10 wells), random sampling of the units with the feature (each of 20 households in a 15-minute radius) is now carried out. In each of the two regions are 200 housholds are surveyed, which are then grouped together into an overall sample.
- Quota samples
First, the elements of the population are divided into groups. The sample now has to be drawn so that the group relation in the sample looks as identical as possible to that in the population, in an attempt to imitate the desired population structure within the sample. The interviewers are also provided with guidelines as to which characteristics those to be interviewed should have. Yet it is up to the interviewer whom he or she chooses.
- Homogeneous and heterogeneous case selection
The observations are selected for the sample in such a way that they display as similar/ – or dissimilar – characteristics as possible. In case studies (which, by definition, do not constitute samples), two observations are for example often investigated with the most contrasting characteristics possible.
- Selection of typical cases
This involves selecting the observations for the samples that one knows – or assumes – to have typical, average or no extreme characteristics.
- Selection of critical cases
The study deliberately includes cases whose inclusion are known – or assumed – to be crucial to the study’s credibility or acceptance.