|
The conceptual framework and the formulation of hypotheses
|
- Evaluations are usually, but not always, based on a theoretical framework derived from a review of the literature and usually involve testable hypotheses.
- Hypotheses are often deductive (based on testable hypotheses derived from theory).
- Hypotheses are usually quantitative and can be evaluated with statistical significance tests.
- The framework often starts from the macro, rather than the micro level.
|
- While some evaluations define and test hypotheses, many do not.
- Many evaluations emphasize the uniqueness of each situation and the conceptual framework may be defined through a process of iteration, with the framework being continuously updated as new information is obtained.
- Hypotheses, if used, are often inductive (derived from information gathered during the course of the study).
|
|
Selection of subjects or units of analysis
|
- Involves random sampling so that findings can be generalized, and to permit statistical testing of differences between groups.
- Requires a sampling frame that lists all the members of the target population(s) to be studied.
- Selection methods are usually defined in advance, clearly documented, and unchanged during the study.
- Typically a fairly large sample is selected from which to collect a finite set of quantitative data.
|
- Choice of selection procedures varies according to the purpose of the study.
- Purposive sampling is used to collect the most useful and interesting data related to the purpose of the study.
- While this is not usually done for QUAL evaluations, sometimes for mixed-method approaches the sample may be selected using the same master sampling frame as for the QUANT component of the research. For example, a subsample of the villages in which samples of households (or other units) are selected for the QUANT survey may be selected for the QUAL analysis (although the types of data collection and the subjects, groups, or organizations to be studied in the QUAL analysis will usually be different).
- Usually a smaller number of people are interviewed in more depth.
|
|
Evaluation design
|
- Normally a quasi-experimental design is used. A randomly selected sample that represents the project participants, and possibly a control or comparison group, is interviewed at one or more points of time during the project
- Where possible, outcomes and impacts are estimated by comparing data collected before and after (and possibly during) the implementation of the project.
|
- The researcher(s) become immersed in the community over a long period of time.
- The effects of the program are studied through collecting information on the many different elements of the community and its economic, political, cultural, ecological, and psychological setting.
- Normally the evaluation does not try to establish a direct cause and effect or linear relationship.
|
|
Data collection and recording methods
|
- Data are usually recorded in structured questionnaires that are administered consistently throughout the study. There is extensive use of pre-coded, closed-ended questions
- The study mainly uses numerical values (integer variables) or closed-ended (ordinal or nominal) variables that can be subjected to statistical analysis.
- Observational checklists with pre-coded responses may be used.
|
- Interview protocols are the most common instrument, often semi-structured.
- The data collection instrument may be modified during the course of the study as understanding grows.
- Interview data are sometimes recorded verbatim (audiotape, videotape) and sometimes in written notes.
- Study may use analysis of existing documents. Textual data from documents are often highlighted in a copy of the original, which is kept as part of the data set.
- Study may use focus groups (usually fewer than 10 people) and meetings with larger community groups.
- Study may use participant and nonparticipant observation.
- Study may use photography.
- Several qualitative methods are used for multiple perspectives and triangulation.
|
|
Triangulation
|
- Consistency checks are built into questionnaires to provide independent estimates of key variables (e.g., data on income may be compared with data on expenditures).
- Direct observation (a QUAL technique) can be used as a consistency check on answers given by the respondent (e.g., information on income can be compared with evidence of the number and quality of consumer durables in evidence inside or outside the house).
- Information from earlier surveys with the same respondents is sometimes used as a consistency check on information given in a later survey.
- Secondary data (e.g., census data, national household surveys, information from government agencies) can be used to check estimates from the evaluation survey.
|
- Triangulation by observation: A monitor can observe a focus group or group meeting both to identify any potential bias resulting from how the session was conducted and also to provide an independent perspective (e.g., reporting on the interactions between group members, observing how certain people respond to the comments or behavior of others).
|
