Annex 5: How QUANT and QUAL Approaches Complement Each Other at Different Stages of an Evaluation

A.   Broadening the conceptual framework

  • Combining theories from different disciplines:
    • Integrating anthropological concepts of culture in economic analysis
    • Combining economic concepts of poverty with sociological/anthropological concepts of vulnerability
    • Integrating support network theory into the economic analysis of poverty
  • Exploratory QUAL studies can help define framework.

 

B.  Combining generalizability with depth and context

  • Random selection of subjects ensures representativeness and generalizability.
  • Case studies, focus groups, etc. can help increase understanding of the characteristics of the different groups selected in the sample.

 

C.  QUAL methods help increase access to difficult to reach groups

  • PRA, focus groups, case studies, etc. can be effective ways to reach women, ethnic minorities and other vulnerable groups.
  • Direct observation can provide information on groups difficult to interview. For example, informal sector and illegal economic activities.

 

D.  QUAL methods help analyze processes

  • Questionnaires are not effective for studying group processes or interaction between people and public agencies. Observation, focus groups and informal conversations are more effective.
  • Observation is useful for studying the organization and effectiveness of work groups and community organizations.

 

E.   QUANT methods can control for underlying structural factors

  • Focus groups, informal interviews, etc. often involve a biased sample (for example people with strong views) and can give misleading impressions about community attitudes or causes of behavior.
  • Sampling and statistical analysis can avoid coming to misleading conclusions from these methods.
  • Propensity scores and multivariate analysis can statistically control for differences between project and control groups.
 

Example:

  • Meetings with women may suggest gender biases in hiring practices of local firms.
  • However, using statistical analysis to control for years of education or experience may show there are no differences in hiring policies for workers with comparable qualifications.
 

Example:

  • Participants who volunteer to attend a focus group may be strongly in favor or opposed to a certain project.
  • A rapid sample survey may show that most community residents have different views.

 

F.   Consistency checks (triangulation)

  • Two or more independent estimates obtained for key indicators by combining surveys, observation, focus groups, secondary data, etc.
  • Estimates are "triangulated."
  • If estimates are consistent there is greater confidence in findings.
  • If estimates are inconsistent, follow-up is required to determine the reasons and to make adjustments to estimates.
 

Example:  Direct observation may identify inconsistencies in interview responses

  • A family may say they are poor but observation shows they have new furniture, good clothes, etc.
  • A woman may say she has no source of income, but an early morning visit may show she operates an illegal beer brewing business.

 

G.  Interpreting findings

Statistical analysis frequently includes unexpected or interesting findings which cannot be explained through the statistics. Rapid follow-up visits may help explain the findings.

Example:

  • A QUANT survey of community water management in Indonesia found that with only one exception all village water supply was managed by women.
  • Follow-up visits found that in the one exceptional village women managed a very profitable dairy farming business – so men were willing to manage water to allow women time to produce and sell dairy produce.

 

Source: Brown (2000)