Annex 1: Strengths and weaknesses of quantitative evaluation designs

Strengths

Weaknesses

  • Study findings can be generalized to the population about which information is required.
  • Samples of individuals, communities, or organizations can be selected to ensure that the results will be representative of the population studied.
  • Structural factors that determine how inequalities (such as gender inequalities) are produced can be analyzed.
  • QUANT estimates can be obtained of the magnitude and distribution of impacts.
  • QUANT estimates can be obtained of the costs and benefits of interventions.
  • Clear documentation can be provided regarding the content and application of the survey instruments so that other researchers can assess the validity of the findings.
  • Standardized approaches permit the study to be replicated in different areas or over time with the production of comparable findings.
  • It is possible to control for the effects of extraneous variables that might result in misleading interpretations of causality (although this can be challenging in the natural settings of evaluations).

 

 

 

 

 

 

 

  • Many kinds of information are difficult to obtain through structured data collection instruments, particularly on sensitive topics such as domestic violence or income.
  • Many groups such as sex workers, drug users, illegal immigrants, squatters and ethnic minorities are always difficult to reach, but the problems are often greater for QUANT data collection methods.
  • Self-reported information obtained from questionnaires may be inaccurate or incomplete.
  • There is often no information on contextual factors to help interpret the results or to explain variations in behavior between households with similar economic and demographic characteristics.
  • The administration of a structured questionnaire creates an unnatural situation that may alienate respondents.
  • Studies are expensive and time-consuming, and even the preliminary results are usually not available for a long period of time.
  • Research methods are inflexible because the instruments cannot be modified once the study begins.
  • Reduction of data to numbers results in lost information.
  • The correlations produced (e.g., between costs and benefits, gender, and access to services or benefits) may mask or ignore underlying causes or realities.
  • Untested variables may account for program impacts.
  • Errors in the hypotheses tested may yield misimpressions of program quality or influential factors.
  • Errors in the selection of procedures for determining statistical significance can result in erroneous findings regarding impact.