Richard Marceau, Professor
École nationale d'administration publique
richard.marceau@enap.ca

From the outset, it should be stated that there is no consensus over how program evaluation is even to be defined. For reasons of practicality, this dictionary will offer only a barebones version of the term's meaning: the evaluation of a public program consists in establishing the social value of the said program.

In the public sector, this definition naturally encompasses other forms of government actions, be they more of a general or particular nature, such as policies, laws and regulations, and public goods and services.

According to its broadest meaning, evaluation consists in determining the value or worth of something. This determination of value necessarily implies the introduction of criteria, standards, points of reference (e.g., benchmarks) or merely values in the appraisal of the characteristics of the thing evaluated (Scriven, 1991, p. 5). In the public sector, evaluating government actions also brings into play the introduction of reference points for the purpose of establishing their worth. Logically, society serves as the first level of reference for the evaluation of government actions. As the concept of society is, in this context, understood as being inclusive, assessments of the worth of a government actions based solely on the interests of a special interest group, political party, elected government or a State will necessarily appear to be partial (Mishan and Quah, 2007, p. 5).

In order to understand changing trends in the field of evaluation, it is vital to distinguish between two schools of thought – namely, the social program evaluation school and the economic evaluation school.

The social program evaluation school has been identified as such from at least as far back as the work of Suchman (1967). It grew out of earlier research in education, social psychology and sociology (Shadish, Cook and Leviton, 1991, p. 21), fields which continue today to nurture this approach. Other disciplines, foremost of which is political science, have also made valuable contributions. Shadish, Cook and Leviton note references to the use of evaluation in respect of government personnel in China more than 4000 years ago; nevertheless, it is easier to discern the evolution of evaluation practice as it has unfolded over the last 200 years. All that being said, the genuine emergence of social program evaluation can be said to date to the 1960s, a period when the US government assumed a robust interventionist role in social policy. Shadish, Cook and Leviton ascribe the contemporary demand for evaluation to the concerns of American elected officials regarding issues of accountability, specifically in relation to the effects and the sound management of such mass intervention. The inability of administrators at that time to satisfy the need for evaluation purportedly prompted American university faculties to expand training and education in the social sciences on a very large scale. Beginning at that time, writes Alkin (2004, p. 13), the researchers Tyler and Campbell, et Scriven and Stufflebeam produced the pioneering work that led to the development of three distinct branches of program evaluation – namely, methods, valuing and use, respectively. Consequently, a variety of more limited definitions emerged that focused either on the methodological dimensions of evaluation (Rossi, Lipsey and Freeman, 2004), utilization (Patton, 1997), or the underlying democratic values (House and Howe, 1999). At the present time, strongly opposed epistemological positions in social science have further fragmented the notion of program evaluation into a multitude of evaluation approaches (Dubois and Marceau, 2005). Owen (2007), for example, has divided up evaluation into 20 different approaches under five main categories. As such, attempts at classification have failed to achieve consensus, with the results varying from one author to the next.

The economic school of program evaluation does not identify itself as a school per se since, clearly, its adherents feel considerable attachment to the discipline of economics, with which they share a set of postulates, concepts, theories and methodologies (Mishan and Quah, 2007). This discipline-centred practice is nevertheless considered to be a school of evaluation, as its contributions are clearly of a scientific nature and its field of application corresponds to government program evaluation; for this reason, moreover, it constitutes an additional alternative for anyone interested in evaluating a public program. Its best-known contribution remains the cost-benefit analysis of government programs and projects, which can be performed either “a priori” or “a posteriori.” Emerging after World War II, this school is based on a normative theory, namely that of welfare economics. Through the theory of market failure and government failure, the economic school thereafter contributed to evaluation's power to place the rationale for government-run programs on sounder footing. Likewise, through the theory of policy instruments, this school made it possible to forecast the consequences of public programs and thereby propose better specifications for evaluating impacts. Quantitative and statistical analysis has always been a hallmark of the evaluation of effects as practiced by the economic school. In that connection, the problems of selection bias associated with quasi-experimental studies were treated with requisite rigor and solutions were advanced, by Heckman and Vytacil (2000), for example. All these theoretical contributions have not been integrated in any particularly noticeable way. However, the economic school does not present the same degree of fragmentation as that characterizing the social program school.

It is important to note that these two schools of evaluation have rarely engaged in a clash with each other, as might well be prescribed by Lakatos' epistemology of research programs (1978). However, it should be pointed out that introductory works on the social program school often discuss cost-benefit analysis as a valuable evaluation option. Conversely, recent research in economics has been increasingly referring to experimental and quasi-experimental models which have, instead, originated with the social program school and which economists have been combining with advanced statistical models.

While the US government was the first to concern itself with program evaluation, the Government of Canada embraced the practice beginning in 1977, in particular in response to reports submitted by the Auditor General (Turgeon, 1994). The Canadian provinces, including Quebec, followed suit in the years after (Marceau, Simard and Otis, 1992; Marceau, 2004). While Europe was slow to adopt program evaluation (Perret, 2004), the obligation to evaluate structural funds proved to be a powerful driver for the dissemination of this practice across the continent. At this time, the gap between North American and European models and practices has been narrowing, in particular as a result of the questions they raise over relevance, efficiency and effectiveness (Migaud, 2004). The stated intention of numerous governments to implement management by results in public administrations is certain to prove conducive to the development of evaluation.

Notwithstanding the obvious differences in perspective surrounding the concept of program evaluation, all evaluators – practitioners and academics alike – articulate evaluation-related questions in one form or another that touch on: the program's rationale, objectives and targets; the nature of the actions pursued under the program (program theory); implementation, via the examination of inputs, activities and outputs; effects; the accomplishment of objectives; and the performance of resources, or, the program's effectiveness and efficiency (Marceau, Otis and Simard, 1992). From that point on, evaluation practices will vary according to the methods used to devise responses to these various questions. Practices also vary depending on the way in which stakeholders are induced to contribute to the evaluation process itself and on the role assumed by the evaluator.

For about 40 years now, the science and practice of evaluation has made considerable progress and experimented with numerous formulas. Today, evaluation researchers and practitioners would benefit from establishing a common conceptual apparatus – one shielded from political discourses that too easily turn to illusory terminological innovations – so as to lay the groundwork for the effective accumulation of knowledge. Not only does evaluation require considerable financial resources and a setting removed from the influence of special and political interests, it also needs to acquire a place within political institutions, particularly in terms of legislation. Such a place would serve the dual purpose of meeting the need for self-evaluation among public program managers and guaranteeing the independence of judgment. Finally, a fundamental role should be accorded to the products of evaluation in the democratic debate of today's societies, whose members are continually on the lookout for information on the quality of goods and services in both the public and private sectors.

Bibliography

Alkin, M. C. (2004). Evaluation Roots: Tracing Theorists' Views and Influences, Newbury Park, CA, Sage Publications.

Dubois, N. and R. Marceau (2005). “Un état des lieux théoriques de l'évaluation : une discipline à la remorque d'une révolution scientifique qui n'en finit pas,” The Canadian Journal of Program Evaluation, vol. 20, no. 1, pp. 1-36.

Guba, E. G. and Y. S. Lincoln (1989). Fourth Generation Evaluation, Thousand Oaks, CA, Sage Publications.

Heckman, J. J. and E. J. Vytacil (2000). “Econometric Evaluation of Social Programs: Part I,” in J. Heckman and E. Leamer (eds.), Handbook of Econometrics, vol. V, Amsterdam, North Holland Publishing Company.

Henard, F. (2000). Le développement des pratiques d'évaluation en France et en Europe, Société wallone d'évaluation et de prospective, www.wallonie-en-ligne.net/2000_evaluation-prospective/2000_henard_fabrice_france.htm (last retrieved in October 2010).

House, E. R. and K. R. Howe (1999). Values in Evaluation and Social Research, Thousand Oaks, CA, Sage Publications.

Lakatos, I. (1978). The Methodology of Scientific Research Programmes: Philosophical Papers Volume 1, Cambridge, Cambridge University Press.

Marceau, R. (2004). Évaluation des programmes publics : l'expérience québécoise, www.constructif.fr/Article_23_34_193/evaluation_des_programmes_publics_l_experience_quebecoise_br.html (last retrieved in October 2010).

Marceau, R., P. Simard and D. Otis (1992). “Program Evaluation in the Government of Quebec,” in J. Hudson, J. Mayne and R. Thomlison (eds.), Action-Oriented Evaluation in Organizations: Canadian Practices, Toronto, Wall & Emerson, pp. 48-63.

Marceau, R., D. Otis and P. Simard (1992). “La planification d'une évaluation de programme,” in R. Parenteau (ed.), Management Public, Sainte-Foy, QC, Presses de l'Université du Québec, pp. 445-480.

Migaud, D. (2004). Une révolution en douceur, www.constructif.fr/Article_23_34_188/Une_revolution_en_douceur_br.html (last retrieved in October 2010).

Mishan, E. J. and E. Quah (2007). Cost Benefit Analysis, 5th ed., New York, Routledge.

Owen, J. M. (2007). Program Evaluation: Forms and Approaches, New York, The Guilford Press, Sage Publications.

Patton, M. P. (1997). Utilization-Focused Evaluation: The New Century Text, Thousand Oaks, CA, Sage Publications.

Perret, B. (2004). Les lents progrès de l'évaluation de l'action publique, www.constructif.fr/Article_23_34_186/Les_lents_progres_de_l_evaluation_de_l_action_publique_br.html (last retrieved in October 2010).

Rossi, P. H., M. W. Lipsey and H. E. Freeman (2004). Evaluation: A Systematic Approach, Thousand Oaks, CA, Sage Publications.

Shadish, W. R., T. D. Cook and L. C. Leviton (1991). Foundations of Program Evaluation: Theories of Practice, Newbury Park, CA, Sage Publications.

Scriven, M. (1991). Evaluation Thesaurus, Newbury Park, CA, Sage Publications.

Suchman, E. (1967). Evaluative Research: Principles and Practice in Public Service and Social Action Programs, New York, Russell Sage Foundation.

Turgeon, J. (1994). “Évolution et leçons de l'évaluation de programme aux gouvernements du Canada et du Québec,” in G. Éthier (ed.), L'administration publique : diversité de ses problèmes/complexité de sa gestion, Sainte-Foy, QC, Presses de l'Université du Québec, pp. 83-108.

________________________

Reproduction
Reproduction in whole or part of the definitions contained in the Encyclopedic Dictionary of Public Administration is authorized, provided the source is acknowledged.

How to cite
Marceau, R. (2012). “Program Evaluation,” in L. Côté and J.-F. Savard (eds.), Encyclopedic Dictionary of Public Administration, [online], www.dictionnaire.enap.ca

Legal deposit
Library and Archives Canada, 2012 | ISBN 978-2-923008-70-7 (Online)