HOME

TheInfoList




Evaluation is a
system A system is a group of Interaction, interacting or interrelated elements that act according to a set of rules to form a unified whole. A system, surrounded and influenced by its environment, is described by its boundaries, structure and purp ...

system
atic determination of a subject's merit, worth and significance, using criteria governed by a set of
standards Standard may refer to: Flags * Colours, standards and guidons * Standard (flag), a type of flag used for personal identification Norm, convention or requirement * Standard (metrology), an object that bears a defined relationship to a unit of ...
. It can assist an organization, program, design, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in
decision-making In psychology, decision-making (also spelled decision making and decisionmaking) is regarded as the Cognition, cognitive process resulting in the selection of a belief or a course of action among several possible alternative options, it could be ...
; or to ascertain the degree of achievement or value in regard to the aim and
objectives
objectives
and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining
insight The Interior Exploration using Seismic Investigations, Geodesy and Heat Transport (''InSight'') mission is a robotic Robotics is an interdisciplinarity, interdisciplinary field that integrates computer science and engineering. Robotics inv ...

insight
into prior or existing initiatives, is to enable
reflectionReflection or reflexion may refer to: Philosophy * Self-reflection Science * Reflection (physics), a common wave phenomenon ** Specular reflection, reflection from a smooth surface *** Mirror image, a reflection in a mirror or in water ** Signal r ...
and assist in the identification of future change. Evaluation is often used to characterize and appraise subjects of interest in a wide range of human enterprises, including the
arts The arts refers to the theory, human application and physical expression of creativity Creativity is a phenomenon whereby something somehow new and somehow valuable is formed. The created item may be intangible (such as an idea, a scienti ...

arts
,
criminal justice 350px, United States criminal justice system flowchart Criminal justice is the delivery of justice Justice, one of the four cardinal virtues, by Vitruvio Alberi, 1589–1590. Fresco, corner of the vault, studiolo of the Virgin of Mercy, Ma ...
, foundations,
non-profit organization A nonprofit organization (NPO), also known as a non-business entity, not-for-profit organization, or nonprofit institution, is a legal entity organized and operated for a collective, public or social benefit, in contrast with an entity that o ...
s,
government A government is the system or group of people governing an organized community, generally a state State may refer to: Arts, entertainment, and media Literature * ''State Magazine'', a monthly magazine published by the U.S. Departmen ...

government
,
health care Healthcare is the maintenance or improvement of health Health, according to the , is "a state of complete physical, and social and not merely the absence of and ".. (2006)''Constitution of the World Health Organization''– ''Basic Docume ...

health care
, and other human services. It is long term and done at the end of a period of time.


Definition

Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. So evaluation can be formative, that is taking place during the development of a concept or proposal, project or organization, with the intention of improving the value or effectiveness of the proposal, project, or organisation. It can also be summative, drawing lessons from a completed action or project or an organisation at a later point in time or circumstance. Evaluation is inherently a theoretically informed approach (whether explicitly or not), and consequently any particular definition of evaluation would have been tailored to its contextthe theory, needs, purpose, and methodology of the evaluation process itself. Having said this, evaluation has been defined as: * A systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement, or outcomes of a program. It is a resource-intensive process, frequently requiring resources, such as, evaluate expertise, labor, time, and a sizable budget * "The critical assessment, in as objective a manner as possible, of the degree to which a service or its component parts fulfills stated goals" (St Leger and Wordsworth-Bell). The focus of this definition is on attaining objective knowledge, and scientifically or quantitatively measuring predetermined and external concepts. * "A study designed to assist some audience to assess an object's merit and worth" (Stufflebeam). In this definition the focus is on facts as well as value laden judgments of the programs outcomes and worth.


Purpose

The main purpose of a program evaluation can be to "determine the quality of a program by formulating a judgment" Marthe Hurteau, Sylvain Houle, Stéphanie Mongiat (2009). An alternative view is that "projects, evaluators, and other stakeholders (including funders) will all have potentially different ideas about how best to evaluate a project since each may have a different definition of 'merit'. The core of the problem is thus about defining what is of value." From this perspective, evaluation "is a contested term", as "evaluators" use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research. There are two function considering to the evaluation purpose Formative Evaluations provide the information on the improving a product or a process Summative Evaluations provide information of short-term effectiveness or long-term impact to deciding the adoption of a product or process. Not all evaluations serve the same purpose some evaluations serve a monitoring function rather than focusing solely on measurable program outcomes or evaluation findings and a full list of types of evaluations would be difficult to compile. This is because evaluation is not part of a unified theoretical framework, drawing on a number of disciplines, which include
management Management (or managing) is the administration of an organization An organization, or organisation (Commonwealth English The use of the English language English is a West Germanic languages, West Germanic language first spok ...

management
and organisational theory,
policy analysis Policy analysis is a technique used in public administration Public administration is the implementation of government policy Public policy is a course of action created and/or enacted, typically by a government A government is th ...
,
education Education is the process of facilitating learning, or the acquisition of knowledge, skills, value (ethics), values, morals, beliefs, habits, and personal development. Educational methods include teaching, training, storytelling, discussion ...

education
,
sociology Sociology is a social science Social science is the branch The branches and leaves of a tree. A branch ( or , ) or tree branch (sometimes referred to in botany Botany, also called , plant biology or phytology, is the scie ...
,
social anthropology Social anthropology is the study of patterns of behaviour in human societies and cultures. It is the dominant constituent of anthropology Anthropology is the scientific study of human Humans (''Homo sapiens'') are the most populous a ...
, and
social change Social change involves alteration of the social order The term social order can be used in two senses: In the first sense, it refers to a particular system of social structures and institution Institutions, according to Samuel P. Huntingt ...
.


Discussion

However, the strict adherence to a set of methodological assumptions may make the field of evaluation more acceptable to a mainstream audience but this adherence will work towards preventing evaluators from developing new strategies for dealing with the myriad problems that programs face. It is claimed that only a minority of evaluation reports are used by the evaluand (client) (Datta, 2006). One justification of this is that "when evaluation findings are challenged or utilization has failed, it was because stakeholders and clients found the inferences weak or the warrants unconvincing" (Fournier and Smith, 1993). Some reasons for this situation may be the failure of the evaluator to establish a set of shared aims with the evaluand, or creating overly ambitious aims, as well as failing to compromise and incorporate the cultural differences of individuals and programs within the evaluation aims and process. None of these problems are due to a lack of a definition of evaluation but are rather due to evaluators attempting to impose predisposed notions and definitions of evaluations on clients. The central reason for the poor utilization of evaluations is arguably due to the lack of tailoring of evaluations to suit the needs of the client, due to a predefined idea (or definition) of what an evaluation is rather than what the client needs are (House, 1980). The development of a standard methodology for evaluation will require arriving at applicable ways of asking and stating the results of questions about ethics such as agent-principal, privacy, stakeholder definition, limited liability; and could-the-money-be-spent-more-wisely issues.


Standards

Depending on the topic of interest, there are professional groups that review the quality and
rigor Rigour (British English British English (BrE) is the standard dialect A standard language (also standard variety, standard dialect, and standard) is a language variety that has undergone substantial codification of grammar and usage ...
of evaluation processes. Evaluating programs and projects, regarding their value and impact within the context they are implemented, can be
ethically Ethics or moral philosophy is a branch of philosophy Philosophy (from , ) is the study of general and fundamental questions, such as those about reason, Metaphysics, existence, Epistemology, knowledge, Ethics, values, Philosophy of min ...

ethically
challenging. Evaluators may encounter complex, culturally specific systems resistant to external evaluation. Furthermore, the project organization or other stakeholders may be invested in a particular evaluation outcome. Finally, evaluators themselves may encounter " conflict of interest (COI)" issues, or experience interference or pressure to present findings that support a particular assessment. General professional
codes of conduct A code of conduct is a set of rules outlining the norms Norm, the Norm or NORM may refer to: In academic disciplines * Norm (geology), an estimate of the idealised mineral content of a rock * Norm (philosophy), a standard in normative ethics ...
, as determined by the employing organization, usually cover three broad aspects of behavioral standards, and include inter-
collegial Collegiality is the relationship between ''colleagues''. Colleague is taken to mean a fellow member of the same profession, a group of colleagues united in a common purpose, and used in proper names, such as Electoral College An electoral college ...
relations (such as respect for
diversity Diversity, diversify, or diverse may refer to: Business *Diversity (business) The "business case for diversity" stems from the progression of the models of diversity within the workplace since the 1960s. The original model for diversity was situ ...
and
privacy Privacy (, ) is the ability of an individual or group to seclude themselves or information about themselves, and thereby express themselves selectively. When something is private to a person, it usually means that something is inherently special ...

privacy
), operational issues (due competence, documentation accuracy and appropriate use of resources), and conflicts of interest (
nepotism Nepotism is a form of favoritism which is granted to relatives and friends in various fields, including business, politics, entertainment, sports, fitness, religion, and other activities. The term originated with the assignment of nephews to im ...
, accepting gifts and other kinds of favoritism). However, specific guidelines particular to the evaluator's role that can be utilized in the management of unique ethical challenges are required. The
Joint Committee on Standards for Educational Evaluation The Joint Committee on Standards for Educational Evaluation is an American/Canadian Canadians (french: Canadiens) are people identified with the country of Canada Canada is a country in the northern part of North America. Its Provin ...
has developed standards for program, personnel, and student evaluation. The Joint Committee standards are broken into four sections: Utility, Feasibility, Propriety, and Accuracy. Various European institutions have also prepared their own standards, more or less related to those produced by the Joint Committee. They provide guidelines about basing value judgments on systematic inquiry, evaluator competence and integrity, respect for people, and regard for the general and public welfare. The American Evaluation Association has created a set of Guiding
Principle A principle is a proposition or value that is a guide for behavior or evaluation. In law, it is a Legal rule, rule that has to be or usually is to be followed. It can be desirably followed, or it can be an inevitable consequence of something, suc ...

Principle
s for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows: * Systematic
Inquiry An inquiry (also spelled as enquiry in British English British English (BrE) is the standard dialect of the English language English is a West Germanic languages, West Germanic language first spoken in History of Anglo-Saxon E ...

Inquiry
: evaluators conduct systematic,
data Data (; ) are individual facts A fact is something that is truth, true. The usual test for a statement of fact is verifiability—that is whether it can be demonstrated to correspond to experience. Standard reference works are often used ...

data
-based inquiries about whatever is being evaluated. This requires quality data collection, including a defensible choice of indicators, which lends credibility to findings. Findings are credible when they are demonstrably evidence-based, reliable and valid. This also pertains to the choice of
methodology Methodology is the study of research methods, or, more formally, "'a contextual framework' for research, a coherent and logical scheme based on views, beliefs, and values, that guides the choices researchers
r other users R, or r, is the eighteenth letter of the modern English alphabet and the ISO basic Latin alphabet The ISO basic Latin alphabet is a Latin-script alphabet A Latin-script alphabet (Latin alphabet or Roman alphabet) is an alphabet that ...
make". It compris ...
employed, such that it is consistent with the aims of the evaluation and provides dependable data. Furthermore, utility of findings is critical such that the information obtained by evaluation is comprehensive and timely, and thus serves to provide maximal benefit and use to
stakeholders Stakeholder may refer to: *Stakeholder (corporate), a group, corporate, organization, member, or system that affects or can be affected by an organization's actions *Project stakeholder, a person, group, or organization with an interest in a projec ...
. * Competence: evaluators provide competent performance to
stakeholders Stakeholder may refer to: *Stakeholder (corporate), a group, corporate, organization, member, or system that affects or can be affected by an organization's actions *Project stakeholder, a person, group, or organization with an interest in a projec ...
. This requires that evaluation teams comprise an appropriate combination of competencies, such that varied and appropriate expertise is available for the evaluation process, and that evaluators work within their scope of capability. *
Integrity Integrity is the practice of being honest and showing a consistent and uncompromising adherence to strong moral and ethical principles and values. In ethics Ethics or moral philosophy is a branch of philosophy Philosophy (from , ) is the ...

Integrity
/
Honesty ''Diogenes Searching for an Honest Man'', attributed to J. H. W. Tischbein () Honesty or truthfulness is a facet Facets () are flat faces on geometric shapes. The organization of naturally occurring facets was key to early developments in crys ...
: evaluators ensure the honesty and integrity of the entire evaluation process. A key element of this principle is freedom from bias in evaluation and this is underscored by three principles: impartiality, independence, and transparency.
Independence is attained through ensuring independence of judgment is upheld such that evaluation conclusions are not influenced or pressured by another party, and avoidance of conflict of interest, such that the evaluator does not have a stake in a particular conclusion. Conflict of interest is at issue particularly where funding of evaluations is provided by particular bodies with a stake in conclusions of the evaluation, and this is seen as potentially compromising the independence of the evaluator. Whilst it is acknowledged that evaluators may be familiar with agencies or projects that they are required to evaluate, independence requires that they not have been involved in the planning or implementation of the project. A declaration of interest should be made where any benefits or association with project are stated. Independence of judgment is required to be maintained against any pressures brought to bear on evaluators, for example, by project funders wishing to modify evaluations such that the project appears more effective than findings can verify.
Impartiality pertains to findings being a fair and thorough assessment of strengths and weaknesses of a project or program. This requires taking due input from all stakeholders involved and findings presented without bias and with a transparent, proportionate, and persuasive link between findings and recommendations. Thus evaluators are required to delimit their findings to evidence. A mechanism to ensure impartiality is external and internal review. Such review is required of significant (determined in terms of cost or sensitivity) evaluations. The review is based on quality of work and the degree to which a demonstrable link is provided between findings and recommendations.
Transparency requires that stakeholders are aware of the reason for the evaluation, the criteria by which evaluation occurs and the purposes to which the findings will be applied. Access to the evaluation document should be facilitated through findings being easily readable, with clear explanations of evaluation methodologies, approaches, sources of information, and costs incurred.
*
Respect Respect, also called esteem, is a positive feeling or action shown towards someone or something considered important or held in high esteem or regard. It conveys a sense of admiration for good or valuable qualities. It is also the process of ho ...

Respect
for People: Evaluators respect the
security Security is freedom from, or resilience against, potential Potential generally refers to a currently unrealized ability. The term is used in a wide variety of fields, from physics Physics is the natural science that studies matter, its El ...

security
,
dignity Dignity is the right Rights are law, legal, social, or ethics, ethical principles of Liberty, freedom or entitlement; that is, rights are the fundamental normative rules about what is allowed of people or owed to people according to some legal s ...

dignity
and
self-worth Self-esteem is an individual's subjective evaluation of their own worth. Self-esteem encompasses beliefs about oneself (for example, "I am unloved", "I am worthy") as well as emotional states, such as triumph, despair, pride, and shame. Smith and ...
of the respondents, program
participants Participation or Participant may refer to: Politics *Participation (decision making), mechanisms for people to participate in social decisions *Civic participation, engagement by the citizens in government *e-participation, citizen participation ...
, clients, and other stakeholders with whom they
interact Interact may refer to: * Interaction Interaction is a kind of action that occurs as two or more objects have an effect upon one another. The idea of a two-way effect is essential in the concept of interaction, as opposed to a one-way causal ef ...
.This is particularly pertinent with regards to those who will be impacted upon by the evaluation findings. Protection of people includes ensuring informed consent from those involved in the evaluation, upholding confidentiality, and ensuring that the identity of those who may provide sensitive information towards the program evaluation is protected. Evaluators are ethically required to respect the customs and beliefs of those who are impacted upon by the evaluation or program activities. Examples of how such respect is demonstrated is through respecting local customs e.g. dress codes, respecting peoples privacy, and minimizing demands on others' time. Where stakeholders wish to place objections to evaluation findings, such a process should be facilitated through the local office of the evaluation organization, and procedures for lodging complaints or queries should be accessible and clear. * for General and
Public Welfare Welfare is a type of government support intended to ensure that members of a society can meet basic human needs such as food and shelter. Social security may either be synonymous with welfare, or refer specifically to ''social insurance'' pro ...
: Evaluators articulate and take into account the diversity of
interest In finance Finance is the study of financial institutions, financial markets and how they operate within the financial system. It is concerned with the creation and management of money and investments. Savers and investors have money availa ...

interest
s and
values In ethics Ethics or moral philosophy is a branch of philosophy Philosophy (from , ) is the study of general and fundamental questions, such as those about Metaphysics, existence, reason, Epistemology, knowledge, Ethics, values, Philosophy ...
that may be related to the general and public welfare. Access to evaluation documents by the wider public should be facilitated such that discussion and feedback is enabled. Furthermore, the international organizations such as the I.M.F. and the World Bank have independent evaluation functions. The various funds, programmes, and agencies of the United Nations has a mix of independent, semi-independent and self-evaluation functions, which have organized themselves as a system-wide UN Evaluation Group (UNEG), that works together to strengthen the function, and to establish UN norms and standards for evaluation. There is also an evaluation group within the OECD-DAC, which endeavors to improve development evaluation standards. The independent evaluation units of the major multinational development banks (MDBs) have also created the Evaluation Cooperation Group to strengthen the use of evaluation for greater MDB effectiveness and accountability, share lessons from MDB evaluations, and promote evaluation harmonization and collaboration.


Perspectives

The word "evaluation" has various
connotations A connotation is a commonly understood cultural Culture () is an umbrella term which encompasses the social behavior and Norm (social), norms found in human Society, societies, as well as the knowledge, beliefs, arts, laws, Social norm, custom ...
for different people, raising issues related to this process that include; what type of evaluation should be conducted; why there should be an evaluation process and how the evaluation is integrated into a program, for the purpose of gaining greater knowledge and awareness? There are also various factors inherent in the evaluation process, for example; to critically examine influences within a program that involve the gathering and analyzing of relative information about a program.
Michael Quinn Patton Michael Quinn Patton (born 1945) is an independent organizational development and program evaluation consultant, and former president of the American Evaluation Association. He is the founder and director of Utilization-Focused Evaluation. After ...
motivated the concept that the evaluation procedure should be directed towards: * Activities * Characteristics * Outcomes * The making of judgments on a program * Improving its effectiveness, * Informed programming decisions Founded on another perspective of evaluation by Thomson and Hoffman in 2003, it is possible for a situation to be encountered, in which the process could not be considered advisable; for instance, in the event of a program being unpredictable, or unsound. This would include it lacking a consistent routine; or the concerned parties unable to reach an agreement regarding the purpose of the program. In addition, an influencer, or manager, refusing to incorporate relevant, important central issues within the evaluation


Approaches

There exist several conceptually distinct ways of thinking about, designing, and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way.


Classification of approaches

Two classifications of evaluation approaches by HouseHouse, E. R. (1978). Assumptions underlying evaluation models. ''Educational Researcher''. 7(3), 4-12. and Stufflebeam and WebsterStufflebeam, D. L., & Webster, W. J. (1980)
"An analysis of alternative approaches to evaluation"
''
Educational Evaluation and Policy Analysis ''Educational Evaluation and Policy Analysis'' is a peer-reviewed academic journal covering all aspects of educational policy analysis. It was established in 1979 and is published by SAGE Publications on behalf of the American Educational Research A ...
''. 2(3), 5-19.
can be combined into a manageable number of approaches in terms of their unique and important underlying principles. House considers all major evaluation approaches to be based on a common
ideology An ideology () is a set of belief A belief is an attitude Attitude may refer to: Philosophy and psychology * Attitude (psychology) In psychology Psychology is the science of mind and behavior. Psychology includes the study of ...
entitled
liberal democracy Liberal democracy, also referred to as Western democracy, is the combination of a liberal Liberal or liberalism may refer to: Politics *a supporter of liberalism, a political and moral philosophy **Liberalism by country *an adherent of a L ...
. Important principles of this ideology include freedom of choice, the uniqueness of the
individual An individual is that which exists as a distinct entity An entity is something that exists as itself, as a subject or as an object, actually or potentially, concretely or abstractly, physically or not. It need not be of material existence. In ...
and
empirical Empirical evidence for a proposition In logic and linguistics, a proposition is the meaning of a declarative sentence (linguistics), sentence. In philosophy, "Meaning (philosophy), meaning" is understood to be a non-linguistic entity which is s ...
inquiry grounded in objectivity. He also contends that they are all based on
subjectivist Subjectivism is the doctrine that "our own mental activity is the only unquestionable fact of our experience", instead of shared or communal, and that there is no external or objective truth. The success of this position is historically attributed ...
ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is
utilitarian Utilitarianism is a family of normative Normative generally means relating to an evaluative standard. Normativity is the phenomenon in human societies of designating some actions or outcomes as good or desirable or permissible and others as ba ...
, in which " the good" is determined by what maximizes a single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is
intuitionist In the philosophy of mathematics, intuitionism, or neointuitionism (opposed to preintuitionism), is an approach where mathematics Mathematics (from Ancient Greek, Greek: ) includes the study of such topics as quantity (number theory), mathe ...
/ pluralist, in which no single interpretation of "the good" is assumed and such interpretations need not be explicitly stated nor justified. These ethical positions have corresponding
philosophies Philosophy, Philosophical school of thought, schools of thought and philosophical movements. Absurdism - Actual idealism - Actualism - Advaita Vedanta -` Aesthetic Realism - Aesthetics - African philosophy - Afrocentrism - Agential realism - A ...

philosophies
for obtaining
knowledge Knowledge is a familiarity or awareness, of someone or something, such as facts A fact is something that is truth, true. The usual test for a statement of fact is verifiability—that is whether it can be demonstrated to correspond to e ...
. The objectivist epistemology is associated with the utilitarian ethic; in general, it is used to acquire knowledge that can be externally verified (intersubjective agreement) through publicly exposed methods and
data Data (; ) are individual facts A fact is something that is truth, true. The usual test for a statement of fact is verifiability—that is whether it can be demonstrated to correspond to experience. Standard reference works are often used ...

data
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic and is used to acquire new knowledge based on existing personal knowledge, as well as experiences that are (explicit) or are not (tacit) available for public inspection. House then divides each epistemological approach into two main political perspectives. Firstly, approaches can take an elite perspective, focusing on the interests of managers and professionals; or they also can take a mass perspective, focusing on
consumer A consumer is a person or a group who intends to order, orders, or uses purchased goods, products, or Service (economics), services primarily for personal, social, family, household and similar needs, not directly related to entrepreneurial or bu ...
s and
participatory Citizen Participation or Public Participation in social science refers to different mechanisms for the Public consultation, public to express opinions – and ideally exert influence – regarding political, economic, management or other social d ...
approaches. Stufflebeam and Webster place approaches into one of three groups, according to their orientation toward the role of
values In ethics Ethics or moral philosophy is a branch of philosophy Philosophy (from , ) is the study of general and fundamental questions, such as those about Metaphysics, existence, reason, Epistemology, knowledge, Ethics, values, Philosophy ...
and ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually is and might be—they call this
pseudo- The prefix A prefix is an which is placed before the of a word. Adding it to the beginning of one word changes it into another word. For example, when the prefix ''un-'' is added to the word ''happy'', it creates the word ''unhappy''. Pa ...
evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object—they call this
quasi Quasi (phonetics 'kwa - zee') is an American indie rock Indie rock is a genre of rock music that originated in the United States and United Kingdom in the 1970s. Originally used to describe independent record label An independent record ...
-evaluation. The values orientation includes approaches primarily intended to determine the value of an object—they call this true evaluation. When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation. Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—
experiment An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...

experiment
al research, management
information systems An information system (IS) is a formal, sociotechnical Sociotechnical systems (STS) in organizational development is an approach to complex organizational work design that recognizes the interaction between people and technology in wiktionary:Wo ...
, testing programs, objectives-based studies, and
content analysis Content analysis is the study of document A document is a writing, written, drawing, drawn, presented, or memorialized representation of thought, often the manifestation of nonfiction, non-fictional, as well as fictional, content. The word ori ...
—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and
policy Policy is a deliberate system of guideline A guideline is a statement by which to determine a course of action. A guideline aims to streamline particular processes according to a set routine or sound practice. Guidelines may be issued by a ...

policy
studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and
connoisseur A connoisseur ( French traditional, pre-1835, spelling of , from Middle-French , then meaning 'to be acquainted with' or 'to know somebody/something') is a person who has a great deal of knowledge Knowledge is a familiarity, awareness, or ...
studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.


Summary of approaches

The following table is used to summarize each approach in terms of four
attributes Attribute may refer to: * Attribute (philosophy) In logic Logic (from Ancient Greek, Greek: grc, wikt:λογική, λογική, label=none, lit=possessed of reason, intellectual, dialectical, argumentative, translit=logikḗ)Also relate ...

attributes
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired
outcome Outcome may refer to: * Outcome (probability), the result of an experiment in probability theory * Outcome (game theory), the result of players' decisions in game theory * ''The Outcome'', a 2005 Spanish film * An outcome measure (or clinical endpo ...
for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together.


Pseudo-evaluation

Politically Politics (from , ) is the set of activities that are associated with making decisions in groups, or other forms of power relations between individuals, such as the distribution of resource A resource is a source or supply from which a ben ...

Politically
controlled and
public relations Public relations (PR) is the practice of managing and disseminating information from an individual or an organization An organization, or organisation (English in the Commonwealth of Nations, Commonwealth English; American and British ...
studies are based on an objectivist epistemology from an
elite In Political philosophy, political and sociology, sociological theory, the elite (French ''élite'', from Latin ''eligere'', to select or to sort out) are a small group of powerful people who hold a economic inequality, disproportionate amount o ...

elite
perspective. Although both of these approaches seek to misrepresent value interpretations about an object, they function differently from each other.
Information Information is processed, organised and structured data Data (; ) are individual facts A fact is something that is truth, true. The usual test for a statement of fact is verifiability—that is whether it can be demonstrated to c ...

Information
obtained through politically controlled studies is released or withheld to meet the special interests of the holder, whereas public relations information creates a positive image of an object regardless of the actual situation. Despite the application of both studies in
real Real may refer to: * Reality Reality is the sum or aggregate of all that is real or existent within a system, as opposed to that which is only Object of the mind, imaginary. The term is also used to refer to the ontological status of things, ind ...

real
scenarios, neither of these approaches is acceptable evaluation practice.


Objectivist, elite, quasi-evaluation

As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal. *
Experimental research An experiment is a procedure carried out to support, refute, or validate a hypothesis A hypothesis (plural hypotheses) is a proposed explanation for a phenomenon. For a hypothesis to be a scientific hypothesis, the scientific method ...

Experimental research
is the best approach for determining
causal relationships Causality (also referred to as causation, or cause and effect) is influence by which one event, process, state or object (a ''cause'') contributes to the production of another event, process, state or object (an ''effect'') where the cause is ...
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized
methodology Methodology is the study of research methods, or, more formally, "'a contextual framework' for research, a coherent and logical scheme based on views, beliefs, and values, that guides the choices researchers
r other users R, or r, is the eighteenth letter of the modern English alphabet and the ISO basic Latin alphabet The ISO basic Latin alphabet is a Latin-script alphabet A Latin-script alphabet (Latin alphabet or Roman alphabet) is an alphabet that ...
make". It compris ...
may not be sufficiently responsive to the dynamically changing needs of most human service programs. *
Management information system A management information system (MIS) is an information system An information system (IS) is a formal, sociotechnical Sociotechnical systems (STS) in organizational development is an approach to complex organizational work design that recognizes ...
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily
quantifiable Quantity is a property that can exist as a Counting, multitude or Magnitude (mathematics), magnitude, which illustrate discontinuity (mathematics), discontinuity and continuum (theory), continuity. Quantities can be compared in terms of "more", " ...
data usually available at regular intervals. * Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected. * Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object. *
Content analysis Content analysis is the study of document A document is a written Writing is a medium of human communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, meaning among Subject (ph ...
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.


Objectivist, mass, quasi-evaluation

*
Accountability Accountability, in terms of ethics and governance, is equated with answerability, blameworthiness, legal liability, liability, and the expectation of account-giving. As in an aspect of governance, it has been central to discussions related to probl ...

Accountability
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the
quality Quality may refer to: Concepts *Quality (business), the ''non-inferiority'' or ''superiority'' of something *Quality (philosophy), an attribute or a property *Quality (physics), in response theory *Energy quality, used in various science disciplin ...
of
products Product may refer to: Business * Product (business), an item that serves as a solution to a specific consumer problem. * Product (project management), a deliverable or set of deliverables that contribute to a business solution Mathematics * Produc ...
and
services Service may refer to: Activities :''(See the Religion section for religious activities)'' * Administrative service, a required part of the workload of Faculty (academic staff), university faculty * Civil service, the body of employees of a governm ...
. However, this approach quickly can turn practitioners and
consumer A consumer is a person or a group who intends to order, orders, or uses purchased goods, products, or Service (economics), services primarily for personal, social, family, household and similar needs, not directly related to entrepreneurial or bu ...
s into adversaries when implemented in a heavy-handed fashion.


Objectivist, elite, true evaluation

* -oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close
collaboration Collaboration (from Latin Latin (, or , ) is a classical language A classical language is a language A language is a structured system of communication Communication (from Latin ''communicare'', meaning "to share" or "to be ...

collaboration
between an evaluator and decision-maker, allowing it to be susceptible to
corruption Corruption is a form of dishonesty Dishonesty is to act without honesty ''Diogenes Searching for an Honest Man'', attributed to J. H. W. Tischbein () Honesty or truthfulness is a facet Facets () are flat faces on geometric shapes. The org ...
and
bias Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded Open-mindedness is receptiveness to new ideas. Open-mindedness relates to the way in which people approach the views and kn ...

bias
. *
Policy studies Policy studies is a subdisicipline of political science that includes the analysis of the process of policymaking (the policy process) and the contents of policy (policy analysis). Policy analysis includes substantive area research (such as health o ...
provide general guidance and direction on broad issues by identifying and assessing potential
costs and benefits In production, research Research is "creativity, creative and systematic work undertaken to increase the stock of knowledge". It involves the collection, organization, and analysis of information to increase understanding of a topic or is ...
of competing
policies Policy is a deliberate system of guideline A guideline is a statement by which to determine a course of action. A guideline aims to streamline particular processes according to a set routine or sound practice. Guidelines may be issued by an ...

policies
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.


Objectivist, mass, true evaluation

*
Consumer A consumer is a person or a group who intends to order, orders, or uses purchased goods, products, or Service (economics), services primarily for personal, social, family, household and similar needs, not directly related to entrepreneurial or bu ...

Consumer
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.


Subjectivist, elite, true evaluation

*
Accreditation Accreditation is a third-party attestation related to a conformity assessment body (such as certification body Professional certification, trade certification, or professional designation, often called simply ''certification'' or ''qualification' ...
/
certification Certification is the formal attestation or confirmation of certain characteristics of an object, person, or organization. This confirmation is often, but not always, provided by some form of external review, education, assessment, or audit ...
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects. *
Connoisseur A connoisseur (French language, French Reforms of French orthography, traditional, pre-1835, spelling of , from Middle-French , then meaning 'to be acquainted with' or 'to know somebody/something') is a person who has a great deal of knowledge ...
studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.


Subject, mass, true evaluation

* The adversary approach focuses on drawing out the pros and cons of
controversial Controversy is a state of prolonged public dispute or debate, usually concerning a matter of conflicting opinion An opinion is a judgement Judgement (or US spelling judgment) is also known as ''adjudication'' which means the evaluation of ...
issues through quasi-
legal proceedingsLegal proceeding is an activity that seeks to invoke the power of a tribunal in order to enforce a law. Although the term may be defined more broadly or more narrowly as circumstances require, it has been noted that " e term ''legal proceedings'' inc ...
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later
cooperation Cooperation (written as co-operation in British English British English (BrE) is the standard dialect A standard language (also standard variety, standard dialect, and standard) is a language variety that has undergone substantial ...
and heighten animosities between contesting parties if "winners" and "losers" emerge.


Client-centered

* Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external
credibility Credibility comprises the objective Objective may refer to: * Objective (optics), an element in a camera or microscope * ''The Objective'', a 2008 science fiction horror film * Objective pronoun, a personal pronoun that is used as a grammatica ...
and a favorable bias toward those who participated in the study.


Methods and techniques

Evaluation is methodologically diverse. Methods may be qualitative or , and include
case studies A case study involves an up-close, in-depth, and detailed examination of a particular case or cases, within a real-world context. For example, case studies in medicine may focus on an individual patient or ailment; case studies in business might co ...
,
survey research In Human subject research, research of human subjects, a survey is a list of questions aimed for extracting specific data from a particular group of people. Surveys may be conducted by phone, mail, via the internet, and also at street corners or in ...
,
statistical analysis Statistical inference is the process of using data analysis to infer properties of an underlying probability distribution, distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical ...
, model building, and many more such as: *
Accelerated aging Accelerated aging is testing that uses aggravated conditions of heat, humidity, oxygen, sunlight, vibration, etc. to speed up the normal aging processes of items. It is used to help determine the long-term effects of expected levels of stress withi ...
*
Action research Action research is a philosophy and methodology of research generally applied in the social sciences. It seeks transformative change through the simultaneous process of taking action and doing research, which are linked together by critical refle ...
*
Advanced product quality planning Advanced product quality planning (APQP) is a framework of procedures and techniques used to develop products Product may refer to: Business * Product (business), an item that serves as a solution to a specific consumer problem. * Product (projec ...
*
Alternative assessment Alternative assessment is also known under various other terms, including: * authentic assessment * integrative assessment * holistic assessment In education, "alternative assessment" is in direct contrast to what is known as "traditional test (as ...
* Appreciative Inquiry * Educational assessment, Assessment * Axiomatic design * Benchmarking * Case study * Change management * Clinical trial * Cohort study * Competitor analysis * Consensus decision-making * Consensus-seeking decision-making *
Content analysis Content analysis is the study of document A document is a written Writing is a medium of human communication Communication (from Latin ''communicare'', meaning "to share") is the act of developing Semantics, meaning among Subject (ph ...
* Conversation analysis * Cost-benefit analysis * Data mining * Delphi Technique * Design Focused Evaluation * Discourse analysis * Educational accreditation * Electronic portfolio * Environmental scanning * Ethnography * Experiment * Experimental techniques * Factor analysis * Factorial experiment * Feasibility study * Field experiment * Fixtureless in-circuit test * Focus group * Force field analysis * Game theory * Goal-free evaluation * Grade (education), Grading * Historical method *
Inquiry An inquiry (also spelled as enquiry in British English British English (BrE) is the standard dialect of the English language English is a West Germanic languages, West Germanic language first spoken in History of Anglo-Saxon E ...

Inquiry
* Interview * Iterative design * Marketing research * Meta-analysis * Performance metric, Metrics * Most significant change technique * Multivariate statistics * Naturalistic observation * Observational techniques * Opinion polling * Organizational learning * Outcome mapping * Outcomes theory * Participant observation * Participatory impact pathways analysis * Policy analysis * Post occupancy evaluation * Process improvement * Project management * Qualitative research * Quality audit * Quality circle * Quality control * Quality management * Quality management system * Quantitative research * Questionnaire * Questionnaire construction * Review * Root cause analysis * Rubric (academic), Rubrics * Sampling (statistics), Sampling * Self-assessment * Six Sigma * Standardized testing * Statistical process control * Statistical survey * Statistics * Strategic planning * Structured interviewing * Systems theory * Test (student assessment), Student testing * Theory of change * Total quality management * Triangulation (social science), Triangulation * Wizard of Oz experiment


See also

* Monitoring and Evaluation is a process used by governments, international organizations and NGOs to assess ongoing or past activities * Educational assessment, Assessment is the process of gathering and analyzing specific information as part of an evaluation * Competency evaluation (language), Competency evaluation is a means for teachers to determine the ability of their students in other ways besides the standardized test * Educational evaluation is evaluation that is conducted specifically in an educational setting * Immanent evaluation, opposed by Gilles Deleuze to value judgment * Performance evaluation is a term from the field of language testing. It stands in contrast to competence evaluation * Program evaluation is essentially a set of philosophies and techniques to determine if a program 'works' * Donald Kirkpatrick, Donald Kirkpatrick's Evaluation Model for training evaluation * Reputation system, Rating * Efficiently updatable neural network A neural network based evaluation function


References


External links


Links to Assessment and Evaluation Resources
- List of links to resources on several topics


Evaluation Portal Link Collection
Evaluation link collection with information about evaluation journals, dissemination, projects, societies, how-to texts, books, and much more
Free Resources for Methods in Evaluation and Social Research

Introduction to and Discussions on Monitoring & Evaluation of Development Programs & Projects

Basic Guide to Program Evaluation, Gene ShackmanBetterEvaluation
- A website and resource library with freely available information about how to conduct, manage, and use evaluation well. {{Authority control Evaluation, Evaluation methods, Educational assessment and evaluation,