Evaluation is a
system
A system is a group of Interaction, interacting or interrelated elements that act according to a set of rules to form a unified whole. A system, surrounded and influenced by its environment (systems), environment, is described by its boundaries, ...
atic determination and assessment of a subject's merit, worth and significance, using criteria governed by a set of
standards Standard may refer to:
Symbols
* Colours, standards and guidons, kinds of military signs
* Standard (emblem), a type of a large symbol or emblem used for identification
Norms, conventions or requirements
* Standard (metrology), an object th ...
. It can assist an organization, program, design, project or any other intervention or initiative to assess any aim, realisable concept/proposal, or any alternative, to help in
decision-making
In psychology, decision-making (also spelled decision making and decisionmaking) is regarded as the Cognition, cognitive process resulting in the selection of a belief or a course of action among several possible alternative options. It could be ...
; or to ascertain the degree of achievement or value in regard to the aim and
objectives and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining
insight
Insight is the understanding of a specific cause and effect within a particular context. The term insight can have several related meanings:
*a piece of information
*the act or result of understanding the inner nature of things or of seeing intu ...
into prior or existing initiatives, is to enable
reflection Reflection or reflexion may refer to:
Science and technology
* Reflection (physics), a common wave phenomenon
** Specular reflection, reflection from a smooth surface
*** Mirror image, a reflection in a mirror or in water
** Signal reflection, in ...
and assist in the identification of future change. Evaluation is often used to characterize and appraise subjects of interest in a wide range of human enterprises, including the
arts
The arts are a very wide range of human practices of creative expression, storytelling and cultural participation. They encompass multiple diverse and plural modes of thinking, doing and being, in an extremely broad range of media. Both hi ...
,
criminal justice
Criminal justice is the delivery of justice to those who have been accused of committing crimes. The criminal justice system is a series of government agencies and institutions. Goals include the Rehabilitation (penology), rehabilitation of o ...
,
foundations
Foundation may refer to:
* Foundation (nonprofit), a type of charitable organization
** Foundation (United States law), a type of charitable organization in the U.S.
** Private foundation, a charitable organization that, while serving a good cause ...
,
non-profit organization
A nonprofit organization (NPO) or non-profit organisation, also known as a non-business entity, not-for-profit organization, or nonprofit institution, is a legal entity organized and operated for a collective, public or social benefit, in co ...
s,
government
A government is the system or group of people governing an organized community, generally a state.
In the case of its broad associative definition, government normally consists of legislature, executive, and judiciary. Government is a ...
,
health care
Health care or healthcare is the improvement of health via the prevention, diagnosis, treatment, amelioration or cure of disease, illness, injury, and other physical and mental impairments in people. Health care is delivered by health profe ...
, and other human services. It is long term and done at the end of a period of time.
Definition
Evaluation is the structured interpretation and giving of meaning to predicted or actual impacts of proposals or results. It looks at original objectives, and at what is either predicted or what was accomplished and how it was accomplished. So evaluation can be
formative, that is taking place during the development of a concept or proposal, project or organization, with the intention of improving the value or effectiveness of the proposal, project, or organisation. It can also be
summative, drawing lessons from a completed action or project or an organisation at a later point in time or circumstance.
Evaluation is inherently a theoretically informed approach (whether explicitly or not), and consequently any particular definition of evaluation would have been tailored to its contextthe theory, needs, purpose, and methodology of the evaluation process itself. Having said this, evaluation has been defined as:
* A systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement, or outcomes of a program. It is a resource-intensive process, frequently requiring resources, such as, evaluate expertise, labor, time, and a sizable budget
* "The critical assessment, in as objective a manner as possible, of the degree to which a service or its component parts fulfills stated goals" (St Leger and Wordsworth-Bell).
The focus of this definition is on attaining objective knowledge, and scientifically or quantitatively measuring predetermined and external concepts.
* "A study designed to assist some audience to assess an object's merit and worth" (Stufflebeam).
In this definition the focus is on facts as well as value laden judgments of the programs outcomes and worth.
Purpose
The main purpose of a program evaluation can be to "determine the quality of a program by formulating a judgment" Marthe Hurteau, Sylvain Houle, Stéphanie Mongiat (2009).
An alternative view is that "projects, evaluators, and other stakeholders (including funders) will all have potentially different ideas about how best to evaluate a project since each may have a different definition of 'merit'. The core of the problem is thus about defining what is of value."
From this perspective, evaluation "is a contested term", as "evaluators" use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research.
There are two function considering to the evaluation purpose Formative Evaluations provide the information on the improving a product or a process Summative Evaluations provide information of short-term effectiveness or long-term impact to deciding the adoption of a product or process.
Not all evaluations serve the same purpose some evaluations serve a monitoring function rather than focusing solely on measurable program outcomes or evaluation findings and a full list of types of evaluations would be difficult to compile.
This is because evaluation is not part of a unified theoretical framework, drawing on a number of disciplines, which include
management
Management (or managing) is the administration of an organization, whether it is a business, a nonprofit organization, or a government body. It is the art and science of managing resources of the business.
Management includes the activities o ...
and
organisational theory,
policy analysis
Policy analysis is a technique used in the public administration sub-field of political science to enable civil servants, nonprofit organizations, and others to examine and evaluate the available options to implement the goals of laws and elected ...
,
education
Education is a purposeful activity directed at achieving certain aims, such as transmitting knowledge or fostering skills and character traits. These aims may include the development of understanding, rationality, kindness, and honesty. Va ...
,
sociology
Sociology is a social science that focuses on society, human social behavior, patterns of Interpersonal ties, social relationships, social interaction, and aspects of culture associated with everyday life. It uses various methods of Empirical ...
,
social anthropology
Social anthropology is the study of patterns of behaviour in human societies and cultures. It is the dominant constituent of anthropology throughout the United Kingdom and much of Europe, where it is distinguished from cultural anthropology. In t ...
, and
social change
Social change is the alteration of the social order of a society which may include changes in social institutions, social behaviours or social relations.
Definition
Social change may not refer to the notion of social progress or sociocult ...
.
Discussion
However, the strict adherence to a set of methodological assumptions may make the field of evaluation more acceptable to a mainstream audience but this adherence will work towards preventing evaluators from developing new strategies for dealing with the myriad problems that programs face.
It is claimed that only a minority of evaluation reports are used by the evaluand (client) (Datta, 2006).
One justification of this is that "when evaluation findings are challenged or utilization has failed, it was because stakeholders and clients found the inferences weak or the warrants unconvincing" (Fournier and Smith, 1993).
Some reasons for this situation may be the failure of the evaluator to establish a set of shared aims with the evaluand, or creating overly ambitious aims, as well as failing to compromise and incorporate the cultural differences of individuals and programs within the evaluation aims and process.
None of these problems are due to a lack of a definition of evaluation but are rather due to evaluators attempting to impose predisposed notions and definitions of evaluations on clients. The central reason for the poor utilization of evaluations is arguably due to the lack of tailoring of evaluations to suit the needs of the client, due to a predefined idea (or definition) of what an evaluation is rather than what the client needs are (House, 1980).
The development of a standard methodology for evaluation will require arriving at applicable ways of asking and stating the results of questions about ethics such as agent-principal, privacy, stakeholder definition, limited liability; and could-the-money-be-spent-more-wisely issues.
Standards
Depending on the topic of interest, there are professional groups that review the quality and
rigor
Rigour (British English) or rigor (American English; see spelling differences) describes a condition of stiffness or strictness. These constraints may be environmentally imposed, such as "the rigours of famine"; logically imposed, such as ma ...
of evaluation processes.
Evaluating programs and projects, regarding their value and impact within the context they are implemented, can be
ethically
Ethics or moral philosophy is a branch of philosophy that "involves systematizing, defending, and recommending concepts of right and wrong behavior".''Internet Encyclopedia of Philosophy'' The field of ethics, along with aesthetics, concerns ...
challenging. Evaluators may encounter complex, culturally specific systems resistant to external evaluation. Furthermore, the project organization or other stakeholders may be invested in a particular evaluation outcome. Finally, evaluators themselves may encounter "
conflict of interest (COI)" issues, or experience interference or pressure to present findings that support a particular assessment.
General professional
codes of conduct
A code of conduct is a set of rules outlining the norms, rules, and responsibilities or proper practices of an individual party or an organization.
Companies' codes of conduct
A company code of conduct is a set of rules which is commonly writt ...
, as determined by the employing organization, usually cover three broad aspects of behavioral standards, and include inter-
collegial
Collegiality is the relationship between colleagues. A colleague is a fellow member of the same profession.
Colleagues are those explicitly united in a common purpose and respect each other's abilities to work toward that purpose. A colleague is ...
relations (such as respect for
diversity
Diversity, diversify, or diverse may refer to:
Business
*Diversity (business), the inclusion of people of different identities (ethnicity, gender, age) in the workforce
*Diversity marketing, marketing communication targeting diverse customers
* ...
and
privacy
Privacy (, ) is the ability of an individual or group to seclude themselves or information about themselves, and thereby express themselves selectively.
The domain of privacy partially overlaps with security, which can include the concepts of a ...
), operational issues (due
competence, documentation accuracy and appropriate use of resources), and conflicts of interest (
nepotism
Nepotism is an advantage, privilege, or position that is granted to relatives and friends in an occupation or field. These fields may include but are not limited to, business, politics, academia, entertainment, sports, fitness, religion, an ...
, accepting gifts and other kinds of favoritism).
However, specific guidelines particular to the evaluator's role that can be utilized in the management of unique ethical challenges are required. The
Joint Committee on Standards for Educational Evaluation The Joint Committee on Standards for Educational Evaluation is an American/Canadian based Standards Developer Organization (SDO). The Joint Committee, created in 1975, represents a coalition of major professional associations formed in 1975 to dev ...
has developed standards for program, personnel, and student evaluation. The Joint Committee standards are broken into four sections: Utility, Feasibility, Propriety, and Accuracy. Various European institutions have also prepared their own standards, more or less related to those produced by the Joint Committee. They provide guidelines about basing value judgments on systematic inquiry, evaluator competence and integrity, respect for people, and regard for the general and public welfare.
The American Evaluation Association has created a set of Guiding
Principle
A principle is a proposition or value that is a guide for behavior or evaluation. In law, it is a Legal rule, rule that has to be or usually is to be followed. It can be desirably followed, or it can be an inevitable consequence of something, suc ...
s for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows:
* Systematic
Inquiry
An inquiry (also spelled as enquiry in British English) is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ...
: evaluators conduct systematic,
data
In the pursuit of knowledge, data (; ) is a collection of discrete values that convey information, describing quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted ...
-based inquiries about whatever is being evaluated. This requires quality data collection, including a defensible choice of indicators, which lends credibility to findings.
Findings are credible when they are demonstrably evidence-based, reliable and valid. This also pertains to the choice of
methodology
In its most common sense, methodology is the study of research methods. However, the term can also refer to the methods themselves or to the philosophical discussion of associated background assumptions. A method is a structured procedure for bri ...
employed, such that it is consistent with the aims of the evaluation and provides dependable data. Furthermore, utility of findings is critical such that the information obtained by evaluation is comprehensive and timely, and thus serves to provide maximal benefit and use to
stakeholders.
* Competence: evaluators provide competent performance to
stakeholders. This requires that evaluation teams comprise an appropriate combination of competencies, such that varied and appropriate expertise is available for the evaluation process, and that evaluators work within their scope of capability.
*
Integrity
Integrity is the practice of being honest and showing a consistent and uncompromising adherence to strong moral and ethical principles and values.
In ethics, integrity is regarded as the honesty and truthfulness or accuracy of one's actions. Inte ...
/
Honesty
Honesty or truthfulness is a facet of moral character that connotes positive and virtuous attributes such as integrity, truthfulness, straightforwardness, including straightforwardness of conduct, along with the absence of lying, cheating, theft, ...
: evaluators ensure the honesty and integrity of the entire evaluation process. A key element of this principle is freedom from bias in evaluation and this is underscored by three principles: impartiality, independence, and transparency.
Independence is attained through ensuring independence of judgment is upheld such that evaluation
conclusions are not influenced or pressured by another party, and avoidance of conflict of
interest, such that the evaluator does not have a stake in a particular conclusion. Conflict of
interest is at issue particularly where funding of evaluations is provided by particular bodies
with a stake in conclusions of the evaluation, and this is seen as potentially compromising the
independence of the evaluator. Whilst it is acknowledged that evaluators may be familiar with
agencies or projects that they are required to evaluate, independence requires that they not have
been involved in the planning or implementation of the project. A declaration of interest should
be made where any benefits or association with project are stated. Independence of judgment is
required to be maintained against any pressures brought to bear on evaluators, for example, by
project funders wishing to modify evaluations such that the project appears more effective than
findings can verify.
Impartiality pertains to findings being a fair and thorough assessment of strengths and
weaknesses of a project or program. This requires taking due input from all stakeholders involved
and findings presented without bias and with a transparent, proportionate, and persuasive link
between findings and recommendations. Thus evaluators are required to delimit their findings to
evidence. A mechanism to ensure impartiality is external and internal review. Such review is
required of significant (determined in terms of cost or sensitivity) evaluations. The review is
based on quality of work and the degree to which a demonstrable link is provided between findings
and recommendations.
Transparency requires that stakeholders are aware of the reason for the evaluation, the criteria
by which evaluation occurs and the purposes to which the findings will be applied. Access to the
evaluation document should be facilitated through findings being easily readable, with clear
explanations of evaluation methodologies, approaches, sources of information, and costs
incurred.
*
Respect
Respect, also called esteem, is a positive feeling or action shown towards someone or something considered important or held in high esteem or regard. It conveys a sense of admiration for good or valuable qualities. It is also the process of ...
for People: Evaluators respect the
security
Security is protection from, or resilience against, potential harm (or other unwanted coercive change) caused by others, by restraining the freedom of others to act. Beneficiaries (technically referents) of security may be of persons and social ...
,
dignity
Dignity is the right of a person to be valued and respected for their own sake, and to be treated ethically. It is of significance in morality, ethics, law and politics as an extension of the Enlightenment-era concepts of inherent, inalienable ...
and
self-worth
Self-esteem is confidence in one's own worth or abilities. Self-esteem encompasses beliefs about oneself (for example, "I am loved", "I am worthy") as well as emotional states, such as triumph, despair, pride, and shame. Smith and Mackie (2007) d ...
of the respondents,
program
Program, programme, programmer, or programming may refer to:
Business and management
* Program management, the process of managing several related projects
* Time management
* Program, a part of planning
Arts and entertainment Audio
* Progra ...
participants
Participation or Participant may refer to:
Politics
*Participation (decision making), mechanisms for people to participate in social decisions
*Civic participation, engagement by the citizens in government
*e-participation, citizen participation ...
,
clients, and other stakeholders with whom they
interact
Advocates for Informed Choice, dba interACT or interACT Advocates for Intersex Youth, is a 501(c)(3) nonprofit organization using innovative strategies to advocate for the legal and human rights of children with intersex traits. The organizati ...
.This is particularly pertinent with regards to those who will be impacted upon by the evaluation findings.
Protection of people includes ensuring informed consent from those involved in the evaluation, upholding confidentiality, and ensuring that the identity of those who may provide sensitive information towards the program evaluation is protected. Evaluators are ethically required to respect the customs and beliefs of those who are impacted upon by the evaluation or program activities. Examples of how such respect is demonstrated is through respecting local customs e.g. dress codes, respecting peoples privacy, and minimizing demands on others' time.
Where stakeholders wish to place objections to evaluation findings, such a process should be facilitated through the local office of the evaluation organization, and procedures for lodging complaints or queries should be accessible and clear.
*
Responsibilities for General and
Public Welfare
Welfare, or commonly social welfare, is a type of government support intended to ensure that members of a society can meet basic human needs such as food and shelter. Social security may either be synonymous with welfare, or refer specificall ...
: Evaluators articulate and take into account the diversity of
interest
In finance and economics, interest is payment from a borrower or deposit-taking financial institution to a lender or depositor of an amount above repayment of the principal sum (that is, the amount borrowed), at a particular rate. It is distinct ...
s and
values
In ethics and social sciences, value denotes the degree of importance of something or action, with the aim of determining which actions are best to do or what way is best to live (normative ethics in ethics), or to describe the significance of dif ...
that may be related to the general and public welfare. Access to evaluation documents by the wider public should be facilitated such that discussion and feedback is enabled.
Furthermore, the international organizations such as the I.M.F. and the World Bank have independent evaluation functions. The various funds, programmes, and agencies of the United Nations has a mix of independent, semi-independent and self-evaluation functions, which have organized themselves as a system-wide
UN Evaluation Group (UNEG),
that works together to strengthen the function, and to establish UN norms and standards for evaluation. There is also an evaluation group within the OECD-DAC, which endeavors to improve development evaluation standards. The independent evaluation units of the major multinational development banks (MDBs) have also created the Evaluation Cooperation Group to strengthen the use of evaluation for greater MDB effectiveness and accountability, share lessons from MDB evaluations, and promote evaluation harmonization and collaboration.
Perspectives
The word "evaluation" has various
connotations
A connotation is a commonly understood cultural or emotional association that any given word or phrase carries, in addition to its explicit or literal meaning, which is its denotation.
A connotation is frequently described as either positive or ...
for different people, raising issues related to this process that include; what type of evaluation should be conducted; why there should be an evaluation process and how the evaluation is integrated into a program, for the purpose of gaining greater knowledge and awareness? There are also various factors inherent in the evaluation process, for example; to critically examine influences within a program that involve the gathering and analyzing of relative information about a program.
Michael Quinn Patton
Michael Quinn Patton (born 1945) is an independent organizational development and program evaluation consultant, and former president of the American Evaluation Association. He is the founder and director of Utilization-Focused Evaluation.
After ...
motivated the concept that the evaluation procedure should be directed towards:
* Activities
* Characteristics
* Outcomes
* The making of judgments on a program
* Improving its effectiveness,
* Informed programming decisions
Founded on another perspective of evaluation by Thomson and Hoffman in 2003, it is possible for a situation to be encountered, in which the process could not be considered advisable; for instance, in the event of a program being unpredictable, or unsound. This would include it lacking a consistent routine; or the concerned parties unable to reach an agreement regarding the purpose of the program. In addition, an influencer, or manager, refusing to incorporate relevant, important central issues within the evaluation
Approaches
There exist several conceptually distinct ways of thinking about, designing, and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way.
Classification of approaches
Two classifications of evaluation approaches by House
[House, E. R. (1978). Assumptions underlying evaluation models. ''Educational Researcher''. 7(3), 4-12.] and Stufflebeam and Webster
[Stufflebeam, D. L., & Webster, W. J. (1980)]
"An analysis of alternative approaches to evaluation"
''Educational Evaluation and Policy Analysis
''Educational Evaluation and Policy Analysis'' is a peer-reviewed academic journal covering all aspects of educational policy analysis. It was established in 1979 and is published by SAGE Publications on behalf of the American Educational Research ...
''. 2(3), 5-19. can be combined into a manageable number of approaches in terms of their unique and important underlying principles.
House considers all major evaluation approaches to be based on a common
ideology
An ideology is a set of beliefs or philosophies attributed to a person or group of persons, especially those held for reasons that are not purely epistemic, in which "practical elements are as prominent as theoretical ones." Formerly applied pri ...
entitled
liberal democracy
Liberal democracy is the combination of a liberal political ideology that operates under an indirect democratic form of government. It is characterized by elections between multiple distinct political parties, a separation of powers into diff ...
. Important principles of this ideology include freedom of choice, the uniqueness of the
individual
An individual is that which exists as a distinct entity. Individuality (or self-hood) is the state or quality of being an individual; particularly (in the case of humans) of being a person unique from other people and possessing one's own Maslow ...
and
empirical
Empirical evidence for a proposition is evidence, i.e. what supports or counters this proposition, that is constituted by or accessible to sense experience or experimental procedure. Empirical evidence is of central importance to the sciences and ...
inquiry grounded in
objectivity. He also contends that they are all based on
subjectivist
Subjectivism is the doctrine that "our own mental activity is the only unquestionable fact of our experience", instead of shared or communal, and that there is no external or objective truth.
The success of this position is historically attribute ...
ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is
utilitarian
In ethical philosophy, utilitarianism is a family of normative ethical theories that prescribe actions that maximize happiness and well-being for all affected individuals.
Although different varieties of utilitarianism admit different charac ...
, in which "
the good
In most contexts, the concept of good denotes the conduct that should be preferred when posed with a choice between possible actions. Good is generally considered to be the opposite of evil and is of interest in the study of ethics, morality, ph ...
" is determined by what maximizes a single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is
intuitionist
In the philosophy of mathematics, intuitionism, or neointuitionism (opposed to preintuitionism), is an approach where mathematics is considered to be purely the result of the constructive mental activity of humans rather than the discovery of fu ...
/
pluralist, in which no single interpretation of "the good" is assumed and such interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding
epistemologies—
philosophies
Philosophical schools of thought and philosophical movements.
A
Absurdism -
Action, philosophy of -
Actual idealism -
Actualism -
Advaita Vedanta -
Aesthetic Realism -
Aesthetics -
African philosophy -
Afrocentrism -
Agential realism - ...
for obtaining
knowledge
Knowledge can be defined as awareness of facts or as practical skills, and may also refer to familiarity with objects or situations. Knowledge of facts, also called propositional knowledge, is often defined as true belief that is distinc ...
. The objectivist epistemology is associated with the utilitarian ethic; in general, it is used to acquire knowledge that can be externally verified (intersubjective agreement) through publicly exposed
methods
Method ( grc, μέθοδος, methodos) literally means a pursuit of knowledge, investigation, mode of prosecuting such inquiry, or system. In recent centuries it more often means a prescribed process for completing a task. It may refer to:
*Scien ...
and
data
In the pursuit of knowledge, data (; ) is a collection of discrete values that convey information, describing quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted ...
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic and is used to acquire new knowledge based on existing personal knowledge, as well as experiences that are (explicit) or are not (tacit) available for public inspection. House then divides each epistemological approach into two main political perspectives. Firstly, approaches can take an elite perspective, focusing on the interests of managers and professionals; or they also can take a mass perspective, focusing on
consumer
A consumer is a person or a group who intends to order, or uses purchased goods, products, or services primarily for personal, social, family, household and similar needs, who is not directly related to entrepreneurial or business activities. T ...
s and
participatory
Citizen Participation or Public Participation in social science refers to different mechanisms for the public to express opinions—and ideally exert influence—regarding political, economic, management or other social decisions. Participato ...
approaches.
Stufflebeam and Webster place approaches into one of three groups, according to their orientation toward the role of
values
In ethics and social sciences, value denotes the degree of importance of something or action, with the aim of determining which actions are best to do or what way is best to live (normative ethics in ethics), or to describe the significance of dif ...
and ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually is and might be—they call this
pseudo-
The prefix pseudo- (from Greek ψευδής, ''pseudes'', "false") is used to mark something that superficially appears to be (or behaves like) one thing, but is something else. Subject to context, ''pseudo'' may connote coincidence, imitation, ...
evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object—they call this
quasi
Quasi (phonetics 'kwa - zee') is an American indie rock band formed in Portland, Oregon in 1993 by former spouses Sam Coomes (vocals, guitar, rocksichord, various keyboards, bass) and Janet Weiss (vocals and drums). Joanna Bolme performed and ...
-evaluation. The values orientation includes approaches primarily intended to determine the value of an object—they call this true evaluation.
When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation.
[ Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—]experiment
An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
al research, management information systems
An information system (IS) is a formal, sociotechnical, organizational system designed to collect, process, information storage, store, and information distribution, distribute information. From a sociotechnical perspective, information systems a ...
, testing programs, objectives-based studies, and content analysis
Content analysis is the study of documents and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic ...
—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and policy
Policy is a deliberate system of guidelines to guide decisions and achieve rational outcomes. A policy is a statement of intent and is implemented as a procedure or protocol. Policies are generally adopted by a governance body within an organ ...
studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur
A connoisseur ( French traditional, pre-1835, spelling of , from Middle-French , then meaning 'to be acquainted with' or 'to know somebody/something') is a person who has a great deal of knowledge about the fine arts; who is a keen appreciator ...
studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
Summary of approaches
The following table is used to summarize each approach in terms of four attributes
Attribute may refer to:
* Attribute (philosophy), an extrinsic property of an object
* Attribute (research), a characteristic of an object
* Grammatical modifier, in natural languages
* Attribute (computing), a specification that defines a proper ...
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together.
Pseudo-evaluation
Politically
Politics (from , ) is the set of activities that are associated with making decisions in groups, or other forms of power relations among individuals, such as the distribution of resources or status. The branch of social science that studies ...
controlled and public relations
Public relations (PR) is the practice of managing and disseminating information from an individual or an organization (such as a business, government agency, or a nonprofit organization) to the public in order to influence their perception. P ...
studies are based on an objectivist epistemology
Objectivism is a philosophical system developed by Russian-American writer and philosopher Ayn Rand. She described it as "the concept of man as a heroic being, with his own happiness as the moral purpose of his life, with productive achievement ...
from an elite
In political and sociological theory, the elite (french: élite, from la, eligere, to select or to sort out) are a small group of powerful people who hold a disproportionate amount of wealth, privilege, political power, or skill in a group. D ...
perspective. Although both of these approaches seek to misrepresent value interpretations about an object, they function differently from each other. Information
Information is an abstract concept that refers to that which has the power to inform. At the most fundamental level information pertains to the interpretation of that which may be sensed. Any natural process that is not completely random ...
obtained through politically controlled studies is released or withheld to meet the special interests of the holder, whereas public relations information creates a positive image of an object regardless of the actual situation. Despite the application of both studies in real
Real may refer to:
Currencies
* Brazilian real (R$)
* Central American Republic real
* Mexican real
* Portuguese real
* Spanish real
* Spanish colonial real
Music Albums
* ''Real'' (L'Arc-en-Ciel album) (2000)
* ''Real'' (Bright album) (2010)
...
scenarios, neither of these approaches is acceptable evaluation practice.
Objectivist, elite, quasi-evaluation
As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal.
* Experimental research
An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a ...
is the best approach for determining causal relationships
Causality (also referred to as causation, or cause and effect) is influence by which one event, process, state, or object (''a'' ''cause'') contributes to the production of another event, process, state, or object (an ''effect'') where the cau ...
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodology
In its most common sense, methodology is the study of research methods. However, the term can also refer to the methods themselves or to the philosophical discussion of associated background assumptions. A method is a structured procedure for bri ...
may not be sufficiently responsive to the dynamically changing needs of most human service programs.
* Management information system
A management information system (MIS) is an information system used for decision-making, and for the coordination, control, analysis, and visualization of information in an organization. The study of the management information systems involves peop ...
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable
Quantity or amount is a property that can exist as a multitude or magnitude, which illustrate discontinuity and continuity. Quantities can be compared in terms of "more", "less", or "equal", or by assigning a numerical value multiple of a unit ...
data usually available at regular intervals.
* Testing
An examination (exam or evaluation) or test is an educational assessment intended to measure a test-taker's knowledge, skill, aptitude, physical fitness, or classification in many other topics (e.g., beliefs). A test may be administered verba ...
programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
* Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
* Content analysis
Content analysis is the study of documents and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic ...
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation
* Accountability
Accountability, in terms of ethics and governance, is equated with answerability, blameworthiness, liability, and the expectation of account-giving. As in an aspect of governance, it has been central to discussions related to problems in the publ ...
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the quality
Quality may refer to:
Concepts
*Quality (business), the ''non-inferiority'' or ''superiority'' of something
*Quality (philosophy), an attribute or a property
*Quality (physics), in response theory
*Energy quality, used in various science discipli ...
of products
Product may refer to:
Business
* Product (business), an item that serves as a solution to a specific consumer problem.
* Product (project management), a deliverable or set of deliverables that contribute to a business solution
Mathematics
* Produ ...
and services
Service may refer to:
Activities
* Administrative service, a required part of the workload of university faculty
* Civil service, the body of employees of a government
* Community service, volunteer service for the benefit of a community or a pu ...
. However, this approach quickly can turn practitioners and consumer
A consumer is a person or a group who intends to order, or uses purchased goods, products, or services primarily for personal, social, family, household and similar needs, who is not directly related to entrepreneurial or business activities. T ...
s into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation
* Decision-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration
Collaboration (from Latin ''com-'' "with" + ''laborare'' "to labor", "to work") is the process of two or more people, entities or organizations working together to complete a task or achieve a goal. Collaboration is similar to cooperation. Most ...
between an evaluator and decision-maker, allowing it to be susceptible to corruption
Corruption is a form of dishonesty or a criminal offense which is undertaken by a person or an organization which is entrusted in a position of authority, in order to acquire illicit benefits or abuse power for one's personal gain. Corruption m ...
and bias
Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individual, a group, ...
.
* Policy studies
Policy studies is a subdisicipline of political science that includes the analysis of the process of policymaking (the policy process) and the contents of policy (policy analysis). Policy analysis includes substantive area research (such as health ...
provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits
In production, research, retail, and accounting, a cost is the value of money that has been used up to produce something or deliver a service, and hence is not available for use anymore. In business, the cost may be one of acquisition, in which ...
of competing policies
Policy is a deliberate system of guidelines to guide decisions and achieve rational outcomes. A policy is a statement of intent and is implemented as a procedure or protocol. Policies are generally adopted by a governance body within an organ ...
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation
* Consumer
A consumer is a person or a group who intends to order, or uses purchased goods, products, or services primarily for personal, social, family, household and similar needs, who is not directly related to entrepreneurial or business activities. T ...
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
Subjectivist, elite, true evaluation
* Accreditation
Accreditation is the independent, third-party evaluation of a conformity assessment body (such as certification body, inspection body or laboratory) against recognised standards, conveying formal demonstration of its impartiality and competence to ...
/ certification
Certification is the provision by an independent body of written assurance (a certificate) that the product, service or system in question meets specific requirements. It is the formal attestation or confirmation of certain characteristics of a ...
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
* Connoisseur
A connoisseur ( French traditional, pre-1835, spelling of , from Middle-French , then meaning 'to be acquainted with' or 'to know somebody/something') is a person who has a great deal of knowledge about the fine arts; who is a keen appreciator ...
studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.
Subject, mass, true evaluation
* The adversary approach focuses on drawing out the pros and cons of controversial
Controversy is a state of prolonged public dispute or debate, usually concerning a matter of conflicting opinion or point of view. The word was coined from the Latin ''controversia'', as a composite of ''controversus'' – "turned in an opposite ...
issues through quasi-legal proceedings
Legal proceeding is an activity that seeks to invoke the power of a tribunal in order to enforce a law. Although the term may be defined more broadly or more narrowly as circumstances require, it has been noted that " e term ''legal proceedings'' i ...
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation
Cooperation (written as co-operation in British English) is the process of groups of organisms working or acting together for common, mutual, or some underlying benefit, as opposed to working in competition for selfish benefit. Many animal a ...
and heighten animosities between contesting parties if "winners" and "losers" emerge.
Client-centered
* Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external credibility
Credibility comprises the objective and subjective components of the believability of a source or message. Credibility dates back to Aristotle theory of Rhetoric. Aristotle defines rhetoric as the ability to see what is possibly persuasive in ...
and a favorable bias toward those who participated in the study.
Methods and techniques
Evaluation is methodologically diverse. Methods may be qualitative or quantitative
Quantitative may refer to:
* Quantitative research, scientific investigation of quantitative properties
* Quantitative analysis (disambiguation)
* Quantitative verse, a metrical system in poetry
* Statistics, also known as quantitative analysis ...
, and include case studies
A case study is an in-depth, detailed examination of a particular case (or cases) within a real-world context. For example, case studies in medicine may focus on an individual patient or ailment; case studies in business might cover a particular fi ...
, survey research
In research of human subjects, a survey is a list of questions aimed for extracting specific data from a particular group of people. Surveys may be conducted by phone, mail, via the internet, and also at street corners or in malls. Surveys are us ...
, statistical analysis
Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical analysis infers propertie ...
, model building, and many more such as:
* Accelerated aging
Accelerated aging is testing that uses aggravated conditions of heat, humidity, oxygen, sunlight, vibration, etc. to speed up the normal aging processes of items. It is used to help determine the long-term effects of expected levels of stress wit ...
* Action research
Action research is a philosophy and methodology of research generally applied in the social sciences. It seeks transformative change through the simultaneous process of taking action and doing research, which are linked together by critical refle ...
* Advanced product quality planning
* Alternative assessment
Alternative assessment is also known under various other terms, including:
* authentic assessment
* integrative assessment
* holistic assessment
In education, "alternative assessment" is in direct contrast to what is known as "traditional testing ...
* Appreciative Inquiry
Appreciative inquiry (AI) is a model that seeks to engage stakeholders in self-determined change. According to Gervase Bushe, professor of leadership and organization development at the Beedie School of Business and a researcher on the topic, "AI ...
* Assessment
Assessment may refer to:
Healthcare
*Health assessment, identifies needs of the patient and how those needs will be addressed
*Nursing assessment, gathering information about a patient's physiological, psychological, sociological, and spiritual s ...
* Axiomatic design Axiomatic design is a systems design methodology using matrix methods to systematically analyze the transformation of customer needs into functional requirements, design parameters, and process variables.*Suh (1990), ''The Principles of Design'', Ox ...
* Benchmarking
Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
Benchmarking is used to measure performan ...
* Case study
A case study is an in-depth, detailed examination of a particular case (or cases) within a real-world context. For example, case studies in medicine may focus on an individual patient or ailment; case studies in business might cover a particular fi ...
* Change management
Change management (sometimes abbreviated as CM) is a collective term for all approaches to prepare, support, and help individuals, teams, and organizations in making organizational change. It includes methods that redirect or redefine the use of ...
* Clinical trial
Clinical trials are prospective biomedical or behavioral research studies on human participants designed to answer specific questions about biomedical or behavioral interventions, including new treatments (such as novel vaccines, drugs, dietar ...
* Cohort study
A cohort study is a particular form of longitudinal study that samples a cohort (a group of people who share a defining characteristic, typically those who experienced a common event in a selected period, such as birth or graduation), performing ...
* Competitor analysis
Competitive analysis in marketing and strategic management is an assessment of the strengths and weaknesses of current and potential competitors. This analysis provides both an offensive and defensive strategic context to identify opportunities an ...
* Consensus decision-making
Consensus decision-making or consensus process (often abbreviated to ''consensus'') are group decision-making processes in which participants develop and decide on proposals with the aim, or requirement, of acceptance by all. The focus on es ...
* Consensus-seeking decision-making
Consensus decision-making or consensus process (often abbreviated to ''consensus'') are group decision-making processes in which participants develop and decide on proposals with the aim, or requirement, of acceptance by all. The focus on es ...
* Content analysis
Content analysis is the study of documents and communication artifacts, which might be texts of various formats, pictures, audio or video. Social scientists use content analysis to examine patterns in communication in a replicable and systematic ...
* Conversation analysis
Conversation analysis (CA) is an approach to the study of social interaction, embracing both verbal and non-verbal conduct, in situations of everyday life. CA originated as a sociological method, but has since spread to other fields. CA began with ...
* Cost-benefit analysis
* Data mining
* Delphi Technique }
The Delphi method or Delphi technique ( ; also known as Estimate-Talk-Estimate or ETE) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The ...
* Design Focused Evaluation
* Discourse analysis
Discourse analysis (DA), or discourse studies, is an approach to the analysis of written, vocal, or sign language use, or any significant semiotic event.
The objects of discourse Analysis ( discourse, writing, conversation, communicative event ...
* Educational accreditation
Educational accreditation is a quality assurance process under which services and operations of educational institutions or programs are evaluated and verified by an external body to determine whether applicable and recognized standards are met. ...
* Electronic portfolio
An electronic portfolio (also known as a digital portfolio, online portfolio, e-portfolio, e-folio, or eFolio) is a collection of electronic evidence assembled and managed by a user, usually on the Web. Such electronic evidence may include input te ...
* Environmental scanning
Market environment and business environment are marketing terms that refer to factors and forces that affect a firm's ability to build and maintain successful customer relationships. The business environment has been defined as "the totality of ph ...
* Ethnography
Ethnography (from Greek ''ethnos'' "folk, people, nation" and ''grapho'' "I write") is a branch of anthropology and the systematic study of individual cultures. Ethnography explores cultural phenomena from the point of view of the subject o ...
* Experiment
An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into Causality, cause-and-effect by demonstrating what outcome oc ...
* Experimental techniques
The design of experiments (DOE, DOX, or experimental design) is the design of any task that aims to describe and explain the variation of information under conditions that are hypothesized to reflect the variation. The term is generally associ ...
* Factor analysis
Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. For example, it is possible that variations in six observed ...
* Factorial experiment
* Feasibility study
* Field experiment
* Fixtureless in-circuit test
* Focus group
* Force field analysis
* Game theory
* Goal-free evaluation
* Grade (education), Grading
* Historical method
* Inquiry
An inquiry (also spelled as enquiry in British English) is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ...
* Interview
* Iterative design
* Marketing research
* Meta-analysis
* Performance metric, Metrics
* Most significant change technique
* Multivariate statistics
* Naturalistic observation
* Observational techniques
* Opinion polling
* Organizational learning
* Outcome mapping
* Outcomes theory
* Participant observation
* Participatory impact pathways analysis
* Policy analysis
* Post occupancy evaluation
* Process improvement
* Project management
* Qualitative research
* Quality audit
* Quality circle
* Quality control
* Quality management
* Quality management system
* Quantitative research
* Questionnaire
* Questionnaire construction
* Review
* Root cause analysis
* Rubric (academic), Rubrics
* Sampling (statistics), Sampling
* Self-assessment
* Six Sigma
* Standardized testing
* Statistical process control
* Statistical survey
* Statistics
* Strategic planning
* Structured interviewing
* Systems theory
* Test (student assessment), Student testing
* Theory of change
* Total quality management
* Triangulation (social science), Triangulation
* Wizard of Oz experiment
See also
* Monitoring and Evaluation is a process used by governments, international organizations and NGOs to assess ongoing or past activities
* Assessment
Assessment may refer to:
Healthcare
*Health assessment, identifies needs of the patient and how those needs will be addressed
*Nursing assessment, gathering information about a patient's physiological, psychological, sociological, and spiritual s ...
is the process of gathering and analyzing specific information as part of an evaluation
* Competency evaluation (language), Competency evaluation is a means for teachers to determine the ability of their students in other ways besides the standardized test
* Educational evaluation is evaluation that is conducted specifically in an educational setting
* Immanent evaluation, opposed by Gilles Deleuze to value judgment
* Performance evaluation is a term from the field of language testing. It stands in contrast to competence evaluation
* Program evaluation is essentially a set of philosophies and techniques to determine if a program 'works'
* Donald Kirkpatrick, Donald Kirkpatrick's Evaluation Model for training evaluation
* Reputation system, Rating
* Efficiently updatable neural network A neural network based evaluation function
* Metric fixation
References
External links
Links to Assessment and Evaluation Resources
- List of links to resources on several topics
Evaluation Portal Link Collection
Evaluation link collection with information about evaluation journals, dissemination, projects, societies, how-to texts, books, and much more
Free Resources for Methods in Evaluation and Social Research
Introduction to and Discussions on Monitoring & Evaluation of Development Programs & Projects
Basic Guide to Program Evaluation, Gene Shackman
BetterEvaluation
- A website and resource library with freely available information about how to conduct, manage, and use evaluation well.
{{Criticism
Evaluation,
Evaluation methods,
Educational assessment and evaluation,