Designing Social Inquiry
   HOME

TheInfoList



OR:

''Designing Social Inquiry: Scientific Inference in Qualitative Research'' (or KKV) is an influential 1994 book written by Gary King,
Robert Keohane Robert Owen Keohane (born October 3, 1941) is an American academic working within the fields of international relations and international political economy. Following the publication of his influential book ''After Hegemony'' (1984), he has beco ...
, and
Sidney Verba Sidney Verba (May 26, 1932 – March 4, 2019) was an American political scientist, librarian and library administrator. His academic interests were mainly American and comparative politics. He was the Carl H. Pforzheimer University Professor at H ...
that lays out guidelines for conducting
qualitative research Qualitative research is a type of research that aims to gather and analyse non-numerical (descriptive) data in order to gain an understanding of individuals' social reality, including understanding their attitudes, beliefs, and motivation. This ...
. The central thesis of the book is that qualitative and
quantitative research Quantitative research is a research strategy that focuses on quantifying the collection and analysis of data. It is formed from a deductive approach where emphasis is placed on the testing of theory, shaped by empiricist and positivist philosop ...
share the same "logic of
inference Inferences are steps in reasoning, moving from premises to logical consequences; etymologically, the word '' infer'' means to "carry forward". Inference is theoretically traditionally divided into deduction and induction, a distinction that in ...
." The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research. The text is often referred to as KKV within social science disciplines. The book has been the subject of intense debate among social scientists. The 2004 book ''Rethinking Social Inquiry,'' edited by Henry E. Brady and David Collier, is an influential summary of responses to KKV.


History

Robert Keohane recounts the origins of KKV as follows,
''Designing Social Inquiry'' was not generated by puzzles of world politics. Instead, it was the result of serendipity. Sid Verba and I were friends, and when I joined the Harvard Government Department in 1985, he said that we should teach a course together. I regarded this remark as a welcoming pleasantry, typical of Sid's grace and warmth. Three years later I became chair of the department and in my first year as chair was forced to listen to 24 job talks. Most of these talks were dead on arrival, since the speaker had made fundamental mistakes in research design. I complained to colleagues, including Sid, and Gary King. Gary said the three of us should teach a course on research design together... I agreed, and we taught the course the following year... After the semester was over, Gary said: “We should teach the course again. And this time, we should write a book on this subject.” The next year we met regularly for a bag lunch, discussing not only themes of the course but drafts that one of us—most often Gary, which is why his name appears first on the book—had produced.


Contents

The goal of the book is guide researchers in producing valid
causal inference Causal inference is the process of determining the independent, actual effect of a particular phenomenon that is a component of a larger system. The main difference between causal inference and inference of association is that causal inference ana ...
s in social science research. The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research. The authors argue “whether we study many phenomena or few… the study will be improved if we collect data on as many observable implications of our theory as possible.” The authors note that
case studies A case study is an in-depth, detailed examination of a particular case (or cases) within a real-world context. For example, case studies in medicine may focus on an individual patient or ailment; case studies in business might cover a particular fi ...
do not necessarily have to be N=1 or few N: a case study can include many observations within a case (many individuals and entities across many time periods). KKV criticize
Harry H. Eckstein Harry H. Eckstein (January 26, 1924 in Schotten, Germany – June 22, 1999) was an American political scientist. He was an influential scholar of comparative politics and political culture, as well as qualitative research methods. Early life and ...
's notion of "crucial case studies", warning that a single observation makes it harder to estimate multiple causal effects, more likely that there is
measurement error Observational error (or measurement error) is the difference between a measured value of a quantity and its true value.Dodge, Y. (2003) ''The Oxford Dictionary of Statistical Terms'', OUP. In statistics, an error is not necessarily a " mistake ...
, and risks that an event in a single case was caused by random error. According to the authors, a strong research design requires both qualitative and quantitative research, a research question that poses an important and real question that will contribute to the base of knowledge about this particular subject, and a comprehensive literature review from which hypotheses (theory-driven) are then drawn. Data that are collected should be operationalized so that other researchers could replicate the study and achieve similar results. For the same reason, the reasoning process behind the analysis needs to be explicit. While gathering data the researcher should consider the observable implications of the theory in an effort to explain as much of the data as possible. This is in addition to examining the causal mechanisms that connect one variable to another. While qualitative methods cannot produce precise measurements of uncertainty about the conclusions (unlike quantitative methods), qualitative scholars should give indications about the uncertainty of their inferences. KKV argue that "the single most serious problem with qualitative research in political science is the pervasive failure to provide reasonable estimates of the uncertainty of the investigator’s inferences." According to KKV, the rules for good causal theories are that they need to: * be
falsifiable Falsifiability is a standard of evaluation of scientific theories and hypotheses that was introduced by the philosopher of science Karl Popper in his book ''The Logic of Scientific Discovery'' (1934). He proposed it as the cornerstone of a sol ...
* have
internal consistency In statistics and research, internal consistency is typically a measure based on the correlations between different items on the same test (or the same subscale on a larger test). It measures whether several items that propose to measure the same g ...
(generate hypotheses that do not contradict each other) * have variation (explanatory variables should be exogenous and dependent variables should be
endogenous Endogenous substances and processes are those that originate from within a living system such as an organism, tissue, or cell. In contrast, exogenous substances and processes are those that originate from outside of an organism. For example, es ...
) * have "concrete" concepts (concepts should be observable) * have "leverage" (the theory should explain much by little). KKV sees process-tracing and qualitative research as being "unable to yield strong causal inference" due to the fact that qualitative scholars would struggle with determining which of many intervening variables truly links the independent variable with a dependent variable. The primary problem is that qualitative research lacks a sufficient number of observations to properly estimate the effects of an independent variable. They write that the number of observations could be increased through various means, but that would simultaneously lead to another problem: that the number of variables would increase and thus reduce
degrees of freedom Degrees of freedom (often abbreviated df or DOF) refers to the number of independent variables or parameters of a thermodynamic system. In various scientific fields, the word "freedom" is used to describe the limits to which physical movement or ...
. In terms of case selection, KKV warn against " selecting on the dependent variable". For example, researchers cannot make valid causal inferences about wars outbreak by only looking at instances where war did happen (the researcher should also look at cases where war did not happen). There is methodological problem in selecting on the explanatory variable, however. They do warn about
multicollinearity In statistics, multicollinearity (also collinearity) is a phenomenon in which one predictor variable in a multiple regression model can be linearly predicted from the others with a substantial degree of accuracy. In this situation, the coefficie ...
(choosing two or more explanatory variables that perfectly correlate with each other). They argue that random selection of cases is a valid case selection strategy in large-N research, but warn against it in small-N research. KKV reject the notion of "
quasi-experiment A quasi-experiment is an empirical interventional study used to estimate the causal impact of an intervention on target population without random assignment. Quasi-experimental research shares similarities with the traditional experimental design ...
s", arguing that either all the key causal variables can be controlled (an experiment) or not (a non-experiment).


Reception

In his 2010 review, James Mahoney writes that the field of social science methodology has "benefited from KKV even as it has also moved beyond it." Critics of KKV have characterized the book's claims as "often simplistic, misleading and inappropriate as a guide for designing social inquiry." Quantitative scholars such as Henry E. Brady, Larry M. Bartels and David A. Freedman have argued that KKV overstate the strengths of quantitative research vis-a-vis qualitative research. Henry Brady and David Collier argue that KKV exaggerate the ability of quantitative research to identify uncertainty. They also argue that KKV exaggerate the risks of conducting inductive research and forming hypotheses post hoc. Numerous scholars disagree with KKV in their claims that qualitative research should integrate standards from quantitative research. There are different logics to the manner in which qualitative research is conducted and what qualitative scholars seek and can do with their data. Brady and Collier argue that KKV give insufficient attention to these divergent logics, as well as the intrinsic tradeoffs between different methodological goals. Gary Goertz and James Mahoney dispute that the main difference between qualitative and quantitative research is the size of N. Instead, a primary difference is that qualitative scholars tend to do within-case analyses whereas quantitative scholars almost by definition do cross-case analyses. Mahoney writes that KKV ignore
set theory Set theory is the branch of mathematical logic that studies sets, which can be informally described as collections of objects. Although objects of any kind can be collected into a set, set theory, as a branch of mathematics, is mostly conce ...
and
logic Logic is the study of correct reasoning. It includes both formal and informal logic. Formal logic is the science of deductively valid inferences or of logical truths. It is a formal science investigating how conclusions follow from premises ...
in terms of evaluating causal inference. Whereas regression-oriented analyses seek to estimate ''average effects'' of certain outcomes, qualitative research seeks to explain ''why'' cases have certain outcomes. Thus, causal inference is not strengthened by expanding the size of N, but rather by carefully choosing cases, whose testing can strengthen or weaken a theory. Mahoney and Gary Goertz make an analogy with a murder case: a single piece of smoking gun evidence can conclusively show whether a person committed a murder. Mahoney also writes that KKV give insufficient attention to concept formation, which is an essential aspect of theory construction and measurement, and one of the important ways that qualitative research can play a key role.
Ronald Rogowski Ronald Rogowski (born May 16, 1944) is a political scientist who focuses on comparative politics and international political economy. He is a professor in UCLA's department of Political Science where he has taught since 1981. He has also taught at ...
criticizes how KKV treat qualitative social science research. Rogowski argues that there is too much focus on hypothesis-testing and too much caution against using single observations. Rogowski argues that KKV promotes a form of qualitative social science that is overly focused on hypothesis-testing, and that this limits scholars' questions, cases and ambitions. John J. Mearsheimer and Stephen M. Walt argue that International Relations scholarship has shifted away from crafting and refining IR theory to "simplistic hypothesis-testing", in part due to the influence of KKV in political science graduate programs. Alexander George and Andrew Bennett say there is "much to agree with" in KKV, but they argue that the book has several flaws in its guidance on qualitative research: * Causal mechanisms: KKV suggest that "causal mechanisms" are less important that "causal effects" in causal explanations – George and Bennett argue that they are equally important * Hypothesis-testing: KKV overly emphasize the role of hypothesis-testing in theory development – George and Bennett argue that the formation of new hypotheses and the choosing of new questions are also important aspects of theory development * Causal complexity: KKV fail to consider problems of causal complexity, such as equifinality, multiple interaction effects,
feedback loops Feedback occurs when outputs of a system are routed back as inputs as part of a chain of cause-and-effect that forms a circuit or loop. The system can then be said to ''feed back'' into itself. The notion of cause-and-effect has to be handled ...
,
path dependency Path dependence is a concept in economics and the social sciences, referring to processes where past events or decisions constrain later events or decisions. It can be used to refer to outcomes at a single point in time or to long-run equilibria ...
, tipping points, selection effects, expectations effects and sequential interactions – George and Bennet argue that case studies,
process tracing Process tracing is a research method used to develop and test theories. It is generally understood as a "within-case" method to draw inferences on the basis of causal mechanisms. It has been used in social sciences (such as in psychology), as well a ...
and typological theories can clarify causality in situations of causal complexity * Increasing N: KKV argue that scholars should always seek to increase the number of cases – George and Bennett argue that KKV fail to consider the costs to increasing the number of cases (such as conceptual stretching and unintentional comparisons of dissimilar cases). George and Bennett note that much value can be derived from single-case studies. * Process-tracing: KKV characterize process-tracing as a way to increase the number of observable implications – George and Bennett argue that the logic of process-tracing is entirely different. The logic behind using process-tracing is to focus on the sequences and timings within a case, not to correlate data across cases. Thus, if one piece of evidence in the sequence is inconsistent with the theoretical expectations, then the theory has been shown to be flawed. * "Degrees of freedom" problem: KKV argue that a single case cannot evaluate competing explanations due problems that arise from
degrees of freedom Degrees of freedom (often abbreviated df or DOF) refers to the number of independent variables or parameters of a thermodynamic system. In various scientific fields, the word "freedom" is used to describe the limits to which physical movement or ...
– George and Bennett argue that it is flawed to apply this statistical logic to qualitative research. George and Bennett say that while quantitative scholars try to aggregate variables to reduce the number of variables and thus increase the degrees of freedom, qualitative scholars intentionally want their variables to have many different attributes and complexity.


Further reading


Review symposium in the ''American Political Science Review'' Vol. 89, No. 2, Jun., 1995
*Gary Goertz and James Mahoney provide a bullet point summary of KKV's contents in their 2012 book,
A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences
'.


References

{{Reflist 1994 non-fiction books Political science books