Wednesday, March 16, 2016

ABM fundamentalism

image: Chernobyl control room

Quite a few recent posts have examined the power and flexibility of ABM models as platforms for simulating a wide range of social phenomena. Joshua Epstein is one of the high-profile contributors to this field, and he is famous for making a particularly strong claim on behalf of ABM methods. He argues that “generative” explanations are the uniquely best form of social explanation. A generative explanation is one that demonstrates how an upper-level structure or causal power comes about as a consequence of the operations of the units that make it up. As an aphorism, here is Epstein's slogan: "If you didn't grow it, you didn't explain it." 

Here is how he puts the point in a Brookings working paper, “Remarks on the foundations of agent-based generative social science” (link; also chapter 1 of Generative Social Science: Studies in Agent-Based Computational Modeling):

"To the generativist, explaining macroscopic social regularities, such as norms, spatial patterns, contagion dynamics, or institutions requires that one answer the following question:
"How could the autonomous local interactions of heterogeneous boundedly rational agents generate the given regularity?
"Accordingly, to explain macroscopic social patterns, we generate—or “grow”—them in agent models." (1)

And Epstein is quite explicit in saying that this formulation represents a necessary condition on all putative social explanations: "In summary, generative sufficiency is a necessary, but not sufficient condition for explanation." (5).

There is an apparent logic to this view of explanation. However, several earlier posts cast doubt on the conclusion. First, we have seen that all ABMs necessarily make abstractive assumptions about the behavioral features of the actors, and they have a difficult time incorporating "structural" factors like organizations. We found that the ABM simulations of ethnic and civil conflict (including Epstein's own model) are radically over-simplified representations of the field of civil conflict (link).  So it is problematic to assume the general applicability and superiority of ABM approaches for all issues of social explanation.

Second, we have also emphasized the importance of distinguishing between "generativeness" and "reducibility" (link). The former is a claim about ontology -- the notion that the features of the lower level suffice to determine the features of the upper level through pathways we may not understand at all. The latter is a claim about inter-theoretic deductive relationships -- relationships between our formalized beliefs about the lower level and the feasibility of deriving the features of the upper level from these beliefs. But I argued in the earlier post that the fact that A is generated by B does not imply that A is reducible to B. 

So there seem to be two distinct ways in which J. Epstein is over-reaching here: he is assuming that agent-based models can be sufficiently detailed to reproduce complex social phenomena like civil unrest; and second, he is assuming without justification that only reductive explanations are scientifically acceptable.

Consider an example that provides an explanation of a collective behavior that has explanatory weight, that is not generative, and that probably could not be fully reproduced as an ABM.  A relevant example is Charles Perrow's analysis of technology failure as a consequence of organizational properties (Normal Accidents: Living with High-Risk Technologies). An earlier post considered these kinds of examples in more detail (link). Here is my summary of organizational approaches to the explanation of the incidence of accidents and system safety:
However, most safety experts agree that the social and organizational characteristics of the dangerous activity are the most common causes of bad safety performance. Poor supervision and inspection of maintenance operations leads to mechanical failures, potentially harming workers or the public. A workplace culture that discourages disclosure of unsafe conditions makes the likelihood of accidental harm much greater. A communications system that permits ambiguous or unclear messages to occur can lead to air crashes and wrong-site surgeries. (link)
I would say that this organizational approach is a legitimate schema for social explanation of an important effect (the occurrence of large technology failures). Further, it is not a generativist explanation; it does not originate in a simplification of a particular kind of failure and demonstrate through iterated runs that failures occur X% of the time. Rather, it is based on a different kind of scientific reasoning, based on causal analysis grounded in careful analysis and comparison of cases. Process tracing (starting with a failure and working backwards to find the key causal branches that led to the failure) and small-N comparison of cases allows the researcher to arrive at confident judgments about the causes of technology failure. And this kind of analysis can refute competing hypotheses: "operator error generally causes technology failure", "poor technology design generally causes technology failure", or even "technological over-confidence causes technology failure". All these hypotheses have defenders; so it is a substantive empirical hypothesis to argue that certain features of organizational deficiency (training, supervision, communications processes) are the most common causes of technological accidents.

Other examples from sociology could be provided as well: Michael Mann's explanation of the causes of European fascism (Fascists), George Steinmetz's explanation of variations in the characteristics of German colonial rule (The Devil's Handwriting: Precoloniality and the German Colonial State in Qingdao, Samoa, and Southwest Africa), or Kathleen Thelen's explanation of the persistence and change in training regimes in capitalist economies (How Institutions Evolve: The Political Economy of Skills in Germany, Britain, the United States, and Japan). Each is explanatory, each identifies causal factors that are genuinely explanatory of the phenomena in question, and none is generativist in Epstein's sense. These are examples drawn from historical sociology and institutional sociology; but examples from other parts of the disciplines of sociology are available as well.

I certainly believe that ABMs sometimes provide convincing and scientifically valuable explanations. The fundamentalism that I'm taking issue with here is the idea that all convincing and scientifically valuable social explanations must take this form -- a much stronger view and one that is not well supported by the practice of a range of social science research programs.

Or in other words, the over-reach of the ABM camp comes down to this: the claims of exclusivity and general adequacy of the simulation-based approach to explanation. ABM fundamentalists claim that only simulations from units to wholes will be satisfactory (exclusivity), and they claim that ABM simulations can always be designed for any problem that are generally adequate to grounding an explanation (general adequacy). Neither proposition can be embraced as a general or universal claim. Instead, we need to recognize the plurality of legitimate forms of causal reasoning in the social sciences, and we need to recognize, along with their strengths, some of the common weaknesses of the ABM approach for some kinds of problems.

2 comments:

polymath said...

This sequence of posts has been very thought provoking. Thank you.

I think your claim that "each [non-generative analysis] identifies causal factors that are genuinely explanatory of the phenomena in question" may miss the ABM point somewhat. Aristotelian physics (substances, natures, potentialities) could be described the same way but its explanatory power was very local (domain specific). Things moved towards the center of the Earth and so on -- not bad as a first approximation but not general. And not helpful in predicting the limits of its generality, where it is likely to break down.

In your technology-failure example, the causal analyses can be expected to apply to some social and organizational contexts more than others: organizations with a "culture of safety" will, we believe, be less likely to experience each kind of failure. A "culture of safety" is more-or-less measurable by survey of responsible individuals in the organization.

Those are real explanations of the phenomena, much better than nothing. But to see how little we know, just try to create an organization with a culture of safety. Do flat organizations in an entrepreneurial context get this culture in the same way as hierarchical organizations in a bureaucratic context? I don't think so. Can we determine, as an organization grows and changes, at which point in time the risks of not having a culture of safety start to get large quickly? I don't think so. If imposing a culture of safety quickly is vital, do we know the most minimal yet effective interventions for each type of organization? I don't think so.

The causal-analysis approach isn't effective at answering those questions because there is nothing in the method to discipline how interacting agents are specified. You can stop when you have something that feels like a real explanation. But only with difficulty will your theorized explanation fit with other theories. Inconsistencies between theories may be hidden not only in definitional gaps but in unspoken domain-local assumptions.

I think ABM has great potential -- perhaps not the totalizing potential as Epstein may wish for -- but only if we take its discipline seriously. Your example today of technology failure, through the lens of ABM, generalizes to a robustness property on systems (whose individuals have specified learning rules, whose organizational component has specified communication practices and decision practices, whose technology component has specified failure modes and probabilities over time). Clearly the ABM lens buys us something, while without the ABM lens we don't even see what we're missing.

Your example of last week, the creation and execution of an ad campaign, was also interesting in this light. You wrote, "It is certainly not explanatorily interesting that GM, Marketing, WXYZ, WXYZ Sales, and the public are all composed of individual actors. Instead, we want to know what it is about the circumstances facing these various social actors (the corporation, its divisions, the PR firm, ...) in virtue of which they do the things they do [...]."

But it is explanatorily interesting that the decisions of Marketing, and of WXYZ and all the other organizational entities, arise from certain communication practices between individuals and certain organizational decision practices. This is especially interesting because some ad campaigns work and others don't, and organizational entities often want to know interventions that might help theirs work more often. This is where ABM comes in. If we're content at a local level of description, the discipline of ABM isn't relevant. My point isn't about intervention per se, it's about the local limits of a theory, although attempted intervention can be where the limitations of an Aristotelian-style theory become visible to all.

Dan Little said...

Dear Polymath, Thanks for these very thoughtful comments. I understand your line of argument, and I like your qualification "not the totalizing potential as Epstein may wish for". I think your criticisms of "gross causal factors" analysis are fair, though I'm not entirely sure how a social scientist can repair this shortcoming. Does "culture of safety" admit of formulation in agent-based terms? I suppose so, in a simplifying way: attribute to each agent a variable "safety sensibility" factor (similar in logic to the "risk aversiveness" factor used in many ABMs), and then somehow see what the distribution of scenarios looks like when that factor is on average low versus high. But this is a very stark operationalization of the idea of "culture of safety"; and I suppose that someone like Perrow might argue that it's a bad theorization because it sacrifices attention to the organizational features that contribute to high or low values of this variable. My own inclination is to think that social explanations don't derive from an exact science, so the gaps that you mention are unavoidable. Anyway -- thanks for giving these two posts your attention and sharing your thoughts here. Dan