ACE Research Area:
Presentation and Evaluation Guidelines
for Agent-Based Models

Last Updated: 3 March 2024

Site Developed By: Leigh Tesfatsion Professor Emerita of Economics Courtesy Research Professor of     Electrical & Computer Engineering Heady Hall 260 Iowa State University Ames, Iowa 50011-1054 https://faculty.sites.iastate.edu/tesfatsi tesfatsi AT iastate.edu ACE Website

Table of Contents: Basic Issues Introductory Presentation and Evaluation Readings Additional Presentation and Evaluation Readings Community Engagement Research Software Support

Site Disclaimer:

The number of research articles dedicated to the presentation and evaluation of ACE/ABM/CAS models has now grown so large that it is no longer possible to provide a comprehensive list. Below are annotated links to materials published between 2010 and 2017 that visitors might still find useful and/or of historical interest.

Basic Issues

• How to present agent-based models to external parties in a transparent and compelling manner?
• How to report findings from agent-based studies to external parties for evaluation and possible use?
• Guidelines or standardized protocols?
• Universal applicability or specialized for particular purposes?
• Can agent-based models facilitate improved communication between citizens and scientists?

Introductory Presentation and Evaluation Readings

• Li An, "Modeling Human Decisions in Coupled Human and Natural Systems: Review of Agent-Based Models" (pdf,713KB), Ecological Modelling, Vol. 229, 24 March 2012, 35-30.

  Abstract: Coupled human and natural systems (CHANS) manifest various complexities such as heterogeneity, nonlinearity, feedback, and emergence. Humans play a critical role in affecting such systems and in giving rise to various environmental consequences, which may in turn affect future human decisions and behavior. In light of complexity theory and its application in CHANS, this paper reviews various decision models used in agent based simulations of CHANS dynamics, discussing their strengths and weaknesses. This paper concludes by advocating development of more process-based decision models as well as protocols or architectures that facilitate better modeling of human decisions in various CHANs.

• Hugues Bersini, "UML for ABM" (html), Journal of Artificial Societies and Social Simulation 15(1)9, 2012, electronic.

  Article Summary (Paraphrased from Author Abstract): Although the majority of researchers interested in ABM increasingly agree that the most natural way to program their models is to adopt object-oriented (OO) practices, UML diagrams are still largely absent from their publications. In the last 15 years, however, UML has become the de facto standard for graphical visualization of software development. The thirteen diagrams comprising the UML core have many universally accepted virtues. Most importantly, these UML diagrams provide a level of abstraction higher than that offered by OO programming languages (Java, C++, Python, .Net ...). This abstraction layer encourages researchers to spend more time on modeling rather than on programming. This paper begins by presenting and explaining the four most commonly used (and useful) UML diagrams: namely, class, sequence, state, and activity diagrams. Next, for concrete illustration, detailed UML analyses are presented for several well known ABM models, including: the Schelling segregation model, a spatial evolutionary game, and a continuous double auction market model.

• Andrew Collins, Mikel Petty, Daniele Vernon-Bido, and Solomon Sherfey, "A Call to Arms: Standard for Agent-Based Modeling and Simulation" (html), Journal of Artificial Societies and Social Simulation 18(3), 2015, electronic.

  Abstract: "As agent-based modeling and simulation matures as a methodology, a discussion of standards applicable to it becomes increasingly important. Descriptive standards for agent-based models, such as the Overview, Design concepts, and Details protocol and agent-based extensions to the Unified Modeling Language, have already begun to emerge. Software tools for implementing such models, such as Netlogo and Repast Simphony, are increasingly well-known and have the potential to become de facto standards among the wider scientific community for agent-based simulation. Based on the findings of a series of workshops that brought together experts throughout the modeling and simulation community, we argue that agent-based modeling and simulation is no different from the other emerging technical subjects in the sense that standards, both existing and new, may be applicable to it, and that the community should both adopt existing standards that are relevant and exploit the already existing standards processes and organizations to develop new ones."

• Steven L. Franconeri, Lace M. Padilla, Priti Shah, Jeffrey M. Zacks, and Jessica Hullman, "The Science of Visual Data Communication: What Works" (pdf,4.3KB), Psychological Science in the Public Interest, Vol. 22(30), 2021, 110-161.

  Abstract: "Effectively designed data visualizations allow viewers to use their powerful visual systems to understand patterns in data across science, education, health, and public policy. But ineffectively designed visualizations can cause confusion, misunderstanding, or even distrust — especially among viewers with low graphical literacy. We review research-backed guidelines for creating effective and intuitive visualizations oriented toward communicating data to students, co-workers, and the general public. We describe how the visual system can quickly extract broad statistics from a display, whereas poorly designed displays can lead to misperceptions and illusions. Extracting global statistics is fast, but comparing between subsets of values is slow. Effective graphics avoid taxing working memory, guide attention, and respect familiar conventions. Data visualizations can play a critical role in teaching and communication, provided that designers tailor those visualizations to their audience."

• Volker Grimm, Jacqueline Augusiak, Andreas Focks, Béatrice M. Frank, Faten Gabsi, Alice S.A. Johnston, Chun Liu, Benjamin T. Martin, Mattia Meli, Viktoriia Radchuk, Pernille Thorbek, and Steven F. Railsback, "Towards Better Modelling and Decision Support: Documenting Model Development, Testing, and Analysis using TRACE" (pdf,806KB), Ecological Modelling, Vol. 280, 2014, 129-139. See also here (pdf,3MB) for a detailed TRACE application.

  Article Summary: TRACE = TRAnsparent and Comprehensive model Evaludation. TRACE is a proposed set of protocols for documenting the steps taken to design, evaluate, and validate an agent-based model (ABM). Although originally developed to support environmental decision-making, it is now being promoted for general ABM purposes. Here is the article abstract:

  "The potential of ecological models for supporting environmental decision making is increasingly acknowledged. However, it often remains unclear whether a model is realistic and reliable enough. Good practice for developing and testing ecological models has not yet been established. Therefore, TRACE, a general framework for documenting a model’s rationale, design, and testing was recently suggested. Originally TRACE was aimed at documenting good modelling practice. However, the word ‘documentation’ does not convey TRACE’s urgency. Therefore, we re-define TRACE as a tool for planning, performing, and documenting good modelling practice. TRACE documents should provide convincing evidence that a model was thoughtfully designed, correctly implemented, thoroughly tested, well understood, and appropriately used for its intended purpose."

• Volker Grimm, Uta Berger, Donald L. DeAngelis, J. Gary Polhill, Jarl Giske, and Steven F. Railsback, "The ODD Protocoll: A Review and First Update" (ODD Paper,pdf,217KB), (ODD Template,pdf,128KB), Ecological Modelling, Vol. 221, 2010, 2760-2768. For an example of an agent-based model carefully presented in accordance with the updated ODD protocol, see here (pdf,1M)

  Article Summary: ODD = Overview, Design concepts, and Details. ODD is a proposed set of protocols for the description of an agent-based model (ABM). Although originally developed for ecological ABMs, it is now being promoted for ABMs in general. Here is the article abstract:

  "The `ODD' (Overview, Design concepts, and Details) protocol was published in 2006 to standardize the published descriptions of individual-based and agent-based models (ABMs). The primary objectives of ODD are to make model descriptions more understandable and complete, thereby making ABMs less subject to criticism for being irreproducible. We have systematically evaluated existing uses of the ODD protocol and identified, as expected, parts of ODD needing improvement and clarification. Accordingly, we revise the definition of ODD to clarify aspects of the original version and thereby facilitate future standardization of ABM descriptions. We discuss frequently raised critiques in ODD but also two emerging, and unanticipated, benefits: ODD improves the rigorous formulation of models and helps make the theoretical foundations of large models more visible. Although the protocol was designed for ABMs, it can help with documenting any large, complex model, alleviating some general objections against such models."

• Birgit Müller, et al., "Standardised and Transparent Model Descriptions for Agent-Based Models: Current Status and Prospects" (pdf,431KB), Environmental Modelling, Vol. 55, 2014, 156-163.

  Abstract: Agent-based models are helpful to investigate complex dynamics in coupled human-natural systems. However, model assessment, model comparison and replication are hampered to a large extent by a lack of transparency and comprehensibility in model descriptions. In this article we address the question of whether an ideal standard for describing models exists. We first suggest a classification for structuring types of model descriptions. Secondly, we differentiate purposes for which model descriptions are important. Thirdly, we review the types of model descriptions and evaluate each on their utility for the purposes. Our evaluation finds that the choice of the appropriate model description type is purposedependent and that no single description type alone can fulfil all requirements simultaneously. However, we suggest a minimum standard of model description for good modelling practice, namely the provision of source code and an accessible natural language description, and argue for the development of a common standard.

• Leigh Tesfatsion, "Modeling Economic Systems as Locally-Constructive Sequential Games" (WP,pdf,680KB), (Slides,pdf,1MB), Journal of Economic Methodology, Vol. 4, Issue 4, 2017, 384-409.

  Abstract: Real-world economies are open-ended dynamic systems consisting of heterogeneous interacting participants. Human participants are decision-makers who strategically take into account the past actions and potential future actions of other participants. All participants are forced to be locally constructive, meaning their actions at any given time must be based on their local states; and participant actions at any given time affect future local states. Taken together, these essential properties imply real-world economies are locally-constructive sequential games. This paper discusses a modeling approach, agent-based computational economics (ACE), that permits researchers to study economic systems from this point of view. ACE modeling principles and objectives are first concisely presented and explained. The remainder of the paper then highlights challenging issues and edgier explorations that ACE researchers are currently pursuing.

• Leigh Tesfatsion, "Elements of Dynamic Economic Modeling: Presentation and Analysis" (WP,pdf,634KB), Eastern Economic Journal 43(2), 2016, 192-216. doi:10.1057/eej.2016.2

  Abstract: The primary goal of these introductory notes is to promote the clear presentation and rigorous analysis of dynamic economic models, whether expressed in equation or agent-based form. A secondary goal is to promote the use of initial-value state space modeling with its regard for historical process, for cause leading to effect without the external imposition of global coordination constraints on agent actions. Economists who claim to respect individual rationality should not be doing for their modeled economic agents what in reality these agents must do for themselves.

• Leigh Tesfatsion, "Economic Systems as Constructively Rational Games: Oh, the Places We Could Go!" (Talk Slides,1MB), Keynote Address, Duke Forest Conference, Durham, NC, Nov 11-13, 2016.

  Abstract: The first part of this presentation provides a brief but careful overview of agent-based computational economics (ACE), the computational modeling of economic processes (including whole economies) as open-ended dynamic systems of interacting agents. The potential applicability of ACE modeling for the study of economic systems is illustrated by means of an ACE macroeconomic study. Several key methodological issues are next highlighted: namely, empirical validation; standardized measures of model development; and standardized presentation protocols. The final part of the presentation considers how ACE modeling tools could be used to undertake edgier creative explorations of issues critical for the operation of real-world economic systems.

• Robert Wallace, Amy Geller, and V. Ayano Ogawa (Eds.), Assessing the Use of Agent-Based Models for Tobacco Regulation, 2015 Report, Institute of Medicine of the National Academies, Washington, D.C. Available by title search from the National Academies Press.

  Note: For a thoughtful constructive discussion of presentation and evaluation issues for agent-based models developed for policy use, see: (i) Chapter 3: "Building Effective Models to Guide Policy Decision Making" (pdf,275KB); (ii) Chapter 4: "An Evaluation Framework for Policy-Relevant Agent-Based Models" (pdf,309KB); and (iii) Chapter 6: "Data and Implementation Needs for Computational Modeling for Tobacco Control" (pdf,270KB).

Additional Evaluation and Presentation Readings

• Volker Grimm and Steven F. Railsback, "Designing, Formulating, and Communicating Agent-Based Models" (pdf,392KB), in A. J. Heppenstall, A. T. Crooks AT, L. M. See, and M. Batty (Eds.), Agent-based Models of Geographical Systems, Springer, Dordrecht, 2012, 361-378.

  Abstract: This paper discusses two approaches for the design, formulation, and communication of agent-based models: The Overview, Design concepts, and Details (ODD) protocol, which provides design patterns for agent-based model (ABM) development and for the description of ABMs in publications; and Pattern-Oriented Modeling (POM), a set of strategies for using observed system patterns to ensure that an ABM captures the salient aspects of the system.

• Volker Grimm, Gary Polhill, and Julia Touza, "Documenting Social Simulation Models: The ODD Protocol as a Standard" (pdf,278KB), in B. Edwards and R. Meyer (Eds.), Simulating Social Complexity, Understanding Complex Systems, Springer-Verlag Berlin Heidelberg, 2013.

  Abstract: This chapter discusses the importance of documenting simulation models and encourages the use of the ODD protocol to facilitate this documentation. An annotated example of the use of ODD is provided for concrete illustration. For those interested in applying the ODD protocol, the authors have provided an ODD Template (Word doc).

• Amelie Schmolke, Pernille Thorbek, Donald L. DeAngelis, and Volker Grimm, "Ecological Models Supporting Environmental Decision Making: A Strategy for the Future" (pdf,599KB), Trends in Ecology and Evolution, Vol. 25, No. 8, 2010, 479-486.

  Abstract: Ecological models are important for environmental decision support because they allow the consequences of alternative policies and management scenarios to be explored. However, current modeling practice is unsatisfactory. A literature review shows that the elements of good modeling practice have long been identified but are widely ignored. The reasons for this might include lack of involvement of decision makers, lack of incentives for modelers to follow good practice, and the use of inconsistent terminologies. As a strategy for the future, we propose a standard format for documenting models and their analyses: transparent and comprehensive ecological modeling (TRACE) documentation. This standard format will disclose all parts of the modeling process to scrutiny and make modeling itself more efficient and coherent.

Community Engagement Research

• Annotated links to research on Iterative Participatory Modeling (IPM) (html)

  Abstract: The Iterative Participatory Modeling (IPM) approach envisions multidisciplinary researchers and stakeholders engaging together in repeated looping through a four-stage cycle: field study and data analysis; role-playing games; agent-based model design and implementation; and intensive computational experiments. The new aspect of IPM relative to other participatory modeling approaches is the emphasis on modeling as an open-ended collaborative learning process. The modeling objective is to help stakeholders manage complex problems over time through a continuous learning process rather than to attempt the delivery of a definitive problem solution.

• Baruch Fischhoff, Noel T. Brewer, and Julie S. Downs (Eds.), Communicating Risks and Benefits: An Evidence-Based User's Guide (pdf,3MB), USDA Food and, US Department of Health and Human Services, August 2011.

  Abstract: "The risk communication research literature is large and diverse, including results from many contributing disciplines (e.g., psychology, decision science, sociology, communications) and a wide range of applications. Unfortunately, the norms of academic research make it inaccessible to outsiders, filling it with jargon and technical details. ... This guide seeks to fill that gap, making evidence-based communication possible. The chapters that follow cover key topics in risk communication, focusing on three questions: (1) What does the science say about that aspect of human behavior?; (2) What are the practical implications of those scientific results?; (3) How can one evaluate communications based on that science?"

• Dan M. Kahan, "Climate-Science Communication and the Measurement Problem" (pdf,6MB), Political Psychology, Vol. 36, 2015, 1–43.

  Abstract: "This article examines the science-of-science-communication measurement problem. In its simplest form, the problem reflects the use of externally invalid measures of the dynamics that generate cultural conflict over risk and other policy-relevant facts. But at a more fundamental level, the science-of-science-communication measurement problem inheres in the phenomena being measured themselves. ... The article shows how appropriately designed "science comprehension" tests - one general and one specific to climate change - can be used to measure individuals’ reasoning proficiency as collective-knowledge acquirers independently of their reasoning proficiency as cultural-identity protectors."

• Ellen Peters, "Why We Don't Believe Science: A Perspective from Decision Psychology" (pdf,1.8MB), Webinar, The OSU Climate Change Outreach Team, Ohio State University, February 11, 2016.

  Abstract: People make hundreds of decisions every day, weighing risks and benefits and coming to conclusions. In a perfect world, people are objective when they perceive risks and make decisions in climate and other domains. But psychological research suggests that this is not always how the human mind works. In this webinar Ellen Peters (Professor of Psychology, Ohio State University) considers examples in climate change and other politically charged domains, and presents evidence-based strategies that can help improve communication

Software Support

• Gary Polhill and Nick Gotts, "Semantic Model Integration: An Application for OWL" (pdf,150KB), James Hutton Institute, Aberdeen, Scotland, circa 2010.

  Abstract: Many agent-based models, especially those embedded in environmental systems, are formed of coupled submodels each providing a component of the whole model. Coupled socio-environmental modelling is not trivial, with many authors reporting a need to ensure that the resulting model has a consistent, integrated ontology. The matter can be expected to be no different in the specific case of coupling agent-based and environmental models. We argue that representing the state and structure of the simulation at each time step using OWL ontologies can at least provide some automated consistency checks for a proposed coupling that would not be possible using black-box model coupling frameworks. We further argue that algorithmic conflicts in coupled modelling exercises are an additional problem for semantic integration of models in comparison with semantic integration of data.

Copyright © Leigh Tesfatsion. All Rights Reserved.