A Big Picture Overview: "From Human-Subject Experiments to Computational-Agent Experiments (and Everything in Between)"(pdf,341KB)
Catherine C. Eckel and J. Barkley Rosser, Jr. (Guest Editors), "Issues in the Methodology of Experimental Economics", Journal of Economic Behavior and Organization (JEBO)Special Issue ,
73(1), January 2010.
Abstract: Binmore and Shaked provide a controversial, scathing review of some well known human-subject experimental economics work by Ernst Fehr, Klaus Schmidt, and their collaborators. They conclude that experimental economists need to eliminate practices "that would be considered unscientific in other disciplines." Since researchers using agent-based modeling tools share many issues in common with human-subject experimenters (e.g., the need to construct rigorous informative experimental designs), it is strongly recommended that they carefully consider both the criticisms raised in this article and the rebuttal to these criticisms given by Fehr/Schmidt and Eckels/Gintis.
"Agent-Based Models and Human-Subject Experiments",
in Leigh Tesfatsion and Kenneth L. Judd (editors),
Handbook of Computational Economics, Vol. 2: Agent-Based Computational
Economics, Handbooks in Economics Series, North-Holland/Elsevier, Amsterdam,
This chapter examines the relationship between agent-based
modeling and economic decision-making experiments with human
subjects. Both approaches exploit controlled "laboratory" conditions
as a means of isolating the sources of aggregate phenomena. Research
findings from laboratory studies of human subject behavior have
inspired studies using artificial agents in "computational
laboratories" and vice versa. In certain cases, both methods have
been used to examine the same phenomenon. The focus of this chapter
is on the empirical validity of agent-based modeling approaches in
terms of explaining data from human subject experiments. We also
point out synergies between the two methodologies that have been
exploited as well as promising new possibilities.
Francesco Guala, History of Experimental Economics(pdf,121KB),
in Steven Durlauf and Lawrence Blume (Eds.), New Palgrave Dictionary of Economics, Second Edition (Palgrave-MacMillan).
John H. Kagel and Alvin E. Roth (eds.), Handbook of Experimental
Economics, Princeton University Press, Princeton, NJ, 1995.
Abstract: This book presents a comprehensive critical
survey of the results and methods of human-subject laboratory experiments in
economics. The first chapter provides an introduction to human-subject
experimental economics as a whole, while the remaining chapters provide
surveys by leading practitioners in areas of economics that have a
concentration of human-subject experimental research: public goods;
coordination problems; bargaining; industrial organization; asset markets;
auctions; and individual decision making. For more information about this
book and related topics, visit Al Roth's
Experimental Economics Handbook Site
and his website on
Game Theory and Experimental Economics.
Hyungna Oh and Timothy D. Mount, "Using Software Agents to Supplement Tests Conducted by Human Subjects"(pdf,133KB),
pp. 29-56 in H. Dawid and W. Semmler (eds.), Computational Methods in Economic Dynamics, Dynamic Modeling and Econometrics in Economics and Finance 13, Springer-Verlag, Berlin Heidelberg, 2011.
The objective of this paper is to test whether or not software agents can match the observed behavior of human subjects in laboratory tests of markets. For this purpose, one set of tests uses four software agents and two human subjects to represent six suppliers in three different market situations: no forward contracts, fixed price forward contracts, and renewable forward contracts. An identical set of tests is also conducted using software agents to represent all of the suppliers. The results show that software agents were able to
replicate the behavior of human subjects effectively in the experiments. This indicates that software agents can be used effectively in testing electricity auctions, doing additional sensitivity tests, and supplementing results obtained using human subjects.
Leigh Tesfatsion, "From Human-Subject Experiments to Computational-Agent Experiments (and Everything In Between)"(pdf,1.7MB),
Revised version (17 Feb 2011) of Plenary Address, International Economic Science Association Meeting, Arlington, VA, June 27, 2009.
Abstract:Human-subject experimentation is now a mainstream tool within economics. Computational-agent research is more recent, having emerged in parallel with the growing capability and availability of computers in the late 1980s. This talk will focus on the largely untouched span between the two methodologies. What advantages might there be to conducting economic experiments within integrated test beds involving the combined use of humans and computational agents? More generally, what advantages might there be to designing economic systems in which humans routinely interact with cognitive computational agents? Indeed, as will be elaborated in the talk, this is not science fiction; it is already on the drawing boards, exemplified by proposed new smart grid technology for undergirding the operation of electric power markets. Smart grid technology envisions artificially intelligent grid-device and trader agents acting in concert with human market operators and traders, who in turn rely in part on models of such combined systems for operational guidance.
is the official journal of the Economic Science Association. The journal serves the growing group of economists around the world who use laboratory methods to study phenomena that are difficult to observe directly in naturally occurring economic contexts. The journal publishes high quality papers in any area of experimental research in economics and in related fields such as accounting, finance, political science, and the psychology of decision-making. Readers will also find state-of-the-art theoretical work and econometric work motivated by experimental data. Lastly, the journal publishes articles with a primary focus on methodology or replication of controversial findings.
James Andreoni and John H. Miller, "Auctions with Artificial Adaptive
Agents,"Games and Economic Behavior 58 (1995), 211-221.
Jasmina Arifovic, "The Behavior of the Exchange Rate in the Genetic
Algorithm and Experimental Economics,"Journal of Political Economy
104(3), 1993, 510-541.
W. Brian Arthur, "Designing Economic Agents that Act Like Human
Agents: A Behavioral Approach to Bounded Rationality,"American
Economic Review Papers and Proceedings 81(2), 1991, 353-359.
P. J. Brewer, M. Huang, B. Nelson, and C. R. Plott, "On the Behavioral
Foundations of the Law of Supply and Demand: Human Convergence and Robot
Randomness,"Experimental Economics 5(3), 2002, 179-208.
Nicholas T. Chan, Blake LeBaron, Andrew W. Lo, and Tomaso Poggio,
"Agent-Based Models of Financial Markets: A Comparison with Experimental
Markets," Working Paper, September 5, 1999.
Bruno Contini, Roberto Leombruni, and Matteo G. Richiardi, "Exploring a New ExpAce: The Complementarities between Experimental Economics and Agent-Based Computational Economics(pdf,129KB),
Journal of Social Complexity, Vol. 3, No. 1, 2006.
This paper addresses whether experimental economics and agent-based computational economics can complement and integrate with each other. The authors argue that the answer is yes, that there are many benefits to be gained by both communities of researchers from increased interactions.
John Duffy, "Learning to Speculate: Experiments with Artificial and
Real Agents,"Journal of Economic Dynamics and Control 25(3/4),
March 2001, pages 295-319.
Abstract: This paper employs parallel experiments with
real and computational agents to explore issues originally raised by Kiyotaki
and Wright in their well-known search model of money (JPE, 1989). The
primary issue of interest is how individuals might come to accept or learn to
adopt a convention in which the particular commodity functioning as "money"
is dominated in rate of return by other assets, in the sense that it has a
higher storage cost. The key offsetting factor is anticipations
("speculation") concerning the ease with which the "money" good can be turned
over in trade for other goods that agents have a higher desire to consume.
The author shows how each type of experiment can contribute to the
experimental design and interpretation of results for the other.
Herbert Gintis, A Framework for the Unification of the Behavioral Sciences(pdf,1MB),
Behavioral and Brain Sciences, Vol. 30, 1-61.
Abstract: "The various behavioral disciplines model human behavior in distinct and incompatible ways. Yet, recent theoretical and empirical developments have created the conditions for rendering coherent the areas of overlap of the various behavioral disciplines. The analytical tools deployed in this task incorporate core principles from several behavioral disciplines. The proposed framework recognizes evolutionary theory, covering both genetic and cultural evolution, as the integrating principle of behavioral science. Moreover, if decision theory and game theory are broadened to encompass other-regarding preferences, they become capable of modeling all aspects of decision making, including those normally considered “psychological,” “sociological,” or “anthropological.” The mind as a decision-making organ then becomes the organizing principle of psychology."
D. K. Gode and S. Sunder, "Allocative Efficiency of Markets
with Zero Intelligence Traders: Market as a partial substitute for individual
Journal of Political Economy, Vol. 101, Number 1,
1993, pp. 119-137.
Abstract: Gode and Sunder report on continuous double
auction experiments with budget-constrained computational traders. They find
that market efficiency levels are close to 100 percent even when buyers
submit purely random bids and sellers submit purely random offers. The
authors conclude that the high market efficiency typically observed in
continuous double auction experiments with human subjects is due to the
structure of the auction and not to learning. Their seminal work has
highlighted an important issue now being actively pursued by many other
researchers: what are the relative roles of learning and institutional
arrangements in the determination of economic, social, and political
Marton Ivanyi, Rajmund Bocsi, Laszlo Gulyas, Vilmos Kozma, and Richard Legendi,
The Multi-Agent Simulation Suite(pdf,600KB),
AITIA International, Inc., Budapest Hungary.
"Agent-based modeling is a branch of computer simulation,
especially suited for studying complex social systems.
It models the individual, together with its imperfections
(e.g., limited cognitive or computational abilities),
its idiosyncrasies and personal interactions. The
approach builds the model from ’the bottom-up’, focusing
mostly on micro rules and seeking to understand the
emergence of macro behavior. Participatory simulation
- a branch of agent-based simulation - is a methodology
building on the synergy of human actors and artificial
agents, excelling in the training and decision-making
support areas. In participatory simulations some agents
are controlled by users, while others are software governed.
The Multi-Agent Simulation Suite is a software
package intended to enable modelers to utilize the tools
of agent-based simulation in various fields, without having
to develop heavy programming skills."
Wander Jager Marco A. Janssen,
"Using Artificial Agents to Understand Laboratory Experiments of
Common-Pool Resources with Real Agents," Chapter 6 in Janssen, M. A.
(ed.), Complexity and Ecosystem Management: The Theory and Practice of
Multi-Agent Systems, Edward Elgar Publishers, Cheltenham UK/Northampton,
Wander Jager and Marco Janssen, "The Need for and Development of Behaviourally Realistic Agents"(pdf,196KB),
pp. 36-49 in J. S. SIchman, F. Bousquet, and P. Davidsson (Eds.), MABS 2002, LNAI2581, Springer-Verlag Berlin Heidelberg, 2003.
Abstract: "In this paper we argue that simulating complex systems involving
human behaviour requires agent rules based on a theoretically rooted structure
that captures basic behavioural processes. Essential components of such a
structure involve needs, decision-making processes and learning. Such a
structure should be based on state-of-the-art behavioural theories and validated
on the micro-level using experimental or field data of individual behaviour. We
provide some experiences we had working with such a structure, which involve
the possibility to relate the results of simulations on different topics, the ease of
building in extra factors for specific research questions and the possibility to
use empirical data in calibrating the model. A disadvantage we experienced is
the lack of suiting empirical data, which necessitates in our view the combined
use of empirical and simulation research."
Robert Kurzban and Daniel Houser, "Experiments Investigating Cooperative Types in
Humans: A Complement to Evolutionary Theory
Proceedings of the National Academy of Sciences, Vol. 102, No. 5, February 1, 2005, 1803-1807.
"Unlike other species, humans cooperate in large, distantly related
groups, a fact that has long presented a puzzle to biologists. The
pathway by which adaptations for large-scale cooperation among
nonkin evolved in humans remains a subject of vigorous debate.
Results from theoretical analyses and agent-based simulations
suggest that evolutionary dynamics need not yield homogeneous
populations, but can instead generate a polymorphic population
that consists of individuals who vary in their degree of cooperativeness.
These results resonate with the recent increasing emphasis
on the importance of individual differences in understanding
and modeling behavior and dynamics in experimental games and
decision problems. Here, we report the results of laboratory
experiments that complement both theory and simulation results.
We find that our subjects fall into three types, an individual’s type
is stable, and a group’s cooperative outcomes can be remarkably
well predicted if one knows its type composition."
Sheri Markose, Jasmina Arifovic, and Shyam Sunder, "Advances in experimental and agent-based modelling: Asset markets, economic networks, computational mechanism design, and evolutionary game dynamics", Journal of Economic Dynamics and Control 31 (2007), pp. 1801-07.
Sheri Markose (Guest Editor), Editorial Overview: Special issue on Developments in Experimental and Agent-Based Computational Economics (ACE)(pdf,119KB),
Journal of Economic Interaction and Coordination,Volume 1, Number 2, December 2006.
"The objective of this paper is to test whether or not software agents can match the observed behavior of human subjects in laboratory tests of markets. For this purpose, one set of tests uses four software agents and two human subjects to represent six suppliers in three different market situations: no forward contracts; fixed price forward contracts; and renewable forward contracts. An identical set of tests is conducted using software agents to represent all suppliers. The results show that software agents were able to replicate the behavior of human subjects effectively in the experiments, and have the potential to be used effectively in testing electricity auctions, doing additional sensitivity tests, and supplementing results obtained using human subjects."
Claudia Pahl-Wostl and Eva Ebenhoh,
Heuristics to Characterise Human Behaviour in Agent-Based Models(pdf,122KB),
Working Paper, Institute of Environmental Systems Research, University of Osnabruck, Germany, downloaded 1/29/05.
Abstract: The authors pursue a pragmatic approach to the representation of human behavior in agent-based models, assuming that agents can be characterized by a set of attributes and their behavior can be described by a set of simple decision heuristics. These assumptions are tested and refined by using data from human-subject experiments describing the behaviors of players in simple resource allocation games.
Mark Pingle and Leigh Tesfatsion, "Evolution of Worker-Employer
Networks and Behaviors Under Alternative Non-Employment Benefits: An
Agent-Based Computational Study", pp. 254-283 in Anna Nagurney (ed.),
Innovations in Financial and Economic Networks, Edward Elgar
Note: The results of this ACE labor market study are
briefly compared against results from a parallel human-subject experiment.
Juliette Rouchier, "Re-Implementation of a Multi-Agent Model Aimed at
Sustaining Experimental Economic Research: The Case of Simulations with
Journal of Artificial Societies and Social
Simulation, Vol. 6(4), 2003.
Note: This paper explores replication issues for the
article by John Duffy (JEDC,2001) cited above.
Leonidas Spiliopoulos, "Human versus Computer Algorithms in Repeated Mixed Strategy Games"(pdf,1.8MB),
Munich Personal RePEc Archive (MPRA) Paper No. 6672, January 2008.
Absract: "This paper is concerned with the modeling of strategic change in humans’ behavior when
facing different types of opponents. In order to implement this efficiently a mixed experimental setup
was used where subjects played a game with a unique mixed strategy Nash equilibrium for 100 rounds
against 3 preprogrammed computer algorithms (CAs) designed to exploit different modes of play.
In this context, substituting human opponents with computer algorithms designed to exploit commonly
occurring human behavior increases the experimental control of the researcher allowing for
more powerful statistical tests. The results indicate that subjects significantly change their behavior
conditional on the type of CA opponent, exhibiting within-subjects heterogeneity, but that there
exists comparatively little between-subjects heterogeneity since players seemed to follow very similar
strategies against each algorithm. Simple heuristics, such as win-stay/lose-shift, were found to model
subjects and make out of sample predictions as well as, if not better than, more complicated models
such as individually estimated EWA learning models which suffered from overfitting. Subjects modified
their strategies in the direction of better response as calculated from CA simulations of various
learning models, albeit not perfectly. Examples include the observation that subjects randomized
more effectively as the pattern recognition depth of the CAs increased, and the drastic reduction in
the use of the win-stay/lose-shift heuristic when facing a CA designed to exploit this behavior."
Quynh Chi Trinh, Marcelo Saguan, and Leonardo Meeus, "Experience with Electricity Market Test Suite: Students Versus Computational Agents"(pdf,1.8MB),
IEEE Transactions on Power Systems, 2013, to appear.
"This paper applies two experimental economics methods (i.e., agent-based modeling and laboratory experiment) to a market test suite that is based on a fictional European wholesale electricity market. Quantitative results of generators' strategic behavior in this market context are separated between generators played by human subjects (i.e., master students) in a laboratory experiment and generators represented by computational agents in an
agent-based model. The behavior is measured through offers that students or agents make when participating in the electricity trading auction and the market outcomes under both methods are discussed in order to illustrate the difference between the behavior of human and computational agents. The paper also identifies the improvements that would need to be made
to the market test suite to allow for a more conclusive comparison in future experiments."
provides a simple setup that is explicitly designed for running classroom game theory experiments. The site supports both normal-form and extensive-form games. The comlabgame software can be downloaded and installed on personal computers.
The ComLabGames support team includes Robert A. Miller, Vesna Prasnikar, and Darko Zupanic.
The Multi-Agent Simulation Suite,
developed by Marton Ivanyi, Rajmund Bocsi, Laszlo Gulyas, Vilmos Kozma, and Richard Legendi, supports
Participatory Simulation in which some agents are controlled by users while other agents are software governed.
Charles A. Holt, Susan K. Laury, and David Lucking-Reiley,Classroom Experiments on the Internet(pdf,16KB),
Notes, January 2001.
Abstract: Classroom experiments can be easily set up to run through standard internet browsers,
which avoids the need to install special software on the students' ("client") personal computers. The
instructions, decisions, and market signals are communicated via interactive web pages, with data
stored in a database on the web server for later use in classroom discussions. The advantages of
web-based interactions (over in-class experiments or programs that run on a local area networKB) are
1) scalability to accommodate potentially large numbers of students, 2) flexible hours to save class
time for discussion, and flexible locations to allow students to connect from any personal computer
with a standard web browser. This paper surveys a number of different sites that provide classroom
applications of economics experiments. Some technical issues, of interest to those who want to
develop their own web applications, are also addressed."
Robert Goldstone, Allen Lee, and Andy Jones (Percepts and Concepts
Laboratory, Indiana University, Bloomington) are encouraging people to
participate in an ongoing on-line group experiment on resource foraging
accessible at the
Group Experimentation Environment (GEE) Project Site.
They describe the experiment as follows: "The experiment is rather
educational and engrossing. Your goal in a four-minute experiment is to pick
up as many resources as you can by moving your icon with the arrow keys. You
compete against other humans when they are available or Artificial
Intelligence Bots (programmed by Michael Roberts) when no other humans are
currently on-line. We have been beta-testing the environment locally at
Indiana University for awhile, and now feel ready to announce it to the
broader community, at least the broader ACADEMIC community at this point. If
you find any bugs or have suggestions for improvements, please email Andy
Jones or Allen Lee , and CC me
Institute for Empirical Research in Economics
(University of Zurich, Switzerland), headed by Prof.Dr. Ernst Fehr, provides
links to a variety of resources related to microeconomics and experimental
economics on its home page. Institute researchers combine insights from
modern economic theory with results from social psychology and sociology to
understand important economic phenomena. Topics stressed include the
functioning of labor markets, the organization of the modern corporation, the
private and public provision of public goods, and intertemporal choice
(Economics Department, Simon Fraser U, Canada): Learning and Adaptation, Experimental Economics, Macroeconomics, Monetary Economics, Evolutionary Game Theory, Computational Mechanism Design
(Department of Economics, University of Pittsburgh, Pennsylvania):
Incorporation of learning in computational economic models; Using genetic
algorithms to model how agents learn and adaptively update their forecasts;
Parallel experiments with real and computational agents.
(Department of Economics, University of Massachusets, Amherst): Agent-based
evolutionary game dynamics; Evolution of strong reciprocity; Moral economy of
communities; Evolution of social norms; Experimental games.
(Graduate School of International Economics and Finance, Brandeis
University, Waltham, Massachusetts): Quantitative dynamics of
interacting systems of adaptive agents, and how these systems
replicate real-world phenomena; Behavior of traders in financial
markets; Nonlinear behavior of financial and macroeconomic time
series; Parallel experiments with real and computational agents.
M. Utku Ünver
(Department of Economics, Boston College): Social learning
in market games using genetic algorithms; Experimental economics; Game
theory; Two-sided and one-sided matching; Auctions; Parallel experiments with
real and computational agents.