ACE Research Area:
Learning and the Embodied Mind

Last Updated: 25 May 2013

Site Maintained By: Leigh Tesfatsion Department of Economics Iowa State University Ames, Iowa 50011-1070 http://www.econ.iastate.edu/tesfatsi/ tesfatsi AT iastate.edu Online ACE Course (Self-Study eBook) ACE Website

Table of Contents: Key Issues Featured Introductory Materials Other Introductory Materials General Readings Software, Toolkits, and Demos Resource Sites and Groups Some Early Individual Researchers

Key Issues

• How to put to rest the misconception, still being expressed by some mainstream economists and bloggers uninformed about the powerful capabilities of modern software, that computational agents are incapable of engaging in anticipatory learning and intertemporal planning?
• Possible modes of learning for computational agents, such as reactive reinforcement learning, evolutionary learning, connectionist learning, and anticipatory learning (e.g., temporal-difference methods permitting the learning of dynamic programming value functions).
• If you had to specify learning algorithms to be used by traders trying to survive and prosper in a decentralized market economy, where the learning algorithms only make use of information and computational capabilities that the traders could reasonably be assumed to possess, and you are not permitted simply to impose equilibrium or other coordination conditions on the traders a priori, how would you go about it?
• Expert AI smarts versus survival Alife smarts:
  • Are "minds" disembodied logical reasoning devices together with a store of information (logic machine + filing cabinet)?
  • Or are "minds" better viewed as controllers for embodied activity, behavior-producing systems that have been evolutionarily adapted for fast and flexible operation in real-time mode? (As noted by Rodney Brooks, "elephants don't play chess"!)

Featured Introductory Materials

• Andrew G. Barto (University of Massachusetts - Amherst) has written a survey (Art.#1604) for Scholarpedia on Temporal Difference (TD) Learning.

  Abstract: TD learning is a a widely used approach to the algorithmic representation of anticipatory learning for computational agents that involves the use of changes (or differences) in predictions over successive time steps to update current predictions. One of the most commonly used forms of TD learning is the "Q-learning" algorithm developed by Watkins (1989) for the estimation and successive updating of dynamic programming value functions over time. For a detailed introduction to TD learning, see Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 1998 (eBook Version, html).

• Martin V. Butz, Anticipatory Learning Classifier Systems, Kluwer Academic Publishers, Dordrecht, the Netherlands, 2002. ISBN: 0-792-36730-7.

  Abstract: From the publisher: "(This book) describes the state of the art of anticipatory learning classifer systems - adaptive rule learning systems that autonomously build anticipatory environmental models. An anticipatory model specifies all possible action-effects in an environment with respect to given situations. It can be used to simulate anticipatory adaptive behavior."

• Vincent Cheung and Kevin Cannon, Introduction to Neural Networks, (pdf,1.5MB)

  Abstract: This slide presentation is a superb introduction to artificial neural networks.

• John Holland, "Genetic Algorithms" (html), Scientific American, Vol. 267, July 1992, pp. 66-72.

  Abstract: The genetic algorithm (GA) is a search technique inspired by the evolutionary effectiveness of mutation and differential reproduction. Standard learning algorithms ask: "Which strategy should I choose from this known domain of possible strategies?" In contrast, the GA permits agents to construct and test out new strategies through recombinations and random (or directed) mutations of currently known strategies.

• William B. Powell, "Perspectives of Approximate Dynamic Programming" (pdf,421KB), Annals of Operations Research, February 2012 (online).

  Abstract: "Approximate dynamic programming has evolved, initially independently, within operations research, computer science and the engineering controls community, all searching for practical tools for solving sequential stochastic optimization problems. More so than other communities, operations research continued to develop the theory behind the basic model introduced by Bellman with discrete states and actions, even while authors as early as Bellman himself recognized its limits due to the "curse of dimensionality" inherent in discrete state spaces. In response to these limitations, subcommunities in computer science, control theory and operations research have developed a variety of methods for solving different classes of stochastic, dynamic optimization problems, creating the appearance of a jungle of competing approaches. In this article, we show that there is actually a common theme to these strategies, and underpinning the entire field remains the fundamental algorithmic strategies of value and policy iteration that were first introduced in the 1950’s and 60’s."

• John Seiffertt and Donald C. Wunsch II, "Higher Order Neural Network Architectures for Agent-Based Computational Economics and Finance" (IGI Site), In Machine Learning: Concepts, Methodologies, Tools and Applications, ed. Information Resources Management Association, USA, 219-233 (2012). doi:10.4018/978-1-60960-818-7.ch207

  Abstract: "As the study of agent-based computational economics and finance grows, so does the need for appropriate techniques for the modeling of complex dynamic systems and the intelligence of the constructive agent. These methods are important where the classic equilibrium analytics fail to provide sufficiently satisfactory understanding. In particular, one area of computational intelligence, Approximate Dynamic Programming, holds much promise for applications in this field and demonstrates the capacity for artificial Higher Order Neural Networks to add value in the social sciences and business. This chapter provides an overview of this area, introduces the relevant agent-based computational modeling systems, and suggests practical methods for their incorporation into the current research."

• Yoav Shoham, Rob Powers, and Trond Grenager, "If Multi-Agent Learning is the Answer, What is the Question?" (pdf,162KB), Artificial Intelligence 171 (2007), 365-377.

  Abstract: "The area of learning in multi-agent systems is today one of the most fertile grounds for interaction between game theory and artificial intelligence. We focus on the foundational questions in this interdisciplinary area, and identify several distinct agendas that ought to, we argue, be separated. The goal of this article is to start a discussion in the research community that will result in firmer foundations for the area."

• Bill Smart, "Reinforcement Learning: A User's Guide" (pdf,430KB)

  Abstract: This delightful ppt tutorial takes the reader through a detailed tour of modern reinforcement learning techniques, with a stress on temporal-difference methods and technical implementation issues at a fairly advanced level.

• Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, The MIT Press, Cambridge, MA, 1998 (eBook Version, html).

  Abstract: A clear cogent introduction to reinforcement learning with a particular stress on anticipatory learning (e.g., Q-learning, Temporal-Difference methods). Perhaps the best-known introductory book on reinforcement learning.

• Leigh Tesfatsion, Learning Algorithms: Illustrative Examples (pdf,860KB)

  Abstract: This ppt tutorial provides a brief overview of various types of learning methods currently being used by agent-based modelers in theoretical work and in applied industry applications. Covered learning methods include: reactive reinforcement learning (e.g. Roth-Erev RL); belief-based learning (e.g. fictitious play, Camerer/Ho's EWA algorithm); anticipatory learning (e.g. Q-learning); evolutionary learning (e.g. genetic algorithms); and connectionist learning (e.g. artificial neural networks).

• Leigh Tesfatsion, "Modeling Behavior, Learning, and Social Interactions in Dynamic Economic Systems: An Agent-Based Computational Approach to Behavioral Economics" (pdf,306KB).

  Abstract: This ppt tutorial discusses how core behavioral economics concerns (human cognition, learning, and social interaction) might be constructively modeled for dynamic markets using powerful new computer capabilities -- in particular, agent-oriented programming.

Other Introductory Materials

• David F. Batten, Discovering Artificial Economics, Westview Press, 2000, Chapters 1-5.

  NOTE: The Batten book is unfortunately out of print. However, a pdf file for the entire Batten book (including figures) can be accessed here (pdf,17MB).

• Valentino Braitenberg, Vehicles: Experiments in Synthetic Psychology, The MIT Press, Cambridge, MA, 1994. 0-262-52112-1 (paperback).

  Abstract: An AI classic. From the back cover: "These imaginative thought experiments are the inventions of one of the world's emminent brain researchers. They are `vehicles,' a series of hypothetical self-operating machines that exhibit increasingly intricate if not always successful or civilized `behavior.' Each of the vehicles in the series incorporates the essential features of all the earlier models, and along the way they come to embody aggression, love, logic, manifestations of foresight, concept formation, creative thinking, personality, and free will. In a section of extensive biological notes, Braitengberg locates many elements of his fantasy in current brain research."

• Colin F. Camerer, George Loewenstein, and Matthew Rabin, Advances in Behavioral Economics, Princeton University Press, Princeton, NJ, 776pp., 2003. The introductory chapter by Camerer and Loewenstein titled Behavioral Economics: Past, Present, Future is particularly recommended.

  Abstract: From the publisher: "Behavioral economics uses facts, models, and methods from neighboring sciences such as psychology, sociology, anthropology, and biology to establish descriptively accurate findings about human cognitive ability and social interaction and to explore the implications of these findings for economic behavior. ... Twenty years ago, behavioral economics did not exist as a field. Most economists were deeply skeptical--even antagonistic--toward the idea of importing insights from psychology into their field. Today, behavioral economics has become virtually mainstream. It is well represented in prominent journals and top economics departments, and behavioral economists, including several contributors to this volume, have garnered some of the most prestigious awards in the profession. This book assembles the most important papers on behavioral economics published since around 1990. Among the 25 articles are many that update and extend earlier foundational contributions, as well as cutting-edge papers that break new theoretical and empirical ground."

  Colin F. Camerer is Rea A. and Lela G. Axline Professor of Business Economics at the California Institute of Technology. He is the author of "Behavioral Game Theory "(Princeton). George Loewenstein is Professor of Economics and Psychology at Carnegie Mellon University. Matthew Rabin, Professor of Economics at the University of California, Berkeley, received the John Bates Clark Medal of the American Economics Association for 2001.

• Andy Clark, Being There: Putting Brain, Body, and World Together Again, MIT Press, 308 pp., January 1998, ISBN 0-262-53156-9.

  Abstract: From the publisher: "Brain, body, and world are united in a complex dance of circular causation and extended computational activity. In Being There, Andy Clark weaves these several threads into a pleasing whole and goes on to address foundational questions concerning the new tools and techniques needed to make sense of the emerging sciences of the embodied mind. Clark brings together ideas and techniques from robotics, neuroscience, infant psychology, and artificial intelligence. He addresses a broad range of adaptive behaviors, from cockroach locomotion to the role of linguistic artifacts in higher-level thought."

  Note: See Richard Menary (Ed.), The Extended Mind (MIT Press, June, 2010) for overview and discussion (pro and con) of the Clark/Chalmers embodied/extended mind hypothesis.

• Daniel Crevier, AI: The Tumultuous History of the Search for Artificial Intelligence, Basic Books, 1993.

• Stan Franklin, Artificial Minds, The MIT Press, Cambridge, MA, 1997 (paperback).

  Abstract: From the back cover: "Stan Franklin is the perfect tour guide through the contemporary interdisciplinary matrix of artificial intelligence, cognitive science, cognitive neuroscience, artificial neural networks, artificial life, and robototics that is producing a new paradigm of mind. Along the way, Franklin makes the case for a perspective that rejects a rigid distinction between mind and non-mind in favor of a continuum from less to more mind."

• Roman Frydman and Edmund S. Phelps (Eds.), Rethinking Expectations: The Way Forward for Macroeconomics, Princeton University Press, January 2013.

  Abstract: "This book originated from a 2010 conference marking the fortieth anniversary of the publication of the landmark `Phelps volume,' Microeconomic Foundations of Employment and Inflation Theory, a book that is often credited with pioneering the currently dominant approach to macroeconomic analysis. However, in their provocative introductory essay, Roman Frydman and Edmund Phelps argue that the vast majority of macroeconomic and finance models developed over the last four decades derailed, rather than built on, the Phelps volume's "microfoundations" approach. Whereas the contributors to the 1970 volume recognized the fundamental importance of according market participants' expectations an autonomous role, contemporary models rely on the rational expectations hypothesis (REH), which rules out such a role by design. The financial crisis that began in 2007, preceded by a spectacular boom and bust in asset prices that REH models implied could never happen, has spurred a quest for fresh approaches to macroeconomic analysis. While the alternatives to REH presented in Rethinking Expectations differ from the approach taken in the original Phelps volume, they are notable for returning to its major theme: understanding aggregate outcomes requires according expectations an autonomous role. In the introductory essay, Frydman and Phelps interpret the various efforts to reconstruct the field -- some of which promise to chart its direction for decades to come."

• Roman Frydman and Edmund S. Phelps (Eds.), "Individual Forecasting and Aggregate Outcomes: `Rational Expectations' Examined", Cambridge University Press, New York, NY, 1983.

  Abstract: The authors in this volume examine the implications for aggregate economic outcomes of the postulate that individual agents in a changing economic environment "model" the models of the other agents on which their own behavior depends. A common theme is that individual rationality in the sense of this postulate does not guarantee the convergence of agent beliefs to the "correct" model, nor does it guarantee the ultimate equilibrium coordination of the agents' plans. The relevance and coherence of the "rational expectations" hypothesis is examined on the basis of these findings.

• Douglass C. North, "Economics and Cognitive Science" (pdf,18KB), Working Paper, Washington University at St. Louis, 1996.

  Abstract: This paper focuses on a key unresolved puzzle (also addressed by Andy Clark): How do humans evolve "scaffolding" (internal belief systems and external institutions) to reduce the uncertainty coming from the strategic interaction of human beings in economic, political, and social market situations? Douglass North is the 1993 recipient of the Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel.

• Vernon L. Smith, Rationality in Economics: Constructivist and Ecological Forms (Amazon Reviews,html), Cambridge University Press, New York, 2008.

  Abstract: "The principal findings of experimental economics are that impersonal exchange in markets converges in repeated interaction to the equilibrium states implied by economic theory, under information conditions far weaker than specified in the theory. In personal, social, and economic exchange, as studied in two-person games, cooperation exceeds the prediction of traditional game theory. This book relates these two findings to field studies and applications and integrates them with the main themes of the Scottish Enlightenment and with the thoughts of F. A. Hayek."

  Vernon L. Smith, Nobel Laureate in Economics in 2002, holds the George L. Argyros Endowed Chair in Finance and Economics at Chapman University (Orange, CA).

General Readings

• Peter S. Albin (Edited and with an Introduction by Duncan K. Foley), Barriers and Bounds to Rationality: Essays on Economic Complexity and Dynamics in Interactive Systems, Princeton University Press, Princeton, NJ, 1998.

  Abstract: From the Preface: "(The subject of this book is) rigorous analysis of an advanced economy's connective and supportive structures and the informational, evolutionary, and adaptive processes that occur within them... an automata-theoretic design for the study of complex economic structures."

• Linda Argote and Dennis Epple, "Learning Curves in Manufacturing,", Science 247(4945), 1990, 920-924.

  Abstract: "Large increases in productivity are typically realized as organizations gain experience in production. These `learning curves' have been found in many organizations. Organizations vary considerably in the rates at which they learn. Some organizations show remarkable productivity gains, whereas others show little or no learning. Reasons for the variation observed in organizational learning curves include organizational `forgetting,' employee turnover, transfer of knowledge from other products and other organizations, and economies of scale."

• Jasmina Arifovic and John Ledyard, "Scaling Up Learning Models in Public Good Games", Journal of Public Economic Theory 6(2), 2004, 203-238.

  Abstract: The authors use simulations to study the performance of three learning rules (reinforcement learning, experience-weighted attraction learning, and individual evolutionary learning) under three different Groves-Ledyard mechanisms for determining the provision of a public good. They conclude that the individual evolutionary learning rule outperforms the other two learning rules, a finding they attribute to its ability to selectively remember and discard history.

• W. Brian Arthur, "Complexity in Economic Theory: Inductive Reasoning and Bounded Rationality," American Economic Review 84 (1994), 406-411.

• Rogert Aunger (ed.), Darwinizing Culture: The Status of Memetics as a Science, Oxford University Press, UK, 2000.

  Abstract: From the Foreward by Daniel Dannett: "...the point of this book is not to ensure that the meme flourishes, but to ensure that if it does, it ought to. It works towards this worthy end by creating a landmark, a fixed point not of doctrine but of evidence and methods, some shared acknowledgement among some leading proponents and critics about how the issues ought to be addressed."

• Jerome H. Barkow, Leda Cosmides, and John Tooby, The Adapted Mind: Evolutionary Psychology and the Generation of Culture, Oxford University Press, 1992, ISBN: 0-19-510107-3.

• Mark H. Bickhard, "Interactivism: A Manifesto" (pdf,35KB), Working Paper, Cognitive Science, Lehigh University, Bethlehem, PA, accessed 1/10/05.

  Abstract: "At its broadest level, interactivism involves a commitment to a strict naturalism. ... Closely related to this naturalism is a process metaphysics: the fundamental nature of the world is the organization of processes. ... Interactivism shares with Piaget's genetic epistemology a pragmatic commitment to process and action as the proper framework for modeling mental phenomena. It shares the entailment from an action base to a constructivism -- the only way that action systems can be created is by construction; action systems cannot be created by passive processes such as transduction or induction. But interactivism differs strongly from Piaget in giving a central (though far from exclusive) importance to processes of variational construction and selection." The point of the manifesto is to outline and argue for such a framework of assumptions.

• Thomas Brenner, "Agent-Learning Representation: Advice on Modelling Economic Learning", in Leigh Tesfatsion and Kenneth L. Judd (editors), Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics, Handbooks in Economics Series, North-Holland/Elsevier, Amsterdam, 2006.

  Abstract: This chapter presents an overview of the existing learning models in the economics literature. Furthermore, it discusses the choice of models that should be used under various circumstances and how adequate learning models can be chosen in simulation approaches. It gives advice for using the many existing models and selecting an appropriate model for each application.

• Rodney A. Brooks, Cambrian Intelligence: The Early History of the New AI, The MIT Press, Cambridge, MA, 1999. ISBN 0-262-02468-3 (paperback).

  Abstract: From the publisher: "This book represents Brooks's initial forumulation of and contributions to the development of the behavior-based approach to robotics. It presents all of the key philosophical and technical ideas that put this `bottom up' approach at the forefront of current research in not only AI but all of cognitive science."

• Colin Camerer, Behavioral Game Theory: Experiments in Strategic Interaction, Princeton University Press, Princeton, NJ, 2003.

  Abstract: From the publisher: "Game theory, the formalized study of strategy, began in the 1940s by asking how emotionless geniuses should play games, but ignored until recently how average people with emotions and limited foresight actually play games. This book marks the first substantial and authoritative effort to close this gap."

• Edmund Chattoe, "Just How (Un)realistic are Evolutionary Algorithms as Representations of Social Processes?" (html), Journal of Artificial Societies and Social Simulation 1:3 (1998). [electronic journal]

• Shu-Heng Chen, "Fundamental Issues in the Use of Genetic Programming in Agent-Based Computational Economics" (pdf,154KB), Working Paper, AI-Econ Research Center, National Chengchi University, Taipei, Taiwan, 2001.

  Abstract: This paper provides a review of some fundamental issues of the applications of genetic programming to agent-based computational economics. The issues under review cover four aspects of genetic programming: namely, primitives, semantics, genetic operators, and architecture. The paper surveys the technical issues encountered in each of these four aspects, and offers some proposed solutions.

• Andy Clark, Natural Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence, Oxford University Press, 240 pp., 2003. ISBN: 0-195-14866-5.

  Abstract: From the publisher: "In (this book), Clark argues that what makes humans so different from other species is our capacity to fully incorporate tools and supporting cultural practices into our existence. Technology as simple as writing on a sketchpad, as familiar as Google or a cellular phone, and as potentially revolutionary as mind-extending neural implants -- all exploit our brains' astonishingly plastic nature. Our minds are primed to seek out and incorporate non-biological resources, so that we actually think and feel through our best technologies. Drawing on his expertise in cognitive science, Clark demonstrates that our sense of self and of physical presence can be expanded to a remarkable extent... the line between the user and her tools grows thinner day by day."

• Rosaria Conte, "Intelligent Social Learning" (html), Journal of Artificial Societies and Social Simulation 4:1 (2001). [electronic journal]

• Leda Cosmides and John Tooby, "Cognitive Adaptations for Social Exchange", pages 163-228 in Jerome H. Barkow, Leda Cosmides, and John Tooby, The Adapted Mind: Evolutionary Psychology and the Generation of Culture, Oxford University Press, 1992.

  Abstract: The authors argue that humans have a faculty of social cognition consisting of a rich collection of dedicated, functionally specialized, interrelated modules organized to collectively guide thought and behavior with respect to the evolutionarily recurrent adaptive problems posed by the social world. The authors provide evidence from human-subject experiments in support of their position.

• Vincent Crawford, "John Nash and the Analysis of Strategic Behavior" (pdf,89KB), USCD Economics Discussion Paper, 2000-01.

  Abstract: This essay describes one economist's view of how the work of John Nash (the focus of the movie "Beautiful Mind") influenced the development of game theory as a tool for analyzing strategic behavior.

• Peter Diamond and Hannu Vartiainen (eds.), Behavioral Economics and its Applications, Princeton University Press, 336pp., February 2007.

  Abstract: "In the last decade, behavioral economics, borrowing from psychology and sociology to explain decisions, inconsistent with traditional economics,has revolutionized the way economists view the world. But despite this general success, behavioral thinking has fundamentally transformed only one field of applied economics -- finance. In this volume, some of the world's leading thinkers in behavioral economics and general economic theory make the case for a much greater use of behavioral ideas in six fields where these ideas have already proved useful but have not yet been fully incorporated -- public economics, development, law and economics, health, wage determination, and organizational economics."

• John Duffy, Agent-Based Models and Human-Subject Experiments, in Leigh Tesfatsion and Kenneth L. Judd (editors), Handbook of Computational Economics: Vol. 2, Agent-Based Computational Economics, North Holland/Elsevier, Amsterdam, the Netherlands, 2006.

  Abstract: This chapter examines the relationship between agent-based modeling and economic decision-making experiments with human subjects. Both approaches exploit controlled "laboratory" conditions as a means of isolating the sources of aggregate phenomena. Research findings from laboratory studies of human subject behavior have inspired studies using artificial agents in "computational laboratories" and vice versa. In certain cases, both methods have been used to examine the same phenomena. The focus of this chapter is on the empirical validity of agent-based modeling approaches in terms of explaining data from human subject experiments. We also point out synergies between the two methodologies that have been exploited, as well as promising new possibilities.

• Margaret Edwards, Sylvie Huet, Francois Goreaud, and Giullame Deffuant, "Comparing an Individual-Based Model of Behavior Diffusion with its Mean Field Aggregate Approximation" (html), Journal of Artificial Societies and Social Simulation, Vol. 6(4), 2003.

• Stan Franklin and Art Graesser, "Is it an Agent, or Just a Program? A Taxonomy for Autonomous Agents" (html), Proceedings of the Third International Workshop on Agent Theories, Architectures, and Languages, Springer-Verlag, 1996.

• Gerd Gigerenzer and Reinhard Selten (eds.), Bounded Rationality: The Adaptive Toolbox, The MIT Press, Cambridge, MA, 2001.

  Abstract: From the publisher: "This book promotes bounded rationality as the key to understanding how real people make decisions. Using the concept of an `adaptive toolbox,' a repertoire of fast and frugal rules for decision making under uncertainty, it attempts to impose more order and coherence on the idea of bounded rationality."

• David Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, Reading, MA, 1989.

• Kevin Gurney, An Introduction to Neural Networks, CRC Press, 1997, 234 pages.

  Abstract: "Though mathematical ideas underpin the study of neural networks, the author presents the fundamentals without the full mathematical apparatus. All aspects of the field are tackled, including artificial neurons as models of their real counterparts; the geometry of network action in pattern space; gradient descent methods, including back-propagation; associative memory and Hopfield nets; and self-organization and feature maps. The traditionally difficult topic of adaptive resonance theory is clarified within a hierarchical description of its operation. The book also includes several real-world examples to provide a concrete focus. This should enhance its appeal to those involved in the design, construction and management of networks in commercial environments and who wish to improve their understanding of network simulator packages. As a comprehensive and highly accessible introduction to one of the most important topics in cognitive and computer science, this volume should interest a wide range of readers, both students and professionals, in cognitive science, psychology, computer science and electrical engineering."

• Douglas R. Hofstadter, "Computer Tournaments of the Prisoner's Dilemma Suggest How Cooperation Evolves", Scientific American, May 1983, 19-26.

  Abstract: This is an early, entertaining, nontechnical account of the famous iterated prisoner's dilemma tournaments run by Robert Axelrod in the late 1970s and early 1980s.

• John Holland, "Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence", MIT Press, Cambridge, MA, 1992 (second edition).

  Abstract: "(This is) a revised edition of the seminal book that first gathered and developed the critical mass of ideas from mathematics, computational science, and systems theory necessary to launch and fuel the ongoing revolution in complex innovating systems."

• John Holland, The Hidden Order: How Adaptation Builds Complexity, Addison-Wesley, 1995, ISBN: 0-201-40793-0.

  Abstract: "The father of genetic algorithms and of Echo explains in a clear and entertaining manner important properties of composite complex adaptive systems, especially those based on computers. Along the way he provides invaluable insights into economics, ecology, biological evolution, and thinking."

• Cars Hommes, "The Heterogeneous Expectations Hypothesis: Some Evidence from the Lab" (pdf,956KB), Journal of Economic Dynamics and Control, Vol. 36, Issue 1, January 2011, 1-24.

  Abstract: "This paper surveys learning-to-forecast experiments (LtFEs) with human subjects to test theories of expectations and learning. Subjects must repeatedly forecast a market price, whose realization is an aggregation of individual expectations. Emphasis is given to how individual forecasting rules interact at the micro-level and which structure they co-create at the aggregate, macro-level. In particular, we focus on the question whether the evidence from laboratory experiments is consistent with heterogeneous expectations."

• John Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press, Cambridge, MA, 1992.

  Abstract: "In this groundbreaking work, (the author) shows how this remarkable (genetic programming) paradigm works and provides substantial empirical evidence that solutions to a great variety of problems from many different fields can be found by genetically breeding populations of computer programs."

• Hongyan Li and Leigh Tesfatsion, "Co-Learning Patterns as Emergent Market Phenomena: An Electricity Market Illustration" (WP pdf,1.4MB), Journal of Economic Behavior and Organization, Vol. 82, 2012, 395-419. The published version is available from Science Direct.

  Abstract: The definition of emergence remains problematic, particularly for systems with purposeful human interactions. This study explores the practical import of the emergence of co-learning patterns for U.S.\ restructured electric power markets. Computational experiments are conducted with co-learning sellers of electric power within the context of an ISO/RTO-managed electric power market operating over an AC transmission grid with congestion managed by locational marginal pricing. Heat maps are used to reveal global market patterns supported by seller co-learning. These patterns arise and persist for seller offer behaviors and for seller net earnings. Co-learning sellers strongly outperform sellers offering true reservation values.

• Kevin McCabe, "Neuroeconomics" pp. 294-298 in Lynn Nadel (ed.), Encyclopedia of Cognitive Science, Macmillan Publishing, New York, 2003.

  Abstract: Neuroeconomics is the study of how the embodied brain interacts with its external environment to produce economic behavior. Research in this field will allow social scientists to better understand individuals' decision making, and consequently to better predict economic behavior.

• Melanie Mitchell, "Genetic Algorithms: An Overview", Complexity, Vol. 1/No. 1, 1995, 31-39.

• Melanie Mitchell, "Can Evolution Explain How the Mind Works?: A Review of the Evolutionary Psychology Debates", Complexity, Vol. 4/No. 3, January/February 1999, 17-24.

• Scott E. Page, The Difference: How the Power of Diversity Creates Better Schools, Firms, Groups, and Societies, Princeton University Press, Princeton, NJ, 448pp., January 2007.

  Abstract: "In this landmark book, Scott Page redefines the way we understand ourselves in relation to one another. (This book) is about how we think in groups -- and how our collective wisdom exceeds the sum of its parts. .. (It) reveals that progress and innovation may depend less on lone thinkers with enormous IQs than on diverse people working together and capitalizing on their individuality." Scott E. Page is Professor of Complex Systems, Political Science, and Economics at the University of Michigan.

• Andreas Duus Pape and Kenneth J. Kurtz, Evaluating Case-Based Decision Theory: Predicting Empirical Patterns of Human Classification Learning (pdf,568KB), Working Paper, Binghamton University (SUNY), January 17, 2012.

  Abstract: "We introduce a computer program which calculates an agent's optimal behavior according to Case-Based Decision Theory (Gilboa and Schmeidler 1995) and use it to test CBDT against a benchmark set of problems from the psychological literature on human classification learning (Shepard, Hovland, and Jenkins 1961). This allows us to evaluate the efficacy of CBDT as an account of human decision-making on this set of problems. We find: (1) The choice behavior of this program (and therefore Case-based Decision Theory) correctly predicts the empirically observed relative difficulty of problems in the benchmark human data, which is a strong vote of confidence in its favor. (2) `Similarity' (how CBDT decision makers extrapolate from memory) is decreasing in Euclidean vector distance, consistent with evidence in psychology (Shepard 1987). (3) Average similarity is rejected in favor of additive similarity. (4) CBDT learns the correct solutions unrealistically fast relative to human learners."

• Mridul Pentapalli, "A Comparative Study of Roth-Erev and Modified Roth-Erev Reinforcement Learning Algorithms for Uniform-Price Double Auctions" (pdf slides,6MB), M.S. Thesis, Computer Science Department, March 2008.

  Abstract: This thesis focuses on multi-agent learning in market contexts. It reports findings from a comparative study of three variants of the Roth-Erev reinforcement learning algorithm currently in use for a variety of market applications. Two different double-auction market testbeds are developed and used to carry out benchmark comparisons involving intensive parameter sweeps with heat map visualization of parameter sensitivities. A primary concern is the degree to which each tested algorithm permits learning agents to converge to the choice of a best action measured in terms of accumulated profits. Some findings from a mathematical analysis of the algorithms' properties are also reported.

• Rethinking Thinking (html,7pp), The Economist 18 (1999), 63-65.

  Abstract: This short review of recent work in behavioral economics explains how and why "economists are starting to abandon their assumption that humans behave rationally, and instead are finally coming to grips with the crazy, mixed up creatures that we really are."

• Bernhard Schölkopf and Alex Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond, The MIT Press, 644 pp., December 2001. ISBN: 0-262-19457-9.

  Abstract: From the publisher: "In the 1990s, a new type of learning algorithm was developed, based on results from statistical learning theory: the Support Vector Machine (SVMB). This gave rise to a new class of theoretically elegant learning machines that use a central concept of SVMs - kernels - for a number of learning tasks. Kernel machines provide a modular framework that can be adapted to different tasks and domains by the choice of the kernel function and the base algorithm. They are replacing neural networks in a variety of fields, including engineering, information retrieval, and bioinformatics. (This book) provides an introduction to SVMs and related kernel methods. Although the book begins with the basics, it also includes the latest research. It provides all of the concepts necessary to enable a reader equipped with some basic mathematical knowledge to enter the world of machine learning using theoretically well-founded yet easy-to-use kernel algorithms and to understand and apply the powerful algorithms that have been developed over the last few years."

  Bernhard Schölkopf is Director at the Max Planck Institute for Biological Cybernetics in Tübingen, Germany, and Professor at the Technical University in Berlin. Alexander J. Smola is Leader of the Machine Learning Group, Research School for Information Sciences and Engineering, the Australian National University.

• Karl Sigmund, Ernst Fehr, and Martin A. Nowak, "The Economics of Fair Play", Scientific American, Vol. 83, January 2002, 83-87.

• John Tooby and Leda Cosmides, "The Psychological Foundations of Culture", pages 19-136 in Jerome H. Barkow, Leda Cosmides, and John Tooby, The Adapted Mind: Evolutionary Psychology and the Generation of Culture, Oxford University Press, 1992.

  Abstract: Strong defense of "evolutionary psychology" as the proper approach to understanding culture and the human mind. The authors define culture to be "the manufactured product of evolved psychological mechanisms situated in individuals living in groups." These psychological mechanisms constitute an "incredibly intricate, contingent set of functional programs that use and process information from the world, including information that is provided both intentionally and unintentionally by other human beings." Consequently, social and cultural attributes are viewed as aspects of evolved human biology to which evolutionary analysis (in particular, natural selection) can properly be applied.

• Nicolaas Vriend, "An Illustration of the Essential Difference Between Individual and Social Learning, and its Consequence for Computational Analyses" (pdf,245), Journal of Economic Dynamics and Control, Vol. 24, 2000, pp. 1-19.

  Abstract: Vriend focuses on the importance of the level of learning for computational agents. An agent is said to employ individual-level learning when it learns from its own past experiences, and to employ population-level learning when it learns from other agents, e.g., through mimicry of their observed behaviors. Using a simple market model for concrete illustration, Vriend demonstrates that substantially different outcomes can result when firms use individual-level genetic algorithm learning versus population-level genetic algorithm learning.

  NOTE: This Vriend article has generated some controversy regarding the robustness of the reported findings. See Arifovic and Maschek (2006) (pdf,226KB) for claims of non-robustness, and see Vallee and Yildizoglu (2007) (pdf,1.9MB) for a detailed rebuttal.

• Yildizoglu M., M.-A. Sénégas, I. Salle and M. Zumpe, "Learning the optimal buffer-stock consumption rule of Carroll" (pdf,419KB), Macroeconomic Dynamics, 2013, to appear.

  Abstract: "This article questions the rather pessimistic conclusions of Allen and Carroll (2001) about the ability of consumers to learn the optimal buffer-stock based consumption rule. To this aim, we develop an agent based model where alternative learning schemes can be compared in terms of the consumption behaviour that they yield. We show that neither purely adaptive learning, nor social learning based on imitation can ensure satisfactory consumption behaviours. By contrast, if the agents can form adaptive expectations, based on an evolving individual mental model, their behaviour becomes much more interesting in terms of its regularity, and its ability to improve performance (which is as a clear manifestation of learning). Our results indicate that assumptions on bounded rationality, and on adaptive expectations are perfectly compatible with sound and realistic economic behaviour, which, in some cases, can even converge to the optimal solution. This framework may therefore be used to develop macroeconomic models with adaptive dynamics."

Software, Toolkits, and Demos

• The Trade Network Game: Demonstration Software (Genetic Algorithm Learning) (html)

• Annotated list of pointers to ACE/CAS General Software and Toolkits

• Annotated list of pointers to ACE/CAS Computational Laboratories and Demonstration Software

Resource Sites and Groups

• ACE/CAS Research Sites and Groups

• The Agents Learning About Agents web site is dedicated to the study of what happens when agents (i.e., pro-active, goal-driven, selfish, independent software/hardware constructs) start to learn about each other, especially if they do so in order to gain a competitive advantage over other agents. Resources available at this site include pointers to classes, laboratories, and other web sites useful for this topic area.

• The American Association for Artificial Intelligence (AAAI) maintains a website on Artificial Intelligence Topics for anyone who would like to explore what artificial intelligence is and what AI scientists do. The stated goal of this extensive and meticulously organized site is to offer a limited number of exemplary non-technical resources catagorized and annotated to provide meaningful access to basic information about AI.

• The Autonomous Learning Laboratory (ALL), co-directed by Andrew G. Barto and Sridhar Mahadevan, carries out foundational interdisciplinary research on machine learning and computational models of biological learning. Autonomous learning refers to what a self-reliant agent must do to learn from its own experiences. The long-term goals of the laboratory are to develop more capable artificial agents, to improve our understanding of biological learning and its neural basis, and to forge stronger links between studies of learning by computer scientists, engineers, neuroscientists, and psychologists. Areas of interest include reinforcement learning, machine learning, abstraction, hierarchy, motor control, robotics, computational neuroscience and developmental psychology.

• The goal of BotSpot is to be a definitive resource for bots, intelligent agents, and artificial intelligence on the web. The site includes: fourteen searchable Bot Classification Databases for bot implementations; FAQs; book recommendations; and pointers to articles, electronic journals, upcoming conferences, previous conference proceedings, and language and code for creating bots and intelligent agents. The site also supports a free monthly BotSpot Newsletter. BotSpot has received over 150 awards in its first twelve months of operation, including a designation by PC Magazine as one of their top 100 recommended web sites.

• The Center for Adaptive Behavior and Cognition (ABC) is directed by Prof.Dr. Gerd Gigerenzer (Max Planck Institute, Berlin), one of the most well-known and articulate critics of the model of the perfectly rational human being claimed to be implicit in the work of Nobel laureate Daniel Kahneman, Amos Tversky, and other researchers investigating perceived anomalies in experimentally observed human decision making under uncertainty. ABC researchers focus on the discovery and study of simple cognitive satisficing algorithms that people use to solve adaptive problems in specific real-world domains, such as avoiding dangers, choosing mates, and investing in offspring. ABC takes an evolutionarily-inspired view of the mind as a collection of these task-specific algorithms and modules rather than as a general-purpose problem solver.

• The Center for Research on Concepts and Cognition (CRCC) at Indiana University is an interdisciplinary center for research in cognitive science directed by Douglas Hofstadter. CRCC research focuses mainly on emergent computational models of creative analogical thinking and its subcognitive substrate -- namely, fluid concepts. The group also conducts research (mostly non-computational) in a number of other areas of cognitive science, including error-making, creative translation, scientific discovery, musical composition, the comprehension and invention of jokes, the nature of sexist language and default imagery, philosophy of mind, and foundations of artificial intelligence.

• Randall Whitaker maintains a site focusing on Cognition, Autopoietic Theory, and Enactive Cognitive Science. Included at the site is a link to a tutorial on autopoeisis, described as follows: "This introductory tutorial is designed to give you a brief overview of autopoietic theory -- the term I use to denote the work of Chilean biologists Humberto R. Maturana and Francisco J. Varela. ... Maturana's early work in neurophysiology and perception ... led him to question the information-theoretic notions of cognition. (H)e subsequently created and refined (his theory) with Varela. ... As a biological phenomenon, cognition is viewed with respect to the organism(s) whose conduct realizes that phenomenon. In autopoietic theory, cognition is a consequence of circularity and complexity in the form of any system whose behavior includes maintenance of that selfsame form. This shifts the focus from discernment of active agencies and replicable actions through which a given process (`cognition') is conducted (the viewpoint of cognitive science) to the discernment of those features of an organism's form which determine its engagement with its milieu."

• The CogWeb Site maintained by Francis Steen (Communication Studies, UCLA, Los Angeles) is devoted to exploring the relevance of the study of human cognition to literary and cultural studies. Resources available at the site include pointers to related sites and articles as well as to bibliographic materials on linguistics, cognitive science, evolution and cognition, and cognitive cultural studies (both early and modern).

• Stan Franklin, Professor of Mathematical Sciences at the University of Memphis, Tennessee, and well-known author of Artificial Minds (MIT Press, 1995), maintains a web site on Conscious Software. From the website introduction: "By a `conscious' software agent we mean a cognitive agent (an autonomous agent with human-like cognitive features) designed within the constraints of Baar's global workspace theory of consciousness. Like the Roman god Janus, the conscious software project has two faces, its science face and its engineering face. Its science side will flesh out the global workspace theory of consciousness, while its engineering side explores architectural designs for information agents that promise more flexible, more human-like intelligence within their domains." Accounts of various conscious software projects by Franklin and his collaborators can be found at the Conscious Software web site.

• Leigh Tesfatsion (Iowa State University, Ames, IA) maintains a site titled Criterian Filtering: A Dual Approach to Bayesian Updating and Adaptive Control. Criterion filtering is the direct updating of criterion functions on the basis of transitional reward assessments in analogy to Bayes' Rule for the updating of probability distributions on the basis of transitional probability assessments.

• Adrian Thompson (COGS, University of Sussex, UKB) maintains a web site titled Evolutionary Electronics Web Links. This site provides pointers to researchers and research groups around the world specializing in evolvable hardware.

• Roger McCain (Economics, Drexel University, Philadelphia, Pennsylvania) has developed a Game Theory Web Site in which he presents an accessible account of elementary game theory principles for non-specialists. See, also, the Behavioral Game Theory Course developed by Vince Crawford (Economics, UCSD, La Jolla, CA).

• A repository of resources on the use of artificial intelligence (AI) in the design of games can be found at the Game AI Page. This site stresses practical approaches to the problem of building better computer opponents and is aimed at both game developers and game players.

• The Laboratory for Natural and Simulated Cognition (LNSC) at McGill University in Montreal, Canada, investigates human cognition through a combination of psychological and computational approaches. Basic psychological phenomena are simulated in a connectionist framework, often leading to predictions that are tested with humans. Current projects concern cognitive development, interactions between knowledge and learning, techniques for analyzing knowledge representations in neural nets, and cognitive consistency phenomena in social psychology.

• MIT CogNet is an electronic community for the cognitive and brain sciences under development by the MIT Press. The intention is to bring together current and classic resources in the field and provide a unique, interactive forum for scholars, students, and professionals. Services will include: a searchable full-text library with a growing collection of books, journals, and other reference works; an academic almanac of cognitive science programs; editorials on groundbreaking or controversial research; job listings; virtual poster sessions; threaded discussion groups; and community member profiles. MIT CogNet is a free service through August 31, 2000, and is actively seeking charter members.

• The goal of the MIT Robust Open-Agent Systems (ROMA) Research Group is to learn how to develop multi-agent systems for open contexts where the constituent agents can come from anywhere, may be buggy or even malicious, and must run in the dynamic and potentially failure-prone environments at hand. This is viewed as an important area of research since many emerging problems (e.g., electronic commerce, large product development projects, multi-national rescue operations) require the ability to rapidly assemble virtual organizations on the Internet with partners who may have never worked together before.

• The Perceptual Science Laboratory at the University of California, Santa Cruz, California, is engaged in a variety of experimental and theoretical inquiries in perception and cognition. A major research area concerns speech perception by ear and eye, and facial animation. Laboratory researchers have also tested a general fuzzy logical model of perception in a variety of domains, including perception and understanding of language, memory, object, shape, depth perception, learning, and decision making. Information about the Perceptual Science Laboratory available at the laboratory web site includes research reports and data.

• The Autonomous Agents Laboratory at Michigan State University maintains an online repository of resources on Reinforcement Learning (RL). The website provides resources on both RL research and applications to areas such as robotics and industrial problems. Resources available include technical publications, sample testbeds, implementations of various algorithms, online simulation packages, workshop information, and discussion forums for a variety of research areas within RL. The website is supported by the National Science Foundation.

• RL-Glue is a standard software protocol for benchmarking and interconnecting reinforcement learning agents and environments. The RL-Glue site is maintained by researchers at the University of Alberta, Canada. This intention of this site is to provide a concise and comprehensive guide to setting up and using RL-Glue.

• Randall Whitaker maintains a site focusing on Cognition, Autopoietic Theory, and Enactive Cognitive Science. Included at the site is a link to a tutorial on autopoeisis, described as follows: "This introductory tutorial is designed to give you a brief overview of autopoietic theory -- the term I use to denote the work of Chilean biologists Humberto R. Maturana and Francisco J. Varela. ... Maturana's early work in neurophysiology and perception ... led him to question the information-theoretic notions of cognition. (H)e subsequently created and refined (his theory) with Varela. ... As a biological phenomenon, cognition is viewed with respect to the organism(s) whose conduct realizes that phenomenon. In autopoietic theory, cognition is a consequence of circularity and complexity in the form of any system whose behavior includes maintenance of that selfsame form. This shifts the focus from discernment of active agencies and replicable actions through which a given process (`cognition') is conducted (the viewpoint of cognitive science) to the discernment of those features of an organism's form which determine its engagement with its milieu."

• The Social Psychology Network is an extensive database on social psychology maintained by Scott Plous (Weslyan University) and supported by the National Science Foundation. The database provides more than 5,000 links to psychology-related resources. The database can be searched by topic or keyword.

• The Stanford Encyclopedia of Philosophy Web Site edited by Edward N. Zalta (Stanford University), is a dynamic encyclopedia of entries for all areas of philosophy, including many entries relevant to agents, cognitive science and AI.

• The Tangible Media Group at the MIT Media Laboratory (Cambridge, Massachusetts), founded and directed by Hiroshi Ishii, focuses on the design of seamless interfaces between humans, digital information, and physical environments. From Hiroshi Ishii: "People have developed sophisticated skills for sensing and manipulating our physical environments. However, most of these skills are not employed by traditional Graphical User Interface (GUI). Tangible Bits, our vision of Human Computer Interaction, seeks to build upon these skills by giving physical form to digital information, seamlessly coupling the dual worlds of bits and atoms. Guided by the Tangible Bits vision, we are designing `tangible user interfaces' which employ physical objects, surfaces, and spaces as tangible embodiments of digital information. These involve foreground interactions with graspable objects and augmented surfaces, exploiting the human senses of touch and kinesthesia. We are also exploring background information displays which use `ambient media' ---- ambient light, sound, airflow, and water movement. Here, we seek to communicate digitally-mediated senses of activity and presence at the periphery of human awareness. Our goal is to realize seamless interfaces between humans, digital information, and the physical environment taking advantage of the richness of multimodal human senses and skills developed through our lifetime of interaction with the physical world."

Some Early Individual Researchers

• Individual Researchers using ACE/CAS Tools

• W. Brian Arthur, (Citibank Professor at the Santa Fe Institute): Designing economic agents that act like human agents; The El Farol Bar Problem; Artificial stock market modelling; Effects of positive feedbacks.

• Andrew G. Barto (Computer Science, University of Massachusetts, Amherst): Learning in natural and artificial systems; Machine learning algorithms; Development of the computational theory and practice of reinforcement learning; Models of motor learning and reinforcement learning methods for real-time planning and control; Autonomous mental development through intrinsically motivated reinforcement learning.

• Mark H. Bickhard (Cognitive Science, Leigh University, Bethlehem, PA): Interactivism; Theoretical psychology; Modeling of biological and social persons; Design of artificial persons.

• Thomas Brenner, (University of Marburg, Germany): Evolution of industrial clusters and milieux; Transfer of knowledge; Modelling of learning in economics; Evolutionary game theory; Consumer interaction and fashions; Diffusion of innovations.

• Andy Clark (Professor of Philosophy, University of Edinburgh, Scotland): Embodied cognition; Connectionism; Neural nets and embodied action; Relation between thought and language; Respective roles of computational, representational, and dynamical analyses in cognitive science; Real-world robotics and animate vision; Interplay between individual cognition and the wider webs of social structure and technological artifact.

• John Duffy (Economics, University of Pittsburgh, Pennsylvania): Incorporation of learning in computational economic models; Using genetic algorithms to model how agents learn and adaptively update their forecasts.

• Gerd Gigerenzer (Psychology, Center for Adaptive Behavior and Cognition, University of Munich, Germany): Models of bounded rationality; Social intelligence; Ecological rationality; Heuristics of scientific discovery; Philosophy, history, and methodology of the social sciences.

• Marvin Minsky (MIT Media Lab and MIT AI Lab, Cambridge, Massachusetts): Human intellectual structure and function (the society of mind); Imparting to machines the human capacity for commonsense reasoning; Artificial intelligence; Cognitive psychology; Computational linguistics; Robotics.

• Sridhar Mahadevan, (Computer Science, University of Massachusetts, Amherst): Artificial intelligence; Credit assignment problem; Computational models of learning and sequential decision-making.

• Al Roth (Standord University, Palo Alto, CA): Reinforcement learning; Game theory and experimental economics; Market matching mechanisms in theory and practice.

• Leigh Tesfatsion (Economics, Iowa State University, Ames, Iowa): Learning in multi-agent market contexts; Effects of learning in restructured wholesale power markets; Learning via criterion filtering.

• Peter M. Todd (School of Informatics and Computing, Indiana University, USA): Evolution of behavior; Simple heuristics for sequential search, categorization, and multi-step processes; Psychological selection; Rhythmic and time-based behavior; Connectionist models of cognition.

• Nick Vriend (Economics, Queen Mary and Westfield College, University of London): Dynamics of interactive market processes; Emergent properties of evolving market structures and outcomes; Learning algorithms; History of economic thought.

• Murat Yildizoglu (Economics, Montesquieu Bordeaux IV University, Pessac, France): Evolutionary modeling and economic dynamics; Industry dynamics; Economics of innovation; Economic growth; Industrial organization; Decision theory and the theory of the firm.

Copyright © 2013 Leigh Tesfatsion. All Rights Reserved.