Archive for February, 2012

Five resources for learning complex adaptive systems

February 21, 2012 3 comments

As I get started contributing more regularly to this blog, I wanted to give everyone a baseline for understanding what I mean when I talk about complex adaptive systems (or CAS). It’s not an easy subject to master—not because the core concepts are difficult, but because the effects of those concepts on the world around as are so rich, varied and profound.

So, I quickly wanted to make everyone aware of a few resources that I consider great ways to get in touch with the core of CAS. Some of them are relatively passive—you can just read or watch them at your leisure. Others are quite interactive, and you can use them to get “hands-on” experience. All of them are the most engaging that I’ve found to date.

  1. Complexity: The Emerging Science At The Edge Of Order And Chaos, by M. Mitchell Waldrop  At this point, this book might be called a classic, given the fact that it was originally published in 1992, but it is still the most engaging introduction to the multi-disciplinary beginnings of complexity theory that I am aware of. By telling the story of the famed Santa Fe Institute, and how key members got interested in both the beginnings of complexity theory and the institute itself, Waldrop manages to bring newcomers to the subject in both an intelligent and entertaining fashion. A must read if you don’t know much of anything about complexity theory and CAS.
  2. Complex Adaptive Systems: An Introduction to Computational Models of Social Life, by John H Miller and Scott E Page – This is an excellent guide to the computational models that scientists are using to understand and experiment with complex adaptive systems. Chapter after chapter introduce you to models both simple and complex, with enough information about how the models were formed that you can create them yourself in the programming environment of your choice. However, even if you don’t choose to recreate the models, it is an eye-opening read.
  3. TUDelft SPM 4530 and SPM 9555 – “Agent Based Modeling of complex energy and industrial networks” and “Agent Based Modeling of Complex Adaptive Systems – Advanced” – I originally found this course material via iTunes, but this Wiki page is the official page of a freely available course delivered by Igor Nikolic, Assistant Professor at the Energy and Industry group, Faculty of Technology, Policy and Management faculty, Delft University of Technology. You can step through each of his lectures, as well as see the practical assignments. The only thing you can’t get access to is the actual output of the students actually taking the class, which makes a ton of sense to me. The cool thing is, the practicals use the next tool extensively. There is hours of material here, so be prepared to spend a few nights and weekends on it.
  4. NetLogo  – This is just about the coolest tool I’ve found to date for modeling complex adaptive systems. Described as a “multi-agent programmable modeling environment”, NetLogo provides a basic engine and a simple modelling language that allows you to create all kinds of amazing CAS models. It also has a big list of sample models, including classics such as flocking birds, forest fires and even a model of PageRank from Larry Page and Sergey Brin’s Stanford days. This one is really fun for understanding a large cross section of CAS modeling. One of these days, I’m hoping to create an original “from scratch” model of some aspect of cloud computing in this tool.
  5. Drift into Failure, by Sidney Dekker – I am currently reading this one, so it may be a bit premature to recommend it as a starting place. It certainly is a dense work, but what makes it important to me is it’s clear statement of situations where complexity takes many apparently good, honest decisions and aggregates them into less than desirable outcomes. This is a must read for those who get CAS to a certain extent, but need to better understand why CAS means we have to think differently about software and systems design and operations in the cloud. This is the very meat of what I think most programmers and operators don’t understand about cloud today.

There will be other resources that I will share with you as time goes by. (For example, Adrian Cockcroft of Netflix pointed me to this post which I haven’t digested yet, but which title intrigues me to no end.) In the meantime, please enjoy these, and please post comments with your own recommendations for key sources of complex adaptive systems knowledge. Or cloud knowledge, for that matter. I look forward to learning from you as much as sharing my own learning with you.

Categories: Uncategorized

Exploring cloud and complex systems

February 13, 2012 3 comments

This post—this blog—have been a long time coming.

While many may know me from my long journey exploring cloud computing (the first few years of which are archived to this blog, followed by three years at CNET writing The Wisdom of Clouds, and now my continuing work on GigaOm/cloud), for some time now I’ve been keenly interested in a more specific topic under the cloud computing umbrella, namely how cloud computing is driving application architectures to adopt the traits of complex adaptive systems.

Complex adaptive systems (CAS) are fascinating beasts. Described by complex systems pioneer John Holland as “systems that have a large numbers of components, often called agents, that interact and adapt or learn”, CAS are the reason major systems in nature work—from biology to ecology to economics and society. CAS allow for constant change, with an emphasis on changes that make the system stronger, although there is a constant risk that the system will see negative events as well.

Why this is fascinating to me is that CAS resist the ability to break them down to component parts, establishing clear cause-and-effect relationships between the agents involved and the emergent behavior of the system as a whole. In fact, the sheer complex nature of these systems means that outcomes from any given action within a system in at best unpredictable, and quite likely impossible.

An excellent example of this, that I’ve used before, is a pile of sand on a table. Imagine dropping more sand, one grain at a time, onto that pile. Quick, how many grains of sand will fall off the table with each grain added to the “system”? It is impossible to predict.

Now, granted, a table of sand isn’t exactly “adaptive”, but it is indicative of what complexity does to predictability.

CAS and IT

When applied to IT—everything from architectures to markets to organizations—the science of complex adaptive systems has real deep ramifications to the ways we plan, design, build, trouble shoot and adapt our most important applications of technology to business. We can’t do everything top-down anymore. Much, much more of our work has to be built and maintained from the bottom up.

Now, I am a novice at all this, so much of what I believe I know today will likely turn out to be wrong. As a quick example, the “Cloud as Complex Systems Architecture” presentation I am giving at Cloud Connect in Santa Clara, CA on Tuesday was supposed to center around focusing on how you automate the operations of individual software components to survive in the cloud. “Focus on tweaking agent automation”, my message was going to be.

However, that is counter to the reality of what has to happen; it is just as important to evaluate the system as a whole, and adjust *whatever* needs to be adjusted to when issues are identified system wide. In other words, an agent-level focus is exactly the kind of thing that gets you in trouble in complex systems. You see? I am still learning.

I hope you will join me as I take this journey. Follow me on Twitter at @jamesurquhart. Subscribe to the RSS feed for this blog. Leave comments. Challenge me. Point me to new sources of information. Tell me I’m full of it and should start over. In return, I promise to listen, and to share my own journey, including insights I gain from books, online courses and the other very smart people I am very lucky to interact with.

This is going to be fun. I’m pumped to get started.