Fast Iteration: lessons from genetic programming

Okay, I am not going to get too geeky here, but there are important lessons that I have learned from hanging around with some serious geeks. One lesson was about the power of value networks (learned from the open source community). The other is the power of fast iteration. This one I learned from practitioners of genetic programming.

The origins of genetic programming dates back to 50’s but it really only began to come into its own after the turn of the century (doesn’t that sound quaint?) with cheap access to serious computing horsepower. The idea was that you could develop programs that could evolve themselves to solve complex problems.

Okay guys, let’s get one thing out of the way. Yes, there is evolution. Not all programs are created in unblemished perfection.

Key concept here: “evolve themselves”.

Basically, what you can do is to take a set of generic programs and point them at a complex question that needs to be solved. Turn them on, and walk way. Lo and behold, the problem is eventually solved.

In essence, in the darkened room of discovery, the programs begin to try to address the problem. They try, and fail, and learn from that failure, then morph, and try again. Over and over again until the answer is found.  This process often takes hours, but it has been successfully demonstrated multiple times.

But if you think about it, this process is the same for any innovation. It needs to be focused. It needs to iterate a potential new solution multiple times, and it needs to learn. At its core, the innovation process needs be obsessive, agile and unrelenting. In time, it will work. Always. Though probably not in the way you expected.

La, La, La, I’m Not Listening …

Recently a client asked us me remove a question from a survey. The question was, “Are there any additional features you’d like us to offer?” It’s a pretty standard question, the purpose of which is to engage customers in the process of co-innovation.

But the client didn’t want it. The reason? “Our product development team has too long a list of features already.”

So let me get this straight – the solution to having too much on your plate is to stop listening to customers?

I find this attitude pretty frightening. To succeed, a company needs passionate customers who will become evangelists. You only create that kind of passion when you know your customers.

Another reason to ask the question is to keep track of changing trends. Suppose they asked the question and found out that the stuff they were developing now was not what the customers were asking for? Wouldn’t that be considered useful information?

You can never stop listening, never assume you know it all, or you’re headed straight for mediocrity.

Architecture and Experience

A project I’m working on now – helping the Portland Police Bureau design their next-generation collaboration system – got me thinking of how people get the process of software design wrong. Far too often, people start by trying to define the functionality of the software application. “We want it to do this. We want to do that.” And they end up with a long list of requirements that they send off to the coders to magically make whole. That’s a sure way to fail.

One thing I’ve learned working with software development is that software is 80% architecture and 20% coding. The real art is in the architecture. Once you’ve got that down, the coding is easy.

But architecture design is hard and complex. You need to do some deep thinking about how the pieces fit together. You also need to understand how it will allow for future development, for good software is a living organism that evolves, not a fixed product. That’s because we – the users – evolve, and so do our needs.

This architecture is also multi-dimensional. There are six key levels you need to consider:

• The experience
• The knowledge
• The information
• The software
• The system
• The components

Let’s go through them one at a time.

The top level of architecture is the experience architecture – the experience that the software creates for the user. The user enters into the experience with a problem or an aspiration. How is this experience going to help them address that problem or aspiration quickly and elegantly? To evaluate the success of this experience, you need to appreciate that it’s both linear and intellectual as well as spacial and emotional.

The next level down is the knowledge architecture. Within every organization are centers of expertise and knowledge. For a police department, it might be the gang team – if you want know who has the pulse on the gang activities – or the district officers, if you want to know what chronic issues are troubling a neighborhood. If you understand where the knowledge is located in an organization, you can begin to knit it all together into a solution.

Below that is the information architecture. Now we’re beginning to get into the software realm. Information resides in different databases or on spreadsheets. Often it’s scattered around and difficult to access. And some of it can’t be accessed by a system at all, because it’s stored in a two-legged computing device called a human being. So it’s critical to understand how to collect and integrate organizational information.

Now and only now do we get down to the software architecture. Only when you understand the architectures above it can you design a software architecture that can support it. But the architecture design does not end there, for next comes the hardware elements of the solution, which need to support the software architecture. You start with the system architecture, then knit together the hardware devices (the computers, storage devices, etc.) and then the device and component architectures that allow the software to perform quickly and consistently.

Sound like a lot of work? It is. That is why it represents 80% of the effort.

But the key point here regarding the innovation process is that to be successful, you need to start with the experience. In the case of the police department, you begin with police officers who need to become as smart as they can in the three minutes they have before they arrive at the scene, while driving 60 miles an hour down city streets and punching a few key words into their onboard computers. How do you do that?

I would argue that companies who understand experience architecture deeply are more successful than ones who don’t – even if the latter make stuff with great functionality. There were mp3 players that did lots more than the iPod, and ones that still do. But clearly Apple started with the experience architecture. They didn’t ask, “What do we want this thing to do?” They asked, “What do we want the user to experience when they use it?”

Apple gets it, and they effectively control everything right down to the component architecture, which is why their user experiences are nonpareil. But how can you do this if you’re not Apple?

You can start by not doing things backwards. Too often creators start by outlining the functionality, and then deal with the user experience only as an afterthought. That’s exactly backwards. Because it’s not the software that creates value – it’s the experience.

But even IBM admits that they don’t have the internal resources to quickly design, build, and evolve fully integrated architectures. That’s why they’ve embraced the principles of open innovation, in which they work in agile partnerships with others to quickly develop new market solutions.

Soccer and Mesh Networking

I played soccer last Saturday morning, first time in 30 years or so. Blame it on World Cup Fever.

What I was reminded of, apart from where my pain points are, was how much I enjoyed the collaborative nature of soccer.

Sports are metaphors of our cultural values. Baseball came of age in the beginning of industrial society, as a way to maintain connection with agrarian life. Here was a game that came and went with the seasons, in which the only clock was the sun.

I grew up in Cleveland, a football town. Why did Cleveland follow football (American football, that is)? Because Cleveland was a steel town, and football was made to order for the factory mindset. The goal of football is to control the field – knock the other guys out of the way to create a space in which to catch the pass and run. Industries like steel, meanwhile, were seeking to control the value chain, from raw material to finished product. Management and unions struggled for the ball and control of the field.

So where does soccer fit in?

Soccer is a network. You can pass to anyone else in order to optimize your team’s chances of a goal. While a batter engages the pitcher in a splendid mindgame, and the football player follows a complex playbook, the soccer player is continually asking, “how do we move together to make this work?”

To put it in IT terms, the players are nodes on a fully-connected mesh network. The paradigm for this is the wireless network, in which nodes are continually connecting and reconfiguring to fix broken pathways, or sidestep blocked ones.

That’s what’s great about soccer – it’s the closest model for how we’re getting things done these days.

Sharing, Innovation and Big Blue

Things change. When Stanley Kubrick imagined HAL, the psychotic computer in 2001:A Space Odyssey, an urban legend had the name being derived by shifting the abbreviation “IBM” over by one letter. It’s not true, but everyone believed it, because IBM represented a kind of early version of the evil empire: secretive and bent on world domination.

Few companies have shifted so dramatically in their internal culture. IBM now thrives through a new strategy: sharing intellectual property. I read with delight a presentation by Anders Quitzau, an IBM executive, detailing their philosophy of opening up their labs to create an environment for innovation that goes beyond IBM’s own walls. The reason they did it:

[The] Pace of innovation outstrips an organization’s ability to “go it alone”

If you have a known market – say, computers in the 1960s, which sold to universities and government – you can thrive by managing the value chain. But most markets these days are unknown, because things are shifting so fast. The only way to go is to get ideas out there and test them. Your IP isn’t enough, probably: you need to combine it with others’. Sharing ideas lets you engage the market in a process of discovery. The money is in the solutions you build using the shared knowledge.

You Heard it Here First – Customers Spur Innovation

Okay, you didn’t hear it here first, but it certainly backs up what I’ve been saying. Siemens has published a detailed report on Open Innovation which BQF Innovation summed up nicely. The upshot: a greater percentage of respondents listed “customers” as a source of ideas.

That’s a greater percentage even than internal R&D or employees.

Open innovation refers to bringing outsiders like customers into the innovation process. Here’s the original one page report [.pdf link], which is worth reading. I find it interesting that Asia Pacific seems to understand best the importance of customer-led innovation. I have a few ideas on why that is, and you’ll be hearing back from me about that.

It would appear that if your company is engaged in active listening, it’s on the right track. And if it isn’t – well, no problem. Your competitors are sure to find a use for all those ideas.

Social Media Can’t Spur Innovation Alone

In my last post I discussed Value Networks. Of course, value networks are not talked about much these days. Scan the marketing blogs today and you’ll find that social networks, not value networks, are the flavor of the month. I’ve got nothing against social networks, but I wonder if Linux would have arrived yet if developers had just been tweeting their ideas.

Facebook and Twitter, the social media poster children, aren’t collaboration tools, so they can’t really create a value network. They are information-enabling tools. The purpose of social networking is to connect. Nothing wrong with that, but to create something complex and valuable, you need to go beyond connecting to collaboration.

I mentioned that the overheard conversation is essential to speeding up innovation. True, you overhear a lot of conversing on Facebook and Twitter, but to really collaborate, you need a tool that can organize conversations into a coherent threaded structure. Complex problems require a lot of people to know where they are in a conversation, and what’s gone before. Jive and Central Desktop are examples of collaboration platforms that do this, and there are others as well.

The upshot – to find out what people are chattering about, monitor social media. To innovate faster, get those conversations threaded up.

Waterfalls, Eyeballs and the Power of the Overheard Conversation

When I first started i-OP, one of my tasks was choosing the tools our team would use to collaborate. My developers insisted that newsgroups – what we now call forums – were the right way to communicate as a company.

It struck me as odd at the time, but I trusted them, and a good thing, too. What my developers taught me was the same lesson that Microsoft learned the hard way when Linux started to move in on their territory.

Linux is open source, developed in a large part collaboratively via the network of newsgroups known as USEnet. The beauty of a newsgroup is that discussions are overheard by others, who can introduce new ideas and possibilities. This is especially important in software development, because it’s just about impossible to know all the requirements and obstacles before you start. The job description constantly changes, so the top-down waterfall model just isn’t flexible enough. You need eyeballs all over the place to find and kill the bugs and suggest new approaches.

Linux was and remains a great model of the value network method of creating, as opposed to the traditional value chain method. Value networks allow for a faster innovation clockspeed, which is why, in just 5 years, Linux was able to take on wealthiest corporation in the world.

These days USEnet, though still around, has largely given way to a new generation of Web-based collaboration platforms which still nevertheless exploit the power of the overheard conversation.

In my next post I’ll talk a bit about how social networking fits – and doesn’t fit – into the value network model.