Rebecca Costa

According to Eric Schmidt, the former CEO of Google, over the next 48 hours the amount of data generated will equal or surpass the amount of data generated from the dawn of humankind up through the year 2003. And you know this is nothing compared to what is coming in the next few years with the influx of unstructured data. Yes, big data is getting bigger, faster, and more complex. So what does that mean for us? It depends...

The effective use of big data frees us to innovate. In fact, many organizations see innovation as a critical piece of their overall strategy and essential for the long-­?term success of their business. Why? The process of big data capture, management, and analysis is quickly becoming a commodity – available to everyone. The real competitive edge lies with those companies that leverage big data to unleash innovation. This is good news for business. And this is good news for our planet!

In addition to being a data scientist, I am a passionate environmentalist and humanitarian. I want to see companies operate in ways that benefit the planet as well as their shareholders. So I was excited to discover Rebecca Costa's book, The Watchman's Rattle: A Radical Theory of Collapse (Vanguard Press, 2010). In this ground-­?breaking book, Costa applies the imperative for innovation to the survival of our entire species! She explains that many of our seemingly insurmountable problems, such as climate change, poverty, global tensions, etc., are the result of overwhelming complexity caused by globalization, technology, and the influx of big data. Basically, it appears that we have lost our ability to solve our problems because this accelerating complexity is outpacing our brain's ability to evolve.

According to Costa's research, we are following the pattern of several great civilizations that collapsed – the Romans, the Mayans, the Khmers. Prior to their collapse, they reached their cognitive threshold. Then each went through the following three stages: 1) gridlock; 2) belief replacing science; and 3) irrational policies. Most of us would agree that we are well into stage three.

We've been dealing with the influx of big data for a number of years. Combine big data with emerging technology and globalization and you can see why businesses are struggling. So why does this complexity make us so vulnerable right now? Because for any given problem, there are many more wrong solutions than there are right ones. And the number of wrong solutions is exponentiating by the picosecond. Therefore, each leader, each individual in society, every worker is now trying to adapt and survive a high failure rate environment where your odds of picking the wrong software, the wrong supplier, the wrong financial investment are far greater than your odds of picking the right one.

Consider our economic systems. We have fiat money that has really has no sense to it whatsoever. So we automatically revert to bartering. Bartering we understand. We each meet on the road. I have a cow; you have some wheat. We argue until we come to something that we both agree is a great value for ourselves and we make the exchange. Human beings can understand that kind of transaction. What they don't understand is the kind of thing that got us into the subprime mortgage problem – where you've

got credit defaults that even the most knowledgeable executives of the largest financial organizations cannot understand.

But there is good news. Costa makes a strong case for enabling insights – sparks of genius, as an approach to overcoming mind-­?bending complexity. It appears that the survival of our species depends on our ability and willingness to make connections in our brains in a unique way – to unleash insight for innovation. We all have insights. Think about it. You're trying to solve a problem and are stuck. You take a break. Perhaps you go for a walk, wash the dishes, or take a nap. All of a sudden, the solution pops into your mind. That's an 'insight.' You might know it as an 'aha' moment. Costa argues that this is the source for the breakthrough ideas and solutions to our global problems. It's a third form of problem solving, which is ideally suited to highly complex and chaotic problems. We just have to create the conditions for these insights to emerge. But how?

In science, whenever you get to a point where you can predict something with 100% accuracy, you're really onto something. The way that researchers are defining insight is 'the sudden and spontaneous connection between two pieces of data that you never connected before to solve that problem.' When monitoring people who are sitting in a lab and failing to answer very complex questions, we see brain activity about 300 milliseconds before they're going to answer a question way above their pay grade. And then a small part of the human brain, called the AFTG, lights up like a Christmas tree. And we know they're going to answer a question. And they do.

Many organizations use empirical data to make the best possible decisions. Costa argues that we quickly left the Information Age where the emphasis was on the production and transmission of data. And we have now moved into what she describes as a Technolific Era, where without data analytics, we fall into the same trap that previous civilizations kept falling into – unforeseen collapse to a more primitive system.

But what should we do?

We need to make our data productive so that we can make better decisions and adapt to a high failure rate environment. We have to depend on better models, and we have to depend on technologies like big data systems and other kinds of analytic algorithms. Then we can synthesize it, look for patterns and generate insights. Insights are those breakthrough ideas, those ingenious solutions that will solve our global problems.

So here's the challenge. Turn big data into useful information that our brains can synthesize to produce insights. Watch this space for Complexity Rules Part II: Five Strategies for Thriving in Complexity.