Kelly talks to Stefan Thomke, a leading authority on the management of innovation at the Harvard Business School about his new book, "Experimentation Works."

In the introduction to the book you write, “for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn't work.” That seems very relevant to what's going on right now.

“Of course. Yes. And in fact, there's another passage that I have actually on the sleeve of the book and it says, ‘don't fly blind. See how the power of experiments works for you.’ And I think flying blind is part of the story right now that we're seeing unfold here as we're trying to figure out what to do and how to manage this public health crisis. A lot of us are, in fact, flying blind because we just don't have the data to make the right decisions. And there is some data there that we need to listen to and that some of our leaders are quite happy to ignore. But the reality is, when you actually look at some of the really good writing about this from biostatisticians, they're telling us that because of the lack of data, the range of possibilities is just so huge that it comes down to testing and experimentation and we just don't have good testing strategy. It's now changing a little bit, but we kind of missed this.”

And this whole book is about testing. You are basically telling the world that to get to the answers you need science to solve, you have to test over and over and over again.

“Kelly, I've been researching and writing about testing and experimentation for more than 25 years now. And what's really amazing is that the number of articles about testing and the general media that I've seen in the last few weeks, of course, completely outdid anything that I've seen in the last 25 years. I mean, there's just the quantity of things that are written about on that. So at one level, I'm quite happy that finally, you know, this awareness of the importance of the value of testing has gone up exponentially. On the other end, I'm a little sad that it took a major health crisis to have that aha moment.”

And to do good testing and good experiments, you have to reduce the noise?

“Yes, absolutely. I'm an engineer by background by the way, and we have a thing called a signal to noise ratio. And what we don't want to be in is an environment where the signal is very strong and the noise is very low, because when you have a high signal to noise, you can then distinguish cause and effect from sort of background noise. And when you have a low signal to noise ratio, you know, you just get a lot of nonsense and you can't really sort of conclude anything from it. So that's what we're doing here. So when you're designing smart experiments or when you're designing smart tests, you want to design them in the way that you have high signal to noise ratio, so you can conclude something from that.”

‍