From my experience the key to serendipity is diversity.

Knowing needs, boundaries and roadblocks is very important - for sure.
But - as I experimented on many occasions - the best thing to do is to match diverse people with different backgrounds in order to get unexpected results which are stretching the boundaries.

People you work with every day most likely know about the same stuff you know.
They all compute in a similar fashion and get to similar results.

The biggest innovation potential lives in the boundaries of common knowledge, where different disciplines overlap (just a little, may be).
I had great stuff coming out when I had an engineer, a designer and a marketer sit together and get rolling.
Their different ways of thinking generates way more creative friction and they complement each other way better than "people of one kind"...

We've been doing breakthrough and some disruptive innovation, including global scale (www.sirruschemistry.com), for over 20 years.

Minimizing or managing if you will serendipity has more to do with being tuned in to the world and all of its major trends and the simplest themes in human needs and their at the moment, emerging and future potential manifestations in how they are or could be met.

Far too few firms engage in the persistent, versus the periodic, activities necessary to do this well.

FYI - take a look at our Adaptive Innovation white paper at www.elemence.net or at at my LinkedIn page.

I think key to encouraging serendipity is to make sure everyone knows what the goals are and what roadblocks stand in the way. It is important to inspire the key innovators. They want to invent something that will be utilized and would love to become the hero. They need to know where the needs of the organization and the market are. Often those with the ability to invent are in the dark about the needs of the company and the needs of the customer.
I have seen examples where R&D staff are immersed in customer research to see how the products are being used and where they are not meeting needs. If it is not possible for everyone to be involved in gathering the insights, sharing videos and stories can go a long way to motivate the technical staff.
PS I remember putting together timelines where we would schedule innovation and label it "a miracle happens".

We have been experimenting with the "innovation pipeline" idea at a largest healthcare system. So far, the results are positive.

The pipeline idea of
ideation -> experimentation -> monetization
is borrowed from pharmaceutical R&D where the serendipity needed to be managed. The funnel shape of the pipeline reflects the "fail fast, fail early" concept.

Our lessons learned are to optimize the "controls" in between the steps. Too little controls burns resources too fast, whereas too much controls crushes creativity.

With such a pipeline, the process can be managed, the culture can be fostered, and the results can be measured.

Creative minds are driving the innovation pipeline. Thus, the end products are "serendipitous".

Innovation is as an incremental process, usually accomplished by those who are not satisfied by stuff working "good enough" and are willing to challenge the old "we've always done it this way" approach.

By incremental, I mean that we couldn't innovate a laptop computer without the desktop just sitting there and asking to be made portable.

Serendipity has something to do with it, but it is a minor component in my view. For example, one could innovate a great product that is ahead of its time and/or due to various market forces (a/k/a bad luck) it never takes off. Sometimes, we just need to cross paths with the right people at the right time (Tesla and Westinghouse...). Yet, without a wide-eyed look at the world, fire in the belly to make it better and a good plan, serendipity just won't do.

Here ya go Julien....hope this is all somehow helpful. Heres our White Paper on Adaptive Innovation.

The best innovation book somewhat recently? "Where Good Ideas Come From" by Steven Johnson.

Here's his Ted Talk - one of the best I've ever heard.

Adam Malofsky
Malofsky@elemence.net

The term "disruptive" was originally used with reference to changing the MARKET; as such it should not be used for inventions ((since it is too early to say at that stage).
A difference should be recognized between INVENTIONS VS INNOVATIONS, with the latter being less likely to have a major impact on the existing market.

A "deliberate" effort to create innovations starts with a question: "How can I do this better?" Serendipity is not likely to play a major role in this process.

On the other hand, investigation of new technical areas while conducting basic research involves accumulating data and knowledge as the work progresses, with a specific goal or destination not necessarily defined. In this case, "serendipity" is in fact a recognition, based on a broader knowledge, that a new observation, property, etc., could have a practical utility. It is in no way a "chance" event.

"Serendipity", as defined in Oxford English Dictionary, is "The occurrence and development of events by chance in a happy or beneficial way". I do not believe that inventions come to pass by chance.

I'm a huge believer in Clay Christensen's 'Jobs to be done' theory from his new book Competing against luck.
I also think its important to define the difference between invention and innovation, I outline that in this article and also look at the definition of a 'valuable idea'.
https://medium.com/@donsmith_64733/on-the-recognition-of-valuable-ideas-f6189e92e7c0

Serendipity in innovation is an outcome, impact or happening that wasn't anticipated or planned. One recently in the Middle East where I found I myself being placed in an office with an influential global leader. After a brief introduction and ongoing discussions, we found very quickly area's of commonality in what we did, what we wanted and in particular risk taking. I was delighted to meet someone with the same vision, a similar approach, experience, market knowledge and mentality (to product development) as my own. Within a few days we were discussing outside the office global scale developments that would add significant value to us both in bringing together our plans - which had nothing to do with the work we both undertaking at the time.

It's the culture that really plays a large role here. Does leadership promote a disruptive innovative culture. as they say culture eat strategy for breakfast