Innovation Engineering: Principles and Methodology

By Ikhlaq Sidhu | May 22, 2019

Let’s start with the problem.  The fact is that most innovation projects fail.  In fact, even when they go forward, they often result in ineffective, overly complex, and/or overly expensive solutions. Sometimes, these projects never get internal funding approvals. Other times, they start but completely miss the mark in terms of serving customers or even solving the problem they were intended to solve.

Innovation Engineering is defined as a method for solving technology and business problems for organizations who want to innovate, adapt, and/or enter new markets using expertise in emerging technologies (e.g. data, AI, system architecture, blockchain), technology business models, innovation culture, and high-performing networks.

When Dave Kelly specified the IDEO process for design in 1971, he changed the predictability of design projects around the world and made each design project more likely to serve its users well. In a similar way, this Innovation Engineering process is intended to make innovation projects in engineering more successful. The process builds upon many best practices in innovation, but it also brings them into a domain of more technically sophisticated areas. 

The concept of Innovation Engineering also integrates many years of observing our students who have engineered novel technologies and companies. The goal is to specify an approach that anyone can use to better architect, design, and more effectively build things that are technically novel, useful, and valuable. And further, the goal is to be able to do this efficiently, on-time, and repeatably.

This article offers both:

1. a list of inductively developed Innovation Engineering principles and,
2. a high-level process based on experimentation with emerging technology projects during the last 15 years at Berkeley and the Sutardja Center. 

At its core, Innovation Engineering is the result of using the approaches, processes, behaviors, and mindsets of entrepreneurs/innovators with the context of engineering projects.  This is illustrated in the figure immediately below.

One thing that we have observed is that innovative technical leaders employ similar behavioral patterns as entrepreneurs even in areas of engineering architecture, design, and implementation.  And further, these behaviors can be amplified within a process.

A high-level process example is shown below.  It simply illustrates the concept of brainstorming a problem/solution, converting the problem/solution into a ‘story’ called a low tech demo, and then using agile sprints to develop the project. The project ends with a final demonstration.  We have used  this simple process in our Data-X course with hundreds of students at UC Berkeley for almost 3 years. And we have found that the results of what people are able to build and demonstrate in just 13 weeks are far more effective than with a standard project process.  (See the actual projects at data-x.blog)

This simple process flow can be extended to include business and/or organizational context. The graphic below shows a process flow for Innovation Engineering with greater detail and broader context.  The flow illustrates that effective projects start always with a story or narrative.  This narrative is generally based on background of the team and an observation of changes in the world (e.g. market, technical, societal, or regulatory changes). When a project does not start with a story narrative, it is typically too narrowly defined and generally goes off target in our experience. Note, the “Low Tech Demo” in the example above maps to the Technical Story in the lower diagram which is used to kick-off an Agile project leading to an Implementation.

The story narrative is used to collect initial stakeholders, resources, and obtain initial validation for the project. In our experience, there is no better way to attract resources than by testing a story and/or initial prototype. 

From here, the story narrative can be broken into two sub-narratives, one for the technical story and another for the broader context or business story.  Each story is the starting point of a learning path, and specifically not an execution path.  The technical path is an agile process that leads to an implementation starting first from the user’s viewpoint.  For example, in Data-X, we use the following components as part of the technical story which we call a “low tech demo”.

Low tech demo outline, an example of a technical story:

1. What is it supposed to do – and ideally why
2. User’s perspective, top 3 user expectations
3. Key technical components with risk levels
4. An architecture, and
5. Short-term plan and assignments towards the simplest demonstration.

In contrast, the business learning path is intended to result in

1. an industry ecosystem of customers, partners, suppliers, etc.  and
2. the discovery of a working business model or fulfillment of a mission in a government organization. 

These learning paths converge when the business model / mission and the technology are all working and integrated. Only after this step can the innovation be scaled via execution and planning. Innovation Engineering tends to focus more on the technical path as required for successful implementation, but must include the broader process as described to be successful.

While all of this is a very quick overview of the process, it does set the context for a set of important principles that are required for the process to be successful. Like with any other organizational activity, Innovation Engineering requires a set of shared beliefs and behaviors to be successful.  These ‘Guiding Principles’ for Innovation Engineering are outlined in the section below are intended to be synergistic with the process flow explained above.

Guiding Principles for Innovation Engineering:

1. Start with Story: Virtually all successful projects start with a story narrative. The story is the means of validation, consensus building, and collecting stakeholders. Any project that starts without a validated story likely jumps to an invalidated conclusion about the problem. Stories can vary in length and complexity, i.e. the problem of a user and its resolution, or the famous NABC story developed at SRI which stands for Needs, Approach, Benefit, and Competition. However, the key to a good story is that there is an insight that others have not seen and that there is substantial benefit of the solution to at least some segment or stakeholder.
2. Scale or Invent: Determine if the project is about creating something new (i.e. a new product, new service, new technology, new customer, etc.) then it’s a learning process, and in that case it requires a team with corresponding behaviors. If the project is about scaling something that already works (i.e. serving more customers, increasing the capacity of a system, etc.) then it’s an execution process best accomplished by someone who has done it or something like it before. In this later case, the team can jump immediately to the scaling phase at the end of the process.
3. User-first: The technical story must highlight a solution first from the user’s viewpoint. Note that entrepreneurial stories typically explain how a venture will both solve a problem and achieve a working business model, the technical story must explain the user’s viewpoint first and only then lead to the system architecture and the implementation.
4. Effectuation: Great technical innovators and entrepreneurs all use “Effectuation Principals” in a natural manner. It roughly means to start with what you have, and sometimes it means you must take inventory of what you have first. To illustrate, if you were to make a dinner, do you first choose an intended dish and then gather the ingredients (not effectuation), or would you look at what you already have in the kitchen and then invent a new recipe from the ingredients you already have (effectuation). This principle can be applied to technical and business projects in the same manner. 
5. Break it down:  Components, interfaces, and interconnections.  Evaluate potential solutions by breaking the proposed system down into simple sub-systems with minimal inter-connections. Understand the interactions and causal relationships between subcomponents.  And of course, if a sub-component already exists or can be easily obtained, then there is no need to build or redesign that subcomponent.  For example, when Tesla created its battery, it created it from thousands of cells that were already being produced in mass scale, instead of designing a completely new battery architecture.
6. Look for Insight in the technical story: This is related to having insight about the location of value, the power, or “the magic” in the system design. What will make it effective or exciting? This principal is a technical parallel to the entrepreneurial behavior of understanding the user’s true needs or what they actually care about, or what they are willing to pay for.
7. Minimal Viable System Architecture: Get as quickly as possible to a 1.0 version. Distill the story as quickly as possible to the simplest possible implementation. From this, a more complex system can be evolved using an agile, iterative model to develop greater capability. This is parallel to the entrepreneurial approach of building a Minimum Viable Product (MVP) for testing product market fit, but in this case the focus is the system architecture for testing technical feasibility.
8. Agile increments: After developing a minimal demonstrable solution, use agile increments to prioritize further development. 
   1. Start with the simplest possible demonstration on the path to the best solution.
   2. Use a technology strategy that allow easiest adaptation.
   3. Be agile driven.  We can’t predict final product in advance.
9. Keep it Simple: The focus of the project should be on keeping the design simple, easy to explain, easy to verify, and easy to debug. Technical architects and designers are often interested in technically brilliant and complex solutions, but true elegance lies in simplicity. As quoted from a historical Apple advertisement, “Simplicity is the ultimate sophistication.”  You might think of this in parallel to timeless works of art, which are characterized by having exactly what is needed to convey the message, but never a single extra music note or an extra paint stroke.
10. Reduce the Downside: Optimize to reduce downside risk and failure, not to maximize performance/cost. Always evaluate corner cases. This is the parallel of broad vs narrow thinking within engineering. The broad thinking version in business would be used to avoid business risk as well as a predict the expected outcome in the broadest terms.
11. Measurable Objectives: Develop measurable objectives to know when goals are being achieved because you cannot improve what you cannot measure. For example, in a data science algorithm, how will you know that the prediction is good enough. Having both a measure and a target allows you to estimate whether the marginal (extra) work to get a better result is worth the expense of doing that extra work.   To understand this more, learn about the concept of “Value of Perfect Information”.
12. Create a support ecosystem: Build a support ecosystem with the highest quality partners that you can both reach and trust. Many technical leaders are tempted to reach out to the lower quality contacts (as team members, suppliers, partners, and customers) who are easiest to contact, but it is better to push our comfort zones to find the best people and organizations that you can — as long as trust can still be generated.

In this article, we have introduced a combination of a process flow as well as a technical set of cultural characteristics or principles that allow people to use Innovation Engineering. Why is this important and powerful? The reason is that even today, most innovation projects continue to fail. Even when skilled technical resources are brought together for a project, the result can still be useless. By compiling and integrating this process and these principles, we have demonstrated at least one effective, validated method for achieving innovation projects. Of course, further variations of this method are inevitable and feedback is always welcome.