The goals of the Output or Impact Metrics are to improve the quality of decision making and the “practice of R&D”. Note that there is overlap between In-Process and Output metrics. Many can be the same; it is just the scope of subject matter under evaluation that changes, e.g. a project or sub-activity within the R&D organization (In-Process), or the output of the entire R&D laboratory or group (Output).

The Output or Impact Metrics use three different approaches for investigating the effectiveness and overall output of R&D. These are:

SECTION 1. The Technology Value Pyramid (TVP) section contains metrics related to:

Value Creation
Managing a Portfolio of R&D Activities
Integration of R&D with Business
Value of Technical Assets
Practice of R&D
What Typical Stakeholders Are Interested In

SECTION 2. The Process of R&D section contains metrics associated with the process of R&D for both a project or a portfolio of projects
Stage Gate / Agile Sprint Model
Stage Gate / Agile Sprint Financial Investment Regimes

SECTION 3. The Critical Success Factors section contains metrics correlated with critical success factors from various studies
Success factors and metrics for new venture success
Success factors and metrics for Strategic Business Unit’s success
Success factors and metrics for a project’s success
A systems approach to sustained growth

SECTION 1. The Technology Value Pyramid (TVP)

This Technology Value Pyramid (TVP) section contains a hierarchy of metrics that evaluate the fundamental elements of R&D and their relationships with business results in the short and long term. This section provides a description of each of the components of the TVP, their interrelationship with one another and implications for their use. The section continues with a list of subjects of interest, a description of the stage-gate process and the use of the Merrifield Criteria for project selection.

The Value Creation sub-section contains hard financial type metrics which directly demonstrate the value of R&D activities to the positioning, profitability and growth of the corporation and creation of shareholder value.

The Managing a Portfolio of R&D Activities sub-section contains metrics which communicate the total R&D program results and which allow optimization of the total R&D program for the corporation’s benefit.

The Integration of R&D with Business sub-section contains metrics which indicate the degree of integration, the commitment of the business to the R&D process and program, teamwork, and ability to exploit technology across the organization.

The Value of Technical Assets sub-section contains metrics that indicate the technology strength of the corporation and relate to the probability of success versus competitors for the opportunities which have been selected by the firm are contained in this category. This is an indicator of the potential for the creation of future value.

The Practice of R&D sub-section contains a wide range of metrics associated with the operation of the R&D organization. Metrics covered include decision making, project generation and selection, use of information technology, environmental policy, personnel development and intellectual property.

The What typical stakeholders Are Interested In sub-section reviews metrics which may be of interest to Stakeholders in the R&D process such as business management, finance, and the CEO.

The TVP encompasses descriptors and a potential ‘menu’ of metrics of the fundamental elements of R&D and the relationships with business results in the short and long term. Not all measurements are right for all companies, or at all times, for describing or tracking the most important aspects of R&D.

This is parallel to a comprehensive financial model of a business that is composed of many more descriptors and ‘metrics’ than are necessary for examining the critical factors of the moment that are needed to guide the decisions of senior managers and investors.

From an R&D perspective the critical factors of the moment are dependent on the situation of the company, the perspective required and the basic dynamics of the model. Let’s start from the top.

The model provides a top-down perspective that is output oriented. ‘Value Creation’ indicators are the prime drivers of overall business returns that are derived from technology-based new products/processes, predictors of business growth and (implicitly) a critical aspect of the soundness of strategic business reviews. These indicators are used to answer critical questions, such as:

Are we spending the right amount on R&D?
Are we getting good returns on our R&D?

If the ‘Value Creation’ indicators are being maintained or going up: (a) the corporation has the likely raw material to extend a technology- based or innovation-based growth program; (b) the investors have the possibility of an extended stream of positive returns from the accumulation of financial pay-offs from technology- based innovations; and (c) the R&D units enjoy the likelihood of consistent funding to reinvest in various aspects of technology application for the near term and base building for the future.

The key words here are likely or possible. ‘Value Creation’ is a necessary but not sufficient condition for growth. It is also only a measure of the moment, whether it is looking to the past or to the future. And, any downward movements will predict the difficulties the business will have in achieving solid gains against the competition. These indicators are crucial to assessing the total returns from R&D investments, whether enough is being spent on R&D, and what is the likely future value of the company from a technology perspective.

‘Value Creation’ is the result of an accumulation of effort by R&D and the business to produce new ideas and to put the best ones into practice. Due to the lag effects and to all of the factors that are involved, momentum is built into these factors over time. They will change, but not rapidly. Change is caused by the drivers of ‘Value Creation’. The drivers are strategies that transform the R&D foundations of competencies, know-how, etc. into specific projects and implementation. These are the strategies that are represented by the ‘Portfolio Assessment’ and the ‘Integration with Business’ indicators.

When ‘Value Creation’ is very positive, these strategies are most likely working well. When ‘Value Creation’ is going in the wrong direction, look first to these areas as to the cause. In fact, one could look at ‘Value Creation’ indicators simply as the double integral of these strategy indicators over time.

Given this importance to ‘Value Creation’, it is no surprise that companies have routinely focused in recent years on methodologies and activities dealing with ‘Portfolio Assessment’ and ‘Integration with Business’ issues. These are correctly perceived as the means to (the ends that) improve the future stream of results from R&D.

‘Portfolio Assessment’ indicators describe the state of the various pipelines that run through the R&D enterprise, as well as the targets that are being pursued. They provide a view of how the R&D $ are being spent in terms of the timing, risks and returns that are possible. The ‘Portfolio Assessment’ indicators are a prime place to look for answers if there are problems with ‘Value Creation’, competitive or market share problems, or if there are problems with internal satisfaction.

Some of the critical management questions affected by the dynamics of these indicators include the following:

Are we allocating the R&D budget optimally (to the elements of the portfolio)?
Are we maximizing our investment yield from R&D?

Unlike ‘Value Creation’ indicators which tend to have modest levels of momentum associated with them and provide significant underpinnings to the business returns, the ‘Portfolio Assessment’ indicators can vary quickly and have little immediate effect on the business. Their major effects are cumulative. This dynamic often leads to short term, risk adverse behaviors that over time undermine the ‘Value Creation’ indicators. Maintaining an aggressive monitoring of the ‘Portfolio Assessment’ indicators is extremely important to the long term support to ‘Value Creation’.

Similarly, when there are problems within the portfolio that are corrected, it is necessary to give the solution enough time to work.

If ‘Portfolio Assessment’ indicators show strategy by what categories of R&D and targets are being developed, the ‘Integration with Business’ indicators show strategy of how it is being done…and, consequently, with what level of quality and execution.

The ‘Integration with Business’ indicators focus on process, culture, teamwork and organization. They also touch on many of the aspects of cycle time. The issues addressed by these indicators change slowly in reality and are probably the true pacing factors that are applied by the organization to the ‘Portfolio Assessment’ indicators that, in turn, put limits on the ‘Value Creation’ that is realizable.

These indicators are often the subject of efforts aimed at TQM, cycle time reduction, or re-engineering. They are difficult to build up to high quality levels because of the various organizational pressures, agendas, incentives, etc. And, they are easy to degrade. The organizational stability of these indicators varies from weak to very strong and are the sum of dozens of behaviors.

When there are difficulties attributable to barriers, for example, that can be removed, then the indicators and the results can be changed relatively rapidly. However, when there are difficulties due to lack of cooperation, lack of contact with the market, lack of good competitive intelligence or with a lack of risk taking, then new attitudes and new behaviors are required. This takes time to build into the culture and the indicators and the results will be slow to change.

In all aspects, the dynamics of ‘Integration with Business’ indicators depend on organizational matters versus those of the ‘Portfolio Assessment” indicators, which depend on investment decisions. And, a consideration of organizational and investment matters quickly brings the factors of business and technology leadership into the model. Second and third order dynamics pushing for immediate change are easy to see as overlays to deeper cultural values.

It is also logical to see both the importance of these two sets of transformation strategy indicators to managers who want immediate change and the criticality of both sets to the impacts on sustained ‘Value Creation’.

Unfortunately, the underlying dynamics do not allow the conversion of management desires for immediate gratification (at low risk) into sustained profitable growth. There must be allowances made for the application of enough time to link all of the elements on an ongoing basis. Over time, all of the indicators will point to a consistent improvement in the transformation strategies and to the attendant output in ‘Value Creation’.

Conversely, monitoring the transformation strategy indicators of an otherwise healthy technology-based enterprise, will show the early warning signs of degradation to the degree possible, presuming the technology foundations are sound.

The foundations for the strategies represented by the ‘Portfolio Assessment’ and the ‘Integration with Business’ are built on the ‘Asset Value of Technology’ and the ‘Practice of R&D Processes’. Some of the critical questions regarding these dynamics are:

Are we becoming more or less productive with our R&D?
Are we building a strong enough future base of competencies?
Are we getting an early warning of any declining capability?

Foundation indicators have the most momentum of any category. They are very slow to change and provide the real rate limitations to growth through technology-based innovation. However, they are also very vulnerable to neglect. They need nurturing, leadership and the execution of well focused technology strategies to become strong elements of a company’s growth foundation. And, just as when weak transformation strategies lead to degradation of ‘Value Creation’, they also degrade the foundations.

Aside from weak transformation strategies, the other major dynamic that affects foundations are external technological changes.

In these cases, technological paradigm shifts occur that undermine the foundations comprised of traditional technical competencies. These bring in new competitors, topple current competitors and may, in fact, redefine the structure of an industry. Although these paradigm shifts take time to develop, R&D and the business are usually both entrenched in the traditional areas, and either don’t see the changes coming or insist on devaluing the importance of them until it is too late. The foundations which, if strong, took considerable time to build, must nonetheless be constantly extended and rebuilt to provide an ongoing set of options that are the logical growth paths for the business’s future. Otherwise, over time, they will significantly erode.

Thus, the dynamics of the foundation indicators are that they are slow to build, a rate limiting source of quality for the options to be taken up by the transformation strategies, and that they are relatively easy to degrade in spite of strong momentum. They are also fundamental to a strong competitive basis for ongoing strategy (based on technical core competencies), to the heart of R&D creativity, productivity and a significant contributor to cycle time reduction. The drivers of these areas are the R&D leadership.

The principal use of the model is for communication and control. This includes the overall R&D program, the research program, the business technical programs and the development of external relationships for which new capabilities are the goal. The metrics are not intended to aid project evaluation except as a project would contribute to the improvement of various categories and their respective indicators.

Depending on particular needs of stakeholders or of decisions to be taken, different categories and factors should be examined. In addition, time periods (unit of analysis) should be adjusted for different industries, technologies and types of R&D.

If the business depends on technology and technology-based innovation, the model has great predictive and diagnostic value. The layers of activity that compose the R&D to business linkage are identified for inspection and monitoring. Taken together, the elements of the model and the structure connecting them, represent a single entity, the R&D enterprise, which may be interpreted and examined in different ways by different groups with different purposes in mind.

Each of the primary stakeholders will tend to concentrate their attention on different parts of the model. And, while that is logical, it is important to note that all of the categories are connected with time lags by the basic dynamics. It remains for R&D and, in particular, the CTO to make sure that some awareness is made of all the categories of the menu, that each stakeholder is reminded of their interconnectedness, and that a consistency is maintained (or that corrective action is taken to achieve consistency) between the collective expectations of the stakeholders and the realities of the model represented by the Technology Value Pyramid.