Below are the specific metrics referenced in the above sections. For each the following is provided:
Advantages and limitations – Benefits of the metric, areas of concern when using the metric.
How to use the metric – Examples of how the metric can be applied. Parameters for measurement.
Options and variations – Different ways to apply the metric.
References – Literature references pertaining to the metric for further investigation.
FINANCIAL RETURN
1. Metric Definition
a. New Sales Ratio
The New Sales Ratio is the % of current sales originating from new products. There are two sub-definitions that are required. What is a new product? and, How old is new?
The most frequently used and simplest definition of a new product is any new SKU (inventory code: Stock Keeping Unit) that has required R&D support to implement. This avoids counting new SKUs which are only packaging changes or other modifications made easily by marketing or manufacturing. When is a new product old is different for each business and technology. In rapidly changing and evolving fields, such as electronic chips and software, new might only be one year, but certainly not more than three. Three years is more likely the norm for businesses that are a mix of fashion and formulation, such as cosmetics & toiletries. And, for more intense capital and industrial products, three to seven is a more likely range to select a number that is right for you.
b. Cost Savings Ratio
The Cost Savings Ratio is the % reduction in cost of goods or cost of operations (including depreciation charges) that are realized in a year to year comparison that have originated from technology changes that are new. Again, the same issues must be resolved as sub-definitions as required for the New Sales Ratio. In other words, what is being attributed to R&D and how long is new?
Since SKUs are not used to catalogue changes made in operations an alternative must be found that will work for your firm. A simple solution can often be created based on the capital approval process. Most companies require a specific approval for all individual capital projects. These can be coded and tracked for R&D involvement and for realized cost savings. However, it is recognized that the accounting involved in examining the impacts on cost savings may be more difficult than that for new sales. It is therefore a more common metric only in those cases which are more significantly impacted by cost savings than by new product sales. And, since new technology for operations or manufacturing has a different useful life than a new producy per se, it must be tracked for a different length of time that is industry specific. In some cases, it may be very linked to the product life in other cases, it may go on much longer. In any case, it is not likely that ‘new’ will reach beyond 7-10 years.
c. R&D Yield
R&D Yield is the contribution of R&D to current financial performance. It is a metric that is composed of definitions from New Sales Ratio and Cost Savings Ratio, plus an evaluation of gross profit from the new sales.
It is the annual combined financial benefit that is derived from the annual gross profit of new products and the annual cost savings of new processes. This is the current contribution that the company receives that is associated with its past stream of R&D investments, i.e. the part of the ‘bottom line’ that is relatively ‘new’ and derived from R&D.
d. R&D Return
R&D Return is the relative ROI measure that relates to R&D. It is composed of the R&D Yield divided by the annual investment in R&D. Hopefully, this is a large number that is proportional to the risks and variances that are part of R&D.
2. Advantages and Limitations
The advantages of these financial metrics are that they relate directly to the financial benefits to the company, they are quantitative and they are comparable to metrics that can be used in different parts of the same firm or between firms. They capture the degree to which R&D is truly making a financial contribution to the value of the enterprise. They answer the question: What have you [R&D] done for me [the Business] lately? However, they only represent the tip of a process that takes place over a number of years and that involves other functions besides R&D. This means that the numbers reflected by these metrics are associated with activities that are in the past. These metrics are lagging indicators. They are a nice track record, but they may not be reflecting accurately a current level of effectiveness.
3. How to Use the Metric
The metrics should be tracked at least on a once a year basis. Because of measurement and definition problems, a baseline of two years or more of historical data is needed before accurate judgments can be made about trends and ratio efficiencies.
The metrics should be examined carefully for consistency with business strategies and the results required vs. the investments in R&D. In situations where the metrics, requirements and available resources are not in balance, there will be a difficulty in executing the overall business and technology strategies. One or the other must be shifted, and variations in how R&D is conducted need to be examined.
If the Financial Return metrics are being maintained or going up: the corporation has the likely raw material to extend a technology-based or innovation-based growth program; the investors have the possibility of an extended stream of positive returns from the accumulation of financial pay-offs from technology-based innovations; and the R&D units enjoy the likelihood of consistent funding to reinvest in various aspects of technology application for the near term and base building for the future.
The key words here are likely or possible. Positive Financial Returns are a necessary but not sufficient condition for growth. It is also only a measure of the moment, whether it is looking to the past or to the future. And, any downward movements will predict the difficulties the business will have in achieving solid gains against the competition. These indicators are crucial to assessing the total returns from R&D investments, whether enough is being spent on R&D, and what is the likely future value of the company from a technology perspective.
4. Options and Variations
There is always something new and innovative that is contributing to revenue or profit. These financial return metrics are intended to capture the new portion of these changes in the business that are related to R&D. They require definitions of what is to be considered new and for how long. These elements can then be tracked separately or together, in ratio or absolute form, by themselves as benefits or as an investment return vs R&D. The options and variations fall into place based on each company’s views of these items.
The most common variations are based on the length of time that is new. the most frequent categories are three years, five years and seven years. Another variation is to use these same metrics in a prospective, future mode.
PROJECTED VALUE OF THE R&D PIPELINE
1. Metric Definition
a. Projected Sales
Projected Sales is the calculated sum of future sales from current R&D projects. This metric may be expressed in absolute terms or a % of future sales.
Definitions must be provided for how this is to be evaluated. Normally, it is the forward side of the newness range, i.e. if new products are those introduced within five years, projected sales will be calculated for five years after commercialization. A probability of attainment is usually figured into this metric.
b. Projected Income
Projected Income is the income stream associated with the Projected Sales. Similar definitions apply. This metric may be expressed as an absolute number or as a fraction of net income.
2. Advantages and Limitations
This metric provides an ongoing anticipation of the expected results from R&D. Because it is projected, it provides an evaluation of the benefits that are being created with today’s R&D investments.
The limitations are due to the intrinsic difficulties of obtaining estimates about the likely commercial benefits if the technology is successful.
3. How to Use the Metric
This is perhaps the single most important and least used metrics. It is the singular indication of the future business that is to be developed as a result of successful outcomes from R&D. Are the sales impacts large enough, are there enough new products, is the timing of elements in the pipeline adequate, are the overall returns related to R&D adequate? Are these metrics on a year to year basis showing constancy, increase or decline?
This metric provides ongoing guidance to the company regarding the future gain to be expected in the business due to R&D. This should be used as a check that both the strategy and the resource allocations are correct.
If this metric is staying constant or increasing, particularly with respect to the R&D resources, then the effectiveness of R&D is being maintained or increasing.
If, alternatively, this metric is declining, then further diagnostics should be examined to understand the reasons and to take corrective action.
4. Options and Variations
Common options are few because this is an underused metric. One is to look at absolute sales or net income over a five year horizon on the presumption that some projects will finish and be productive within the next two-three years and that others will impact a bit later, i.e. in the fourth and fifth year. A related option is not to adjust the commercial impact by any probabilities.
COMPARATIVE MANUFACTURING COST
1. Metric Definition
Benchmarked manufacturing cost data vs. competition for same type of unit cost.
2. Advantages and Limitations
It is extremely useful and in some companies paramount to know how R&D is helping to provide an advantaged cost position to the operations. This metric reflects the quantification of that goal. Unfortunately, cost accounting and even further comparative cost accounting can be extremely difficult.
3. How to Use the Metric
While most firms have very accurate manufacturing cost data for thremselves, the generation of accurate manufacturing cost data for competition is considerably more difficult. Therefore, when using this measurement, there should be an estimate made of the variance of the competitive estimates.
4. Options and Variations
This metric is intended to be based on unit process comparisons. There can be many options created that are aggregates of production processes, but these simplifications can be misleading. Therefore caution is urged.
PRODUCT QUALITY AND RELIABILITY
1. Metric Definition
1.1. Customer or Consumer Evaluation.
Relative quality and reliability compared to competitive products through evaluation by customers or consumers.
1.2. Reliability/Defect Rate Assessment.
Fraction of a firm’s output, either by individual product or by sum of all products, that meets or exceeds the established quality standards.
2. Advantages and Limitations
2.1 The advantage of this metric is that product benefits resulting from R&D activities are directly evaluated by the customer or consumer. Comparison with the competitors’ products is usually the basis for evaluation. Limitations of the metric chiefly are related to the reliability and accuracy of the survey techniques chosen as appropriate for the industry, though firms usually gain confidence in their preferred methods through repeated use and incremental improvement.
2.2 Similar to the preceding discussion, the advantage to this metric is that the direct benefit from R&D activities can be obtained through specific measurements made by the firm.
3. How to Use the Metric
For the Customer or Consumer Evaluation metric, each firm will generally have a preferred technique for directly or indirectly obtaining data showing how well the firm’s products perform in comparison to competitive products. Data for the Reliability/Defect Rate
Assessment result from internal quality measurements. When both metrics are utilized, the impact of product quality improvements on customer satisfaction should be demonstrated. Product Quality and Reliability metrics are retrospective, showing the results of past technology or product introduction to the market.
4. Options and Variations
This metric fits well with the trend toward greater input to R&D planning from customers and consumers and with the “Quality” protocols that have adopted by firms of all types. Though basically a retrospective measure, product needs that arise during the data collection can be used for prospective purposes.
GROSS PROFIT MARGIN
1. Metric Definition
Gross Profit as a percentage of sales, where gross profit equals net sales minus cost of goods sold (product costs plus direct manufacturing costs).
2. Advantages and Limitations
To some extent, gross profit margin reflects value of the firm’s technology assets and the value created by R&D. However, raw material, production, and distribution costs also directly affect the firm’s gross profit margin. Each firm should attempt to understand the correlation of Gross Profit Margin to R&D effectiveness.
3. How to Use the Metric
Value assessment should be based on change in gross profit margin from period to period. (Periods should be appropriate to an industry and may be in excess of one year.) Changes in Gross Profit Margin in relationship to changes in values of other metrics (e.g., financial return , technology transferred to manufacturing, sales protected by proprietary position) should be followed in an attempt to uncouple the contribution of R&D from other factors. This is a retrospective metric that can be used as a benchmark with the competition, if gross margin data from competitive firms are available.
MARKET SHARE
1. Metric Definition
Firm (or business unit) market share in various product categories measured as appropriate for the industry or category, expressed as a percentage of the total market.
2. Advantages and Limitations
This metric is meant to reflect value creation for the firm and the value of the firm’s technology. Similar to Gross Profit Margin metrics, caution must be taken in the interpretation of Market Share data from the perspective of measurement of the contribution of R&D activities to the whole. There can be many confounding factors in a market share determination, such as the size and quality of the marketing effort, the competitive response, the relative state of the economy, etc.
3. How to Use the Metric
Changes in Market Share should be assessed at least annually to determine the rate of progress or decline. The expectation is that improvement in a firm’s technologies and products will result in a greater share of the market. Market Share is a retrospective metric, showing the results of past technology or product introduction to the market. Since competitors’ market share data is usually also available, this is a metric that can be used as a benchmark with the competition.
4. Options and Variations
As an indication of threats or opportunities, share data in markets related to a firm’s products can be followed. This “Related Market Share” metric can serve as a component of a strategy to anticipate the potential application of similar technology into a firm’s marketplace.
STRATEGIC ALIGNMENT
1. Definition
This R&D metric assess the degree of alignment of an R&D project or an R&D portfolio with the strategic intents of the company or a division of the company. The strategic intents are often the corporate goals embodied in its business plan.
2. Why it is used
This metric is used to gauge the degree of relevance of the R&D program to the corporate goals. The strategic intents of a company may change more rapidly than the R&D program can respond to those changes creating various degrees of misalignment. The misalignment can be with regard to work area, long term versus short term needs, or degree of risk.
3. How to use metric
There are a number of ways this metric can be applied, including both prospective and retrospective views. It can be applied by R&D management, general management, or by both working in partnership. An alignment index would be assigned to each project; a linear scale of 1-5, for example, would work well. These scores may also have value when considering relative merit of individual projects in the portfolio. A composite score for the entire portfolio would then be determined. This could be a weighted average reflecting sizes of projects with regard to technical head count, project budget or some other appropriate weighting factor. Once a baseline for alignment has been established, R&D management can then decide if and how this index should migrate to greater or lesser degrees of alignment through modifications of the portfolio. Applied to individual projects, there could be a cut-off point for the alignment parameter below which projects are not supported.
Consideration must be given to the degree of alignment desired. Although the desired state in many cases is toward greater degrees of alignment, you can envision situations where that may not be the case. A research organization charged with taking the company in new directions may not want its project portfolio highly aligned with the current business plan. A more visionary business plan may capture new directions as well as current businesses, but many organizations find that some degree of decoupling is desirable.
4. Options
Both retrospective and prospective views are options of this metric. The retrospective view entails applying the metric to an existing portfolio of R&D projects to determine the degree of the portfolio with corporate goals.
The prospective view is to apply the metric to a proposed project or slate of projects. If the R&D organization is trying to increase its alignment index, then the management will be less likely to initiate projects that move the composite score in the wrong direction. Similarly, an alignment index cut-off may be instituted. Projects falling below some minimum value of alignment would not be supported.
5. References:
Third Generation R&D, P. A. Roussel, K. N. Saad, T. J. Erickson, Harvard
Business School Pres, Boston, MA (1991); ISBN 0-87584-252-6
Winning in High Tech Markets, Joseph G. Morone, Harvard Business School
Press, Boston, MA (1993), ISBN 0-87584-325-5
DISTRIBUTION OF TECHNICAL INVESTMENT
1. Metric Definition
This metric provides a means of assessing how well an R&D program is protecting the technology investment and technical position of the company. It forces consideration of how the technical assets should be distributed, setting directions for modifying the R&D portfolio.
2. Advantages and Limitations
The portfolio of an R&D organization may not be protecting the strategic interests of the company for any number of reasons such as skill set mismatch, slow response to changes in the company’s mission and markets, and a rapidly changing competitive environment. There can be an over-emphasis on certain business units and products. This metric causes the management to examine how well the R&D effort is protecting and expanding the technical position of the company in areas of greatest importance.
3. How to Use the Metric
This metric is applied by first determining how the technology investment should be distributed. As an example, consider a company with six business units. The R&D portfolio can be distributed among these six business according to a number of models. Six examples of distribution models are listed below:
The revenue that each business generates.
The opportunity market share (potential market growth). [1]
The impact that technology can make in the different business units.
Competitive Impact (Base, Key, Pacing) [2]
Some combination of the above distribution models.
The profitability that each business demonstrates.
These considerations often involve the concept of the technological basis of competition [1], that is how does technology provide a sustainable competitive advantage in a particular product or market. The discussion could also consider the distribution of the competitive impact of the company’s technology investment by categorizing them as Base, Key or Pacing [1]. Base technologies are essential to the business but widely exploited by competitors. Key technologies are highly differentiating to the company’s current products. Pacing technologies are new technologies where the competitive impact is less certain but likely to be high. All three of the impact categories require protection of the competitive position, but the distribution of resources among the three categories may vary based on the company’s business plans.
Once a distribution model has been agreed upon, the portfolio is measured against that model. The distribution could be with regard to number of R&D projects in each segment of the distribution, head count devoted to each segment, or expected value of projects in each segment. Modifications are made to the portfolio to move the distribution toward the desired state.
4. Options
Both prospective and retrospective views are supported by this metric. In the retrospective view, the R&D projects are categorized according to the corporate investments or markets they are intended to protect or create, or the competitive impact that they offer. This does not have to address the entire R&D effort of the company, since the metric can be applied to any subset of the portfolio. The projects are appropriately weighted to reflect their size and cost. A distribution of the R&D efforts supporting each of the categories is determined. This current state distribution is then compared with the desired state. The degree of misappropriation can then be quantified as the fraction or percentage of the R&D effort that is improperly distributed.
The prospective view for this metric involves consideration of how a proposed project shifts the distribution of technology investment toward or away from the desired state.
5. References
Competing for the Future, G. Hamel and C.K. Prahalad, Harvard Business School Press, Boston, MA (1994); ISBN 0-87584-416-2
Third Generation R&D, P.A. Roussel, K. N. Saad, T. J. Erickson, Harvard Business School Press, Boston, MA (1991); ISBN 0-87584- 252-6
Curtis, C. C., Non-Financial Performance Measures in New Product Development, J. of Cost Management, 8(3): 18-26. (This article addresses the distribution in terms of major projects, minor projects and extensions, and relates these to financial measurements.)
NUMBER OF WAYS TECHNOLOGY IS EXPLOITED
1. Metric Definition
This metric assess the number of ways a technical asset can bring value to the corporation.
2. Advantages and Limitations
This metric is applied to gauge project attractiveness, or to understand the value of a technical asset already developed. It is generally agreed that a larger number of potential uses, both within the company’s current markets and in markets not yet developed, makes a technical asset more valuable. The metric is a bit arbitrary and can be misleading in cases where there are few, though very large and/or lucrative commercialization.
3. How to Apply the Metric
This metric is applied by taking an existing or potential technical asset, such as a project to develop a new type of lower cost, light weight composite material, and conducting a thoughtful analysis of how many ways this asset can be exploited commercially. The count could consider:
Number of business units in the corporation that could make use of the asset
Number of markets the company serves that could be impacted by this technology
Total number of markets served by the corporation and other companies where the technology may have an impact.
Number of products that could utilize the technical asset.
Used in this way, the metric is a single numerical value. A larger number of potential uses means that the corporation is not depending on a single or small number of products to succeed in order for the technical asset to deliver value. The risks associated with the exploitation of the technical asset are spread over a larger number of potential uses. A larger number also provides greater opportunity for unforeseen benefits, like taking the company into new markets and new products.
Using the low cost, light weight composites as an example, the primary market for the company may be the automotive market, with four different auto parts that could use the strong, light weight tubes produced by the new process. In addition, there could be a market for the technology in the aeronautical industry, served by another business unit of the company. The third exploitation could be in high performance bicycle frames, a market that is new to the company.
4. Options
An option in the implementation of this metric can involve adjusting the number for relative importance of the commercialization, or keeping sub-metrics of the number of exploitations with certain value ranges. These might be segmented as:
No. of Markets for Technical Asset: 3
No. of Product Offerings 8
No. of Product Offerings with (Value > $10M) 2
No. of Product Offerings with ($2M < Value < $10M) 3
No. of Product Offerings with (Value < $2M) 3
Another technical asset which also had eight product exploitations, but with two products in the ($2M < Value < $10M) category and six products in the (Value < $2M) category is not as attractive as the light weight composite project.
5. References
Competing for the Future, G. Hamel and C.K. Prahalad, Harvard Business School Press, Boston, MA (1994); ISBN 0-87584-416-2
NUMBER OF PROJECTS HAVING BUSINESS/MARKETING APPROVAL
1. Metric Definition
Percent of projects in the total R&D portfolio with explicit business unit and or corporate business management sign-off.
The intent of this metric is to provide an indicator of the degree of alignment with business and corporate strategy and tactics. The metric is closely related in some corporate structures to metric 12, “percent Funding by the Business”
2. Advantages and Limitations
Advantages: Several studies have suggested that close alignment of R&D to marketing and to business and corporate strategies increases the odds of success for new products and processes. Thus actions which drive this metric to higher values can be expected to improve the amount of R&D spent on successful projects and the predictability of the outcome from R&D efforts.
Limitations: The metric will be valuable to the extent business/marketing management and R&D management jointly develop strategy and plans. Use of the metric to drive R&D without such teamwork will likely lead to short term projects and suboptimal use of R&D resources. In those companies where R&D is corporately funded, business/marketing management may also be tempted to give approval to projects in their market segments to insure that they receive “their share” of R&D resources. Finally if the corporation uses a formal innovation process which requires business/marketing approval at some stage, the metric runs the risk of becoming a measure of compliance with use of the process or a measure of the percent of project past the approval stage.
3. How to Use the Metric
Explicit approval may be sought at any point in the innovation process. Seeking approval early in the innovation process probably provides maximum value. One form of approval is the provision of a sales forecast from marketing management for each new product and agreement to commercialize if the product meets technical requirements in a timely manner.
The level of approval from the marketing/business management and the point where approval should be sought should be explicitly defined if the firm uses a formal innovation process. If not, the level should be commensurate with the amount of R&D resources and commercialization resources which will be required.
4. Options and Variations
For projects having broad corporate strategic value, approval of a director of corporate planning or director of corporate business development might be an appropriate substitute for the business/marketing management approval. For corporations where out licensing of technology is a major thrust, approval of a director or vice president of licensing may be an appropriate substitution.
USE OF PROJECT MILESTONE SYSTEM
1. Metric Definition
1.1 Percent of projects in the total portfolio going through a defined project management system with defined milestones.
1.2 Percent of R&D expenditure on projects using a defined project management system with defined milestones.
2. Advantages and Limitations
Project management systems including milestones can provide a way of reducing cycle time and providing R&D and business management with a sense of the health of projects. These systems also can improve the linkage between R&D, marketing and business management by enabling coordination with manufacturing and marketing to hit key windows. When these systems include documentation through status reports, they can be used to promote organizational learning. Driving behaviors which increase the numerical value of this metric should therefore improve both the linkage of R&D to the business and improve the effectiveness of R&D.
When used with a formal stage gate process this metric provides a measure of compliance with that system. Since companies will generally use a defined project management system and establish milestones in the later phases of innovation, this metric may also be an indicator of the distribution of projects in the innovation pipeline (see metric 8 “Distribution of Technology Investment” ).
The metric may be limited by the difficulty of counting projects outside the project management system. Further since project management systems may not be appropriate early in the innovation process, the ideal value for this metric will depend on the firm’s desired balance of early and late stage projects. For short term projects such as minor product or process variations, use of formal project management systems and this metric may create unnecessary red tape and potential delays.
3. How to use the Metric
3.1 As a concurrent metric, the total number of projects with defined (written) project plans including definite milestone dates can be divided by the total number of identifiable R&D projects (X100) to calculate the metric.
3.2 When used with the appropriate accounting system this metric can be calculated from the cost of projects divided by the total R&D cost. In this case the budgeted projects should be audited to determine compliance with requirements for plans and milestones.
4. Options and Variations
The metric can be used as a concurrent metric (a snapshot of the current R&D activities) or as a retrospective measure to determine how many R&D projects used a defined process. It should be equally suitable for service and manufacturing companies.
PERCENT FUNDING BY THE BUSINESS
1. Metric Definition
Fraction (or percent) of the R&D budget or actual expenditure from business unit sources.
As used in this metric, the intent is that the funding carries with it program control. That is, an R&D “Tax” imposed at the corporate level on businesses should not be counted as a business source if the business does not have program control and the ability to vary the level of funding.
2. Advantages and Limitations
For companies which use central or corporate laboratories, this metric can provide an indicator of linkage to the business unit strategy since presumably only programs and projects which support that strategy would be funded. The ideal value of this metric will depend on the nature of the company (whether businesses are closely aligned to each other or the company is more of a conglomerate) and the corporate strategy (growth and improvement in existing businesses versus R&D for growth outside current business). The metric could be used for comparisons within an industry and for tracking trends within a company.
Limitations: By itself, this metric cannot indicate the degree of collaboration between the business and technology communities in establishing strategy. An emphasis on driving this metric to high values in the absence of long term business strategies can lead to short term oriented R&D which could endanger the long term health of the firm. Moreover, overemphasis on business funded R&D can reduce funding of projects which support overall corporate competencies to benefit more than one business.
3. How to use the Metric
In most companies the accounting system should be able to provide the required numbers. R&D overhead costs should be treated in a consistent manner to enable comparisons to be made between companies. Generally these are allocated as a percentage of direct R&D costs.
4. Options and Variations
The metric can be used prospectively as budgets are assembled or retrospectively with actual expenditure data. It should be equally applicable to service and manufacturing industries.
TECHNOLOGY TRANSFER TO MANUFACTURING
1. Metric Definition
1.1. Amount of Technology Successfully Transferred to Manufacturing.
The output of R&D is technology which must be embodied into products or services. This metric measures the amount of technology which is successfully transferred to manufacturing. It is a measure of value created by the R&D function. The form of the metric may vary:
Number or percent of projects transferred
$ of R&D expense or percent of total R&D expense related to transferred projects
Projected Value (future sales or profits) of projects transferred.
1.2. Quality of the Technology Transfer Process.
A firm may rate its technology transfer process by using a subjective scale ranging from 1 to 4. Firms operating at Level 1 are characterized by unsupported hand-offs. Firms operating at Level 4 are characterized by a development process which includes involvement by manufacturing in all phases of the project in order to ensure that the transfer will be seamless. This metric is one of many relating to how well the firm conducts the Practice of the R&D Process. It also relates to how well R&D is integrated with the business.
2. Advantages and Limitations
The advantage of this type of metric is that it relates to value created by the R&D function without needing several years to collect data. It is a surrogate for financial return. Presumably, the more technology transferred to manufacturing, the higher will be the financial return. However, it should be clear that this is not necessarily the case. Some technologies transferred could result in financial losses.
The measurement of the quality of the transfer process can be used to diagnose problems with the process and plan for improvement.
3. How to use the Metric
The amount of technology transferred to manufacturing during a time period, usually one year, should be tracked over time. The expectation is that a steady state will be achieved with a flow of technology occurring at a rate which can be accommodated by manufacturing and which meets the needs of the firm for new products and services.
The projected value of the financial return from projects transferred during a time period should also be tracked over time. The expectation is that the value will increase due to the choice of better projects and more efficient management of the R&D process. Actual financial returns should be compared to the projections, although there will be a large time lag. Efforts should be made to improve the quality of the projections.
These metrics can be used retrospectively to measure the output of R&D over the past period, or prospectively to set targets for future accomplishments. The use of projected value is a prospective estimate of financial return.
4. Options and Variations
Service companies and companies who sell technology may wish to use this metric to relate to transfer of the output of R&D in a more general sense to manufacturing, into services, for sale, for license, or any other use appropriate to the firm and which would be considered a successful outcome that creates value.
USE OF CROSS-FUNCTIONAL TEAMS
1. Metric Definition
1.1. Number of Cross-functional Teams
Current management philosophy suggests that the use of cross-functional teams will improve the effectiveness and efficiency of the R&D Process and will help R&D to be integrated with the business. The number of such teams can be counted if they are established on a formal basis.
1.2. Evaluation of the use of Cross-functional Teams.
A firm may rate its practice of using cross-functional teams by using a subjective scale ranging from 1 to 4. Firms operating at Level 1 are characterized by the existence of strong organizational boundaries, lack of cross-functional involvement in R&D projects, and no cross-functional team structure. Firms operating at Level 4 are characterized by a well developed and supported team structure which effectively places all R&D work in cross-functional teams responsible for the entire project rather than functional silos responsible for parts. This metric is one of many relating to how well the firm conducts the Practice of the R&D Process. It also relates to how well R&D is integrated with the business.
2. Advantages and Limitations
This metric will be important if the use of cross-functional teams contributes to the effectiveness of the R&D function in the firm, given its peculiar situation. This is usually thought to be the case, but there may be cases where other factors are more dominant.
Evaluation of the use of teams can be used to diagnose problems with the organization or the R&D process and to plan for improvement.
3. How to use the Metric
A simple count of teams is rarely as valuable as an assessment of the how the firm uses such teams. This can be accomplished using the rating scale suggested above. Input for the evaluation should be gathered from a broad cross-section of the firm. The evaluation is important for planning improvements.
Trends across time are probably more valuable than benchmarking.
RATING OF TECHNOLOGY FEATURES AND BENEFITS
1. Metric Definition
The value of technology output from an R&D organization is closely related to the Technology Features (those attributes of the technology which are intended to enhance competitiveness) and Technology Benefits (those attributes of the technology which are recognized and valued by the market).
1.1 Product Features & Benefits
Metric 1: Competitive Technical Performance of Product (Project Metric)
Comparison of technical performance of a product in those dimensions where the customer is likely to perceive a benefit. This may be used in a prospective sense to appraise the value of a feature or in a retrospective sense to register the value of a benefit which the market has recognized.
Examples include:
The use level required to achieve a needed result in the customer application.
The yield strength of a high-performance alloy
The measured softness provided by a textile softener.
The measured UV resistance of an external architectural coating.
Metric 2: Customer rating of Products (Business Segment or Firm)
Customer rating using a scale of 1-5 of the technology benefits that is perceived in a firm’s products. This can be compared to the rating for the best competitor, usually in the form of a ratio. This is an aggregate subjective measure for a business segment or for the firm.
Metric 3: Economic Value of Products (Project, Business Segment, or Firm) This metric is the price differential per unit obtained by virtue of the technology feature minus the cost of providing the feature. The differential can be multiplied by the volume to assess the total benefit to the firm.
Metric 4: Market Share Evaluation (Business Segment or Firm)
If differential pricing does not occur, the advantages of superior product technology can appear as differential market share. In this case, the relative market share (the firm’s share divided by the largest share) can be used as a surrogate for the value of technology embodied in the products.
1.2 Process Features & Benefits:
Metric 5: Competitive Technical Performance of Process (Project Metric)
Comparison of technical performance of a product in those dimensions which are important to manufacturing cost or product performance. This may be used in a prospective sense to appraise the expected value of a new or improved process or in a retrospective sense to register the demonstrated value.
Examples include:
Manpower requirements
Efficiencies of raw material conversion
By-product or coproduct costs or values
Consistency, controllability, and other such quality parameters.
1.2.2 Metric 6: Economic Value of Processes (Project, Business Segment, or Firm)
The differential in profitability (versus the target or competitor) attributable to new or improved process technology.
Metric 7: Profitability Evaluation of Processes (Business Segment or Firm)
In the same way that Market Share is a surrogate measure of product performance in those business areas where the basis of competition is product performance, overall profitability in a business segment of the firm is a measure of process performance, in those business areas where the basis of competition is cost and/or quality.
2. Advantages and Limitations
Product Metrics 1 through 4 attempt to assign value to technology used in products. However, differential value or market share are normally the result of many different factors. These metrics can give some indications if factors are carefully sifted, but may be misleading if the analysis is superficial. Objective measurement of product performance and customer ratings relative to competitors are the most accurate measures of product technology. But note that comparison of features that the market has not recognized as benefits may be self-serving and deceptive. Competitive rankings of measured product performance and or customer ratings may be averaged over market segments or over the firm to obtain average values.
Process Metrics 5 through 7 attempt to assign value to new or improved process technology resulting from R&D. However, economic value and profitability are normally the result of many different factors. Objective measurement of process performance is the most accurate measure of process technology.
3. How to use the Metric
Metric 1
Define the key parameters which measure features the customer is likely to perceive as benefits. Measure product performance as accurately as methods allow. Compare to the same measurements of competitive products. Rank performance versus best competitors.
Metric 2
Ask customers to rate the technology attributes of a product line relative to solving their problems. This requires a carefully constructed survey instrument.
Metric 3
This metric is valid for products which are differentiated by performance. It is not applicable to commodity products.
Metric 4
See Definition
Metric 5
Define the key parameters which measure or impact cost or quality. Measure process performance as accurately as methods allow. Compare to the same measurements of competitive processes. Rank performance versus best competitors.
Metric 6
See Definition
Metric 7
See Definition
These metrics may be used retrospectively to measure the output of R&D over the past period and prospectively to set targets for future accomplishments.
4. Options & Variations
There are many variants on these metrics. Exploring these is beyond the scope of this document.
5. References
Ellis, L. W., and Curtis, C. C. 1995. Measuring Customer Satisfaction. Research/Technology Management, in press.
Kaplan, R.S. and Norton, D. P., 1993. Putting the Balanced Scorecard to Work, Harvard Business Review, September-October, 134-147.
RESPONSE TIME TO COMPETITORS MOVES
1. Metric Definition
This metric measures the ability of the firm to respond to new technical innovations introduced by competitors. Depending upon corporate strategy, it could be the time required to match or exceed the competitive offering.
2. Advantages and Limitations
This metric is an indicator of technical leadership in a given field. The technical leader will not spend a significant amount of time matching competitive innovations, while a follower will be more reactive than proactive. The utilization of resources for this function as a percentage of total resources should be tracked over time. An increase in this percentage would indicate that new technology programs are not as effective as desired and technical position relative to the competition is eroding. This metric also measures the flexibility and creativity of the firm to change priorities to meet competitive challenges. A strong market intelligence function is required to identify competitive entries at an early stage in the introduction and to assess the technical merits of the offering so that appropriate responses can be made. The technical merits of competitive offerings must be critically assessed to differentiate from market repositioning of existing technology.
3. How to use the metric
The time between the introduction of a competitive offering and the internal development of a comparable or superior offering can be measured and compared to product development times for similar products. A rating system of 1 to 4 can be used.
1. Organization slow to recognize significant competitive offering in the marketplace; slow to launch program to respond; unable to get their offering into the marketplace in acceptable time to be combatant. Slow to recognize impact of technical innovations.
2. Organization recognizes need for competitive offering; has trouble in launching program to develop counter-offering and gets offering to market barely in time to have impact.
3. Organization responds to competitive offering and develops counter offering to maintain relative position.
4. Organization anticipates potential for competitive offering and has counter offering into the marketplace with superior product allowing a gain in market/competitive position. Very fast to recognize impact of any hints of technical innovations.
The percentage of resources used in matching competitive moves could be measured over time. An increase in spending in this function should raise questions regarding core research and development efforts for organizations who strive for technical leadership.
4. Options and Variations
The importance of this metric will depend upon the technology strategy of the firm. Firms pursuing technical leadership will be very interested in minimizing response time. The importance may vary across different strategic segments in the same firm.
COMPARATIVE TECHNOLOGY INVESTMENT
1. Metric Definition
This measures the current annual expenditure for R&D staff and capital compared to the best competitor and/or the industry average.
2. Advantages and Limitations
This metric measures the rate of current activity in developing the technology of interest with the intent of predicting whether the firm is expected to gain or lose ground in the technology. It should be kept separately for the KEY and PACING technologies most critical to the strategy. Retrospectively it measures the efficiency of the investment in meeting new product and technology development goals.
This metric should be as quantitative as possible, but in some industries it may be necessary to make estimates as to the size of the development effort of the best competitor and the industry average. Analyses must be based on a comparison of similar functions. For example, some organizations include sales support as part of their report for technology expenditures. The components of the expenditures under study must be understood in making the comparison.
3. How to use the Metric
Information on a firm s overall technology expenditures are available in the firm s annual reports or industry publications. These can be used for comparison to internal overall investment. Information by industry is available in industry publications and from organizations such as IRI (IRI/CIMS survey).
Rationing the firm s current investment in technology versus the best competitor and industry averages provides an insight into the efficiency of the technology investment. Performance exceeding expectations in value creation goals at competitive investment rates indicates an efficient organization while sub-standard performance raises concern about the quality of the investment. Smaller firms may require an investment higher than industry norms to maintain a competitive position to offset critical mass issues.
QUALITY OF PERSONNEL
1. Metric Definition
This is a measurement of the skills and ability of the R&D staff to execute strategic programs.
1.1 Internal Customer Ratings.
Internal customers rate the quality of the R&D staff on their ability to execute programs. Measures such as percentage of mileposts met versus project plans, novelty of concepts, patentability of concepts, and competitive advantage of the technology are parameters that can be considered.
1.2 External Customer Ratings.
External customers rate the quality of the R&D staff of their ability to meet customer expectations. Problem solving, novelty of approach, responsiveness, knowledge of customer’s operations are parameters that can be considered.
1.3 External Recognition.
Publications in refereed or industry trade journals, external presentations, citations in the literature, invited lectures and patents are parameters to be considered.
2. Advantages and Limitations
The internal and external customer ratings measure the ability of R&D to meet customer expectations and contribute to the growth of the corporation or enhance competitiveness. They are largely objective measures that can be tied to tangible value.
The value of external recognition via patents, publications and presentations is more difficult to measure objectively. The numbers of different subjects covered by public disclosures should be evaluated rather than the total number of all disclosures. It is easy to become subverted to a self servicing metric if only numbers are considered. Maintenance of technology as trade secrets must be considered in this evaluation.
3. How to use the Metric
Internal customer surveys can be conducted using a 1 to 4 scale for rating. A 1 represents below standard execution on a given project. The causes for this poor performance have to be determined since they could arise from inadequate skills, poor judgment, lack of responsiveness, poor planning, etc. The causes may not be related to the quality of the personnel but poor management practices. Having multiple internal customers (marketing, manufacturing, sales, etc.), conduct the evaluation is a form of 360o review. Superior performance by reaching targets ahead of schedule, lower than expected costs, developing a significant competitive advantage, etc., should be rated as a 4. In establishing this survey system, agreement should be reached on the different levels of performance. The survey should be applied to different projects with the same population of the internal customers as raters. This rating should be conducted on a regular basis and over time trends will emerge.
External customer surveys should be conducted using the same 1 to 4 scale. A 1 rating would indicate that the customer was not satisfied with the parameter being measured, while a 4 would indicate that expectations were exceeded. Key parameters should be selected beforehand and could include timeliness of response, knowledge of products, knowledge of customer s operations and knowledge of customer needs. The parameters will vary by industry. Several levels of the customer s organization should be sampled such as plant operators, first line supervisors and management. A simple postcard type of survey instrument mailed after customer contact can be used. A database can be developed over time and trends will emerge.
The subjects covered in public disclosure should be tabulated and compared to strategic technology goals. A subjective 1 to 4 rating system can be created by R&D management to determine fit with the goals. A rate of 1 corresponds to a poor fit and a need to enhance skills, while a 4 implies that all areas are being addressed.
4. Options and Variation
Service and consumer product companies may find the external customer survey to be a valuable tool in assessing the effectiveness of their R&D organization. One may also measure the quality by the number and type of external awards from recognized organizations (ACS, AIChE, IRI …)
DEVELOPMENT CYCLE TIME
1. Metric Definition
1.1 Market Cycle Time
This metric measures the elapsed time from identification of a customer product need until commercial sales commence.
1.2 Project Management Cycle Time
This metric measures the elapsed time from establishment of a discrete project to address an identified customer product need until commercial sales commence.
For both 1.1 and 1.2 above, the end point can be time when manufacturing feasibility is established for those cases where no commercialization occurs. Compare to historical values and benchmark vs. competition, if possible. Group by categories of projects (e.g. major new product, minor product variation, etc.) Can also be used to track milestone attainment rate for firms using a stage gate management process.
2. Advantages and Limitations
Market Cycle Time
The advantages of this measure is that it is quantitative and can be used to measure the entire process or various parts of the process if stage gates are examined. The process can be analyzed to determine what parts are driving the overall cycle time so that improvements to the process can be made.
The limitations of this metric could include R&D’s position that it does not adequately influence the process until after a need has been more defined. Another limitation is that a strong documentation system is helpful to make the cycle time metric as accurate as possible. An additional limitation is that defining the commencement of sales as the end of the cycle does not account for post start-up issues such as efficiency, waste, % of manufactured product within specification, etc.; this could lead to focusing on shortening the cycle time at the expense of later, non-measured parts of the cycle. One must also keep in mind that for breakthrough or paradigm shifting projects,, cycle time measurements. The advantage of this metric is that it supports having the clarity of when a project is initiated based on approvals, assignment of resources, start of spending, etc. The metric can be used to measure the entire process or various parts of the process if stage gates are examined. The process can be analyzed to determine what parts are driving the overall cycle time so that improvements to the process can be made.
3. How to use the metric
Both metrics should preferably be used in combination with a project reporting system that can track the project initiation date (based on approval and assignment of resources), the length of time in each stage gate of the innovation process, and date of sales commencement. For the first metric, the initiation date could be the date the customer need was determined (i.e. marketing request or date of customer research results). Cycle times for different types of projects (new products, cost savings, product improvements, etc.) should be compared to help predict and manage resource allocation. Cycle times for different divisions could also be compared (with caution) to identify practices driving lower cycle time to adapt where possible.
4. Options and Variations
Variations could exist regarding looking at cycle times for only certain parts of the process for which R&D feels it has most control or influence. Cycle time could be extended past the commencement of sales based on what is important to the organization and R&D’s involvement, i.e. when target efficiency is achieved, target manufacturing cost, % of product within spec, etc.
5. References
The first three references state the case for faster R&D response time, while the last three references raise cautions about going too fast:
Burkart, R. E. 1994. Reducing R&D Cycle Time, Research-Technology Management, 37(3), May – June, 27-32.
Patterson, M.L., Accelerating Innovation: Improving the Process of Product Development, New York: Van Nostrand Reinhold, 1993.
Smith, P.G., and Reinertsen, D.G. 1992. Shortening the Product Development Cycle. Research-Technology Management. 35(3), May – June, 44-49.
Curtis, C.C. 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8(3), 18-26.
Von Braun, C.F. 1990. The acceleration trap. Sloan Management Review. 32(1), 49-58.
Ellis, L.W., and Curtis, C.C. 1995. Speedy R&D: How Beneficial? Research-Technology Management, 38(4), July – August, forthcoming. This article calls metric 1.1 idea-to-customer time, and defines other times.
CUSTOMER RATING OF TECHNICAL CAPABILITY
1. Metric Definition
This metric measures the average customer rating (internal or external) of overall technical capability of the firm (interval rating scale) in providing technical service and/or new product innovations. It can be rationed to ratings for relevant competitors for benchmarking purposes.
2. Advantages and Limitations
The advantages of this measure is that is based on customer feedback and is therefore based on what is important to them vs. the area being measured. It also is a good measure to assess overall technical capability vs. some quantitative output or result. As with any subjective, measure, it is less objective and date-based than some other measures. It must also be clarified if the area is being measured for proving technical service OR new product innovations, as the results could be very different.
3. How to use the Metric
A rating system of 1 to 4 can be used for this measure. Regarding technical service, a 1 could be described as an organization having poor technical service (extremely slow response time, long problem-solving time vs. expectation, and poor/failing results); a 4 could describe an organization which provides an immediate or proactive response with extremely short turn-around time, with the problem being prevented into the future, and consistent exceptional proactive service. Regarding new product innovations, a 1 could describe an organization whose products consist of outdated, archaic systems and technology which are difficult to use, maintain, adapt, etc.; a 4 could describe an organization whose products are constantly exceeding customers’ expectations in their rate of introduction, new features, adaptability, cost, anticipating of needs, while clearly standing out from the competition.
4. Options and Variations
Other criteria could be considered for technical capability rather than technical service or new product innovation and a similar scale could be developed. These criteria could include an overall technical assessment of design capability to quickly meet customer needs, development of products which leverage existing core capabilities or competencies (vs. requiring new technologies), ability to create products which have a significant competitive or sustainable advantage, etc.
5. References
Ellis , L. W. and Curtis, C. C. 1995 . Measuring Customer Satisfaction. Research -Technology Management, forthcoming (hopefully in time to identify the issue & date ) .
NUMBER AND QUALITY OF PATENTS
1. Metric Definition
1.1 Percent Useful is a metric which measures the percentage of active patents from the company’s total patent estate which are incorporated into or used to defend the firm’s commercial products or processes.
1.2 Value Ratio is a metric which measures the interval rating (1 to 5) for potential strategic value times rating (1 to 5) for strength of protection divided by 25 (maximum attainable value). It yields a number between 0 and 1.
1.3 Retention Percent is a metric which measures the percent of granted patents maintained.
1.4 Cost of Invention measures the number of patents from R&D/R&D effort costs. One can also calculate this just using the number of useful patents from R&D
2. Advantages and Limitations
1.1 The advantage of this metric is that it allows one to look at the utilization of patents and not just the number generated. By examining the percent useful, the value of the patents is much more obvious. The disadvantage of this metric is that it weights all of the useful patents equally, whereas some could be more valuable than others based on the amount of revenue they protect, etc.
1.2 The advantage of this metric is that it considers the strategic value of a given patent and the degree of protection, vs. just the potential use of the patent. The limitation of the measure is that it is more qualitative and subjective and involves multiplying, which increases the variability around the measure.
An additional limitation is that the calculation across all patents can become rather cumbersome.
1.3. The advantage of this metric is that it looks at patents which are being maintained, likely due to their potential for future utilization vs. just current utilization. A limitation is that it does not include the potential value of maintaining these patents.
1.4 The advantage of this metric is that it allows the user to assess the cost of the invention process to better determine if the cost is reasonable vs. the strategic value. A limitation is that it potentially focuses on a negative vs. the positive outcomes of the patented technologies.
3. How to use the Metric
3.1 This measure can be used to track over time the use of technology’s patents. This can provide signals as to whether or not patent work is being supported, is of adequate usefulness, or if the company’s core and protected technologies are deteriorating.
3.2 This measure can be used to assess the true financial and business value of patents to the organization and should be done by a cross-functional team knowledge about both the business potential and the technology. Depending on the size of the businesses in which the company invests, a 1 for strategic value would represent a business of minimal attractiveness in size (i.e. volume, revenue, or profits) and a 4 would represent a business of maximum attractiveness in size. Depending on the product life cycle for the business, a 1 for strength of protection could mean minimal “tactical” (less than 2 years) competitive advantage or easily replicated; a 4 for strength of protection could mean a transforming technology which would be expected to last 10 years or beyond or where the technology is virtually impossible to replicate by competition or the patent almost impossible to circumvent successfully.
3.3 This measure could be tracked by the legal department and updated as part of an analysis done at a set frequency to assess whether or not certain patents should be maintained. Key decision criteria should be developed i.e. likelihood of future use, strategic value of future use, etc.
3.4 This measure could be used as some set frequency to assess the cost of invention vs. the strategic value so that the organization can track this measure and relationship over time.
4. Options and Variations
4.0 In addition to any of the above metrics, one could also consider measuring the number of times a patent is cited by others. This is assumed to indicate the value it offers to others in developing technology. If of interest only internally, then only references made via internal (vs. external) patents should be counted.
4.2 The interval rating scale for potential strategic value could be based on an even more qualitative assessment such as 1 being of minimal strategic value (in which case one would question why a patent was utilized) up to 4 being of critical strategic value (i.e. new business category, key competitive threat, high growth potential, etc.)
5. References
Acs, Z. J. and Audretsch, D. B. 1989. Patents as a Measure of Innovative Activity, Kyklos, 42 (2)171 – 180.
Chakrabarti, A. K. , and Anyanwu, C. L. 1993. Defense R & D , Technology, and Economic Performance: A Longitudinal Analysis of the U. S. Experience, IEEE Transactions on Engineering Management , 40 (2) , 136 – 145, May. This article has many references on patents as a technology indicator.
Curtis , C. C. 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8 (3) : 18 – 26. This article covers the relationship of patent number and patent quality with other metrics.
Griliches, Z. 1990. Patent Statistics as Economic Indicators : A Survey, Journal of Economic Literature, XXVIII, 1161 – 1707 , December.
Honig – Haftel , Sandra. 1990. The effect of reward systems on the development of patents in high technology firms . Sc. D. diss. , University of New Haven.
SALES PROTECTED BY PROPRIETARY POSITION
1. Metric Definition
Sales of products protected by patents owned by the company, and a result of the R&D effort, is a simple and meaningful research outcomes measure. The patent can cover a composition, process, or use, provided that it clearly creates monopoly or obvious market leadership position for the company. Rigorous review is required to assure that incremental improvement patents are excluded unless clearly leading to market leadership. The following formula best describes the metric;
Sales of products in year x protected by patents
Total business unit sales in year x
Foreign sales and patents may be considered if they represent a significant portion of the total. A broader metric encompassing all proprietary positions, e.g., via licensing, acquisition, joint venture agreement, trade secret, etc., could be used but it would likely be much more subjective and much less credible.
2. Advantages and Limitations
The main advantages of this metric are;
Easy to measure
Very credible if used properly. Patents are clearly a top R&D outcome
Quantitative
The disadvantages are;
Could be a lagging indicator; may measure progress a decade or more old. May be beneficial to also include some measure of remaining years of protection, or list of expiration dates to show how current the protection is.
Some businesses or market segments may place more emphasis on trade secrets or market prowess than on patents because of short product lifetimes or other reasons.
3. How to use the metric
Using the formula shown above, all of the previous year’s sales are examined to determine which are protected by patents. The Patent Department should participate and corroborate the assessment. Only those patents that clearly give a product a major technological and hence market advantage should be included, e.g., a composition patent on a me too performance drug for a minor market share, probably should not be included. International patents and sales should be included if they are significant ( over 15% of total ).
4. Options
A log of patent expiration dates should be reviewed and discussed along with the metric annually to assure that a precipitous drop in the metric is not imminent. Projections could be made 5 or so years out to illustrate continuity of the protection.
5. References
Acs, Z.J. and Audretsch, D.B. 1989. Patents as a Measure of Innovative Activity, Kyklos, 42 (2 ), 171-180.
Chakrabarti, A.K., and Anyanwu, C.L., 1993. Defense R&D, Technology, and Economic Performance: A Longitudinal Analysis of the U.S. Experience, IEEE Transactions on Engineering Management, 40 (2 ), 136-145, May. This article has many references on patents as a technology indicator.
Curtis, C.C., 1994. Nonfinancial Performance Measures in New Product Development. Journal of Cost Management, 8 ( 3 ): 18-26. This article covers the relationship of patent number and patent quality with other metrics.
Griliches, Z., 1990. Patent Statistics as Economic Indicators: A Survey, Journal of Economic Literature, XXVIII, 1161-1707, December.
Honig-Haftel, Sandra, 1990. The Effect of Reward Systems on the Development of Patents in High Technology Firms Sc.D dissertation, University of New Haven.
PEER EVALUATION
1. Metric Definition
External peer evaluation is another method of assessing the strength of a firm’s technology. Alternately, an internal panel of top scientists could be assembled, but this group would probably have less credibility because of their stake in the outcome. The evaluation in either case would involve a comparison of the firm’s technology in a particular area compared to the current state of the known art and particularly against competition, if known.
2. Advantages and Limitations
Advantages of the metric are;
Good diagnostic tool for future growth and progress if unbiased panel of experts can be assembled
Can be very credible if the panel members are credible to the major stakeholders ( suggests external, blue-ribbon panel )
Can lead to remedial action based on panel suggestions
Limitations/Disadvantages
May be difficult to benchmark competitive technology
Subjective, and qualitative
Confidentiality problems with outside panels
Some stakeholders distrust all technical people
3. How to use the metric
Selection of the panel is extremely critical. People must be selected with credibility to the stakeholders as the predominant criteria. Technical competence in the field is essential, and the panelist must be impartial. Internal panels generally will be less believable to top management because of the stake they have in the R&D organizations success. An external panel could include outside directors, local university faculty and department chairmen, consultants, members of technical or trade organizations, etc.
A numerical rating scale should be used, e.g., 5=well above art or competition, 3=equal to , and 1= well below.
Panelists should vote independently ( perhaps secretly ) after extensive discussion with R&D participants and by themselves.
4. Options
The use of internal panels probably would have little credibility outside the technical community. However, it could be an excellent reality check within an R&D organization.
The best, and most credible results, might be obtained via paid, technical consultant teams, e.g., A.D.Little, SRI, etc. These organizations would have no incentive to benchmark low. They could also conduct blind surveys to determine competitive positions.
CUSTOMER SATISFACTION
1.0 Metric Definition
The customer satisfaction metric has two variations:
1.1 Measures of external (end-customer) satisfaction. These may be such metrics as ratings of quality of technical personnel or technical capabilities, or technology benefits within products or processes.
1.2 Internal customer satisfaction. Since the immediate customer of R&D is normally the businesses within the corporation that R&D serves, measures such as customer satisfaction in engineering, marketing, or manufacturing may be appropriate. Typical metrics might include on-time technology delivery, competitiveness or appropriateness of the technology solutions delivered, and overall satisfaction with the R&D track record of technological support.
2.0 Advantages and Limitations
There are few disadvantages to good customer satisfaction metrics. In the case of external, or end-customer satisfaction metrics, one complication may be that the entire innovation cycle is under review by the end customer. A bad grade by the customer, while a valid rating of the corporate innovation process, may not be merely an indictment of the R&D operation, but a judgment of the overall product development process within the company, involving manufacturing and product engineering, market forecasts, consumer needs and attitudes, and competence of corporate management.
One the other hand, a well-thought-out customer satisfaction metric (or set of metrics) for the internal or immediate customer within the corporation — normally the corporate businesses and their various R&D-related organizations — may be the key diagnostic to indicate that R&D processes are lacking and need adjustment or redesign. The R&D organization itself is probably better served with well thought-out internal metrics than with external metrics that complicate the diagnostic process for R&D when problems are indicated.
3.0 How to Use the Metric
3.1 External customer satisfaction metrics. These metrics will normally be marketing-related or implemented. One variation would be to use a marketing survey, in which various aspects of technology benefits are rated on a five -point scale
3.2 There are two dimensions for internal customer satisfaction metrics: strategic and tactical.
3.2.1 Strategic metrics deal specifically with whether the R&D function is meeting the strategic needs of the customer. The review process might involve matching technology and product roadmaps in a joint meeting, in which technology timing mismatches are resolved. Information can be exchanged; technology previews by R&D to alert the businesses to possible market-creating or market share increasing discontinuities, and the businesses to share future market window and product definitions with R&D. Various metrics can be used, including the five-point rating system mentioned above, or a metric which highlights number or percent of mismatches between product and technology roadmaps.
3.2.2 Tactical metrics deal with whether specific projects are meeting the goal or delivery requirements of the internal customer base. For technology projects in the latter stages of development (near to or entering product development), regular project reviews with the intended customer(s) is important. At quarterly or perhaps semiannual reviews, customer and R&D representatives join in a review of project progress. A useful metric in this case is a report card which each customer representative is required to fill out in the review meeting. This is a very simple questionnaire which has 3-5 survey questions on project progress, and suitability of both the project and the technological approach to satisfy the customer(s) needs. Typical questions might be:
Does this project meet your product technology needs?
Do project milestone dates meet your market window?
Have any strategy changes on your part not been addressed?
What is the overall project score (typical scale 1-5)?
As a related metric, trend analyses can be made both by project and organizationally.
4.0 Options and Variations
One variation on the customer satisfaction process is to establish steering teams to address technology, business and market issues and provide guidance to the R&D organization on strategic issues. For external issues, the teams might consist of focus groups that meet periodically, or focus teams that convene groups which are diverse either geographically, ethnically, or with respect to age group, for example. The focus might be on functional needs that technology capabilities address. The metric would be meetings held, or issues addressed and settled. Another metric might be problems surfaced to be addressed and reported on by the R&D team.
Internal issues might be addressed by customer teams composed of manufacturing, engineering, marketing, financial, and related personnel (including even external consultants) who provide guidance and assist in forming cross-functional project teams. Appropriate metrics are meetings held, issues settled, or problems surfaced. An ongoing metric can also keep track of the % of problems addressed and resolved versus those surfaced.
DEVELOPMENT PIPELINE MILESTONES ACHIEVED
1.0 Metric Definition
Development Pipeline Milestones Achieved is a metric which is useful in grading the effectiveness of management and planning of each R&D project. It may also be used as an indicator of performance problems on a given project which can be used to initiate diagnostic and recovery procedures to give each project the best possible chance of success. There are two possible variations of the metric which may be used:
1.1 Percent of project milestones achieved — the percent, by project or overall by sub-organization or laboratory, of all project milestones completed on schedule or within some acceptable time window (90 days, for example) of the forecast date. Trend studies can then be established for organizational performance based on analysis of the data collected by quarter or for whatever other time period is appropriate for the particular industry group.
1.2 Performance level at each milestone — on a project basis, percent of all expected objectives met at the milestone date by which they are forecast to be completed.
2.0 Advantages and Limitations
The primary advantages of this metric are for project diagnostics and as an indicator of the overall planning and management health of the organization. There are several possible limitations. First, a consistent management system must be in place to assure that the variability of the number/quality/difficulty of the milestones does not cause random fluctuations in the metric. Secondly, the calibration of the metric is important. Since both the stage of the R&D project and the industry of the business involved can greatly affect the trend analyses, the metric user must be careful to identify what the real danger signals are in terms of management and planning deficiencies, and what really constitutes an indicator of project problems for this metric to be useful.
3.0 How to Use the Metric
1.1 Percent of project milestones achieved — a simple approach is to tabulate milestones met on schedule, 1-90 days late (which may constitute on schedule in some cases), 90-180 days late, and so forth. The tabulation can be done quarterly, and the trend information publicized to the organization to focus attention on performance issues.
1.2 Performance level at each milestone — This is more useful as a project diagnostic. One approach is to require project managers to use this metric to summarize performance quarterly or semiannually at an operations review and be prepared to cover diagnostic or recovery steps if a given project shows trends over two or more time periods of failing to meet an acceptable performance level (which will depend on the R&D discipline, the industry, and the stage of the innovation process).
4.0 Options and Variations
One problem with using milestone achievement as a key metric is that pressure is exerted to make all the milestones. With strong pressure to meet milestone dates, the project manager may be tempted to populate the project plan with easy milestones, to assure that his milestone hit rate is good. It is important therefore to make sure that milestones are realistic and aggressive. Especially in the later stages of the innovation cycle, in product design and productization or manufacturing process development, it is important to assure that the business customer, marketing representative, etc. approve the project milestones to assure that the product development cycle meets the market window requirements of the business. In this case, a much closer tracking of milestone achievement may be necessary than in the earlier stages of the innovation cycle (for example, monthly or in some cases even more frequently).
Another problem with using the milestone approach is that early in the innovation cycle (the R part of R&D), milestones are often hard to define and even harder to forecast. With the heavy level of uncertainty in a long-term research project, the definition of a milestone may be vague, or even made in terms such as define or develop concept. In these circumstances, project managers feel especially uneasy in setting milestone dates for which they become accountable. An approach here is to realize that (1) exact dates are less important, since product or market need windows probably have not been established (and in some cases, market needs or even product existence!), and (2) many of these projects which are early in the innovation cycle will certainly fail, since in the research phase, many more ideas are explored than have a positive outcome. In this context, using milestones as a performance metric may not be useful. If used, the metric should probably be relaxed in some way. For example, instead of setting a goal of completing 100% of milestones, perhaps a goal of 50, 60, or 80% may be set. Another possibility is to declare any milestone made within 90 or 120 days of the forecast date to be on time. The specific approach taken will depend on the industry, the research area, and the stage in the innovation cycle.
CUSTOMER CONTACT TIME
1.0 Metric Definition
A major concern in corporate America is that R&D output be relevant to the businesses which it supports. There are several metrics which can be enforced to assure that R&D is satisfying its immediate customers in the corporation. These include Strategic Alignment, Business Funding of R&D Projects, Customer Satisfaction, and Customer Contact Time. This metric is normally a simple count of time (hours or fractional days) spent with the customer. The time may be counted for R&D management and scientists separately, as well as in a total for all customer engagement time. Note that the term customer can refer to both internal customers (representatives of the businesses that R&D supports, either engineering or perhaps marketing) as well as external customers (the end customers of the business of the company).
2.0 Advantages and Limitations
The amount of time that scientists on research staff spend with customers appears to correlate well with successful innovation in many companies. Probably the more normal scenario for contact will be with internal customers. With strategic planning maturing in many corporations, businesses today are able to share relatively accurate long-term plans with R&D, which can give the R&D organization an insight into competitive requirements in the future. Correspondingly, R&D can highlight differentiating technologies which can provide future competitive advantage. A metric which enforces the regular association of R&D with internal customers ensures that this regular communication occurs. A metric which counts contact time with external or end customers may be especially desirable in certain industries.
A limitation is that simply counting contact time provides only a quantitative measure of interaction time. It does not ensure that the time spent with customers produces the desired result of coupling business and market needs into technological planning and alignment efforts.
3.0 How to Use the Metric
A simple method is to count hours spent with customers in conferences or meetings. Most contact metrics will probably be with respect to the internal customer, since R&D will normally rely on business and marketing sources to translate the end-user needs into technology or functional requirements. There may be a separate count for managers spending time with counterparts in the businesses or in marketing, as well as for scientists. In some organizations, the process of converting end-customer needs to product requirements may impose special end-customer contact requirements on R&D. In this case, there might be metrics for both end-customer and immediate-customer (I. e., business) contact time.
4.0 Options and Variations
An alternative is to count functions as the contact metric. For example, the R&D organization may have a steering committee (or several technology- or business-related steering committees) which meet regularly; the regularity and frequency of these meetings may be a key strategic metric. Also, R&D organizations may have operations reviews with selected customers regularly to discuss progress on key projects. Counting these meetings may be an acceptable way to judge customer contacts in a class of projects which are entering the mature (near development) stage.
PRESERVATION OF TECHNICAL OUTPUT
1.0 Metric Definition
The product of research is information. In the latter part of the innovation cycle (the D of R&D, also known as engineering), preservation of the knowledge gained is relatively easy, since it is embodied in the production drawings, manuals, and source code related to the products and processes of a business. Earlier in the cycle, in the research process, the preservation of information is sometimes less orderly and institutionalized. The purpose of the Preservation of Technical Output metric is to ensure the documentation of information gained in the research portion of the innovation process.
2.0 Advantages and Limitations
The advantage to using a preservation metric (and associated information process) is that important information is preserved. This is normally accomplished via a documentation requirement associated with each research project (details discussed below). It is especially important to establish a reporting process, using the metric to enforce process discipline, in the long-term research area, since often knowledge gained in such efforts does not have an immediately apparent use. Documenting and preserving the knowledge can be extremely important so that retrieval at a later (possibly much later) date, in an entirely different context, is guaranteed.
The major potential limitation or disadvantage is that the value of measuring the number and/or quality of research reports for research organizations has not correlated well with financial success or value creation of the R&D organization or its associated businesses. As utilized in many R&D organizations, the practices of counting research reports, publications, conference papers, etc. has had debatable value to industry.
3.0 How to Use the Metric
Preservation of Technical Output is a report metric. In many companies, it simply consists of counting the number of research reports per organization, per project, or by technical competency area. This metric can also be expressed as the % of key outcomes of projects that are captured in reports. A related metric is to require a certain number of reports for each research project, either timed (by quarter, semiannually or annually, etc.), or related to certain specific project milestones which are especially significant. Some organizations feel that a more important metric is report requests. In this view, corporate research is for the benefit of internal customers — the businesses of a company that R&D supports with technology inputs. Since report requests indicate interest on the part of the business units, requests for reports show the level of interest in each technology by the businesses which represent the R&D customer base.
The fact remains that reporting metrics do not correlate well with successful innovation (one study showed a counter correlation). Since the preservation of technical information is important, possibly the simplest approach is to require that each project document results (for example, an annual or semiannual report); the metric is simply a check-mark that the documentation is accomplished in an acceptable form.
4.0 Options and Variations
To reduce the reporting overhead, a simplification is to allow any major publication or conference report which covers the reporting requirement to also satisfy the reporting metric. The report request metric can be dealt with more easily if reports are distributed in electronic form (or at least with electronic abstracts) and requests are limited to electronic requests, which can be easily tabulated.
EFFICIENCY OF INTERNAL TECHNICAL PROCESSES
1. Metric Definition
This set of metrics seeks to provide a measure of both the efficiency and effectiveness of the operation of R&D processes within the firm.
1.1 Project Assessment
1.11 The total cost of all commercially successful projects divided by the number of commercially successful projects. (Useful when tracked over time with similar projects with similar scopes)
1.12 The ratio of actual to projected costs (and timing) for all projects.
1.13 Percentage of costs devoted to commercially successful projects.
1.2 Portfolio Assessment
1.21 The total R&D budget divided by the number of projects with commercial output. Subdivide by projects of similar type (technical service, short term, long term) and used in conjunction with project value assessment.
1.22 Use the various approaches to Portfolio Assessment from Third Generation R&D (see ref. 1)
2. Advantages and Limitations
This set of metrics needs to be adapted to the needs of each firm — considering the goals, objectives, and priorities for the firm. Assessments need to be made for individual projects (stage-gate, PACE, or similar processes), and for the collection of projects (Portfolios).
3. How to use the metric
4. Options and Variations
Each firm will need to set the metrics relative to its specific goals and objectives. An assessment of the selection termination and management of projects can be made using the following four stages: (ref. 7)
Technical projects: Selection, termination and project management:
Level1.
a) favors short term projects
b) politically driven selection
c) no project monitoring or pre-project planning
d) little inter-functional participation in project teams
e) erratic turnover of team staffing
f) project leader roles not defined
g) no training for project leaders
h) unclear charters for project teams
Level 2.
a) mix of short and medium-term projects
b) no inter-product-line analysis
c) priorities set erratically
d) project tracking
e) some inter-functional participation but not all key functions represented.
f) formal release process for new products
g) some project team stability but conflicts over work priorities
h) project leaders given only minimum guidance or training
Level 3.
a) selection based on multiple inputs from internal and external sources
b) balance of short-, medium- and long-term projects
c) risk analysis incorporated at key phases
d) projects still schedule driven
e) inter-functional teams wherever needed
f) clear allocation of project and functional responsibilities
g) training for project leaders
Level 4.
a) clear links between selection criteria and business and product-line strategy
b) disciplined process for project termination
c) cross-functional planning and execution
d) continual improvement- postmortens, quality measures of both project process and product performance
e) projects are milestone driven
f) differentiated project management procedures for different types of projects
g) scheduling and capacity planning avoid resource contention by competing projects
5. References
Portfolio Analysis: P.A. Roussel, K.N. Saad, and T.J. Erickson,Third Generation R&D, Harvard Business School Press, 1991
M.E. McGrath, M.T. Anthony, and A.R. Shapiro, Product Development; Success Through Product and Cycle-time Excellence, Butterworth-Heinemann, Newton MA, 1992
Assessment of work processes: G.A. Rummler and A.P. Brache, Improving Performance: How to Manage the White Space on the Organization Chart, Jossey-Bass, San Francisco 1990
C.C. Curtis, “Nonfinancial Performance Measures in New Product Development, Journal of Cost Management, 8(3):18- 16, 1994. Addresses the distribution of the portfolio in terms of major projects, minor projects and extensions.
R.N. Foster, L.H. Linden, R.I. Whiteley and A. Kantrow, “Improving the Return on Research and Development”, Research Management, 28(1): 12-17, and 28(2): 13-22, 1985.
L.W. Steele, “Selecting R&D Programs and Objectives”, Research-Technology Management, 31(2) March-April, 1988 17-36
P.S. Adler, D. William McDonald, F. MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37
EMPLOYEE MORALE
1. Metric Definition
This metric takes quantitative ratings of key aspects of employee satisfaction and morale as shown by direct employee survey. It is recognized that employees may feel good and have high morale, yet produce nothing of value for the firm — the real question is are they motivated and committed to create and innovate profitably?
2. Advantages and Limitations Extensive surveys are time consuming and expensive to conduct and must be conducted with sufficient frequency to establish base lines and understand real trends. One must understand also that technical populations tend to have certain biases in such surveys. Many employees feel “surveyed-out”.
3. How to use this metric The typical survey uses five point scales for agreement (strongly agree, agree, neither agree or disagree, disagree, strongly disagree) for importance (extremely important, very important, somewhat important, of little importance, not at all important) and for performance (very good, good, fair, poor, very poor) in answering sets of questions related to work environment, feelings about the company, ratings of the company, ratings of the organization/work location/work group, feelings about the individual’s job, and general satisfaction. Opportunities are given for comments. Such extensive surveys are most often conducted by third parties to maintain confidentiality.
Four tested questions are job satisfaction are:
3.1 If a good friend was interested in a job like yours for your firm, what would you tell the friend?
3.2 All in all, how satisfied are you with your present job?
3.3 Knowing what you know now, if you had to decide all over again, would you take your current job?
3.4 How satisfied are you with the overall employee- employer relations at your firm?
4. Options and Variations
One firm which uses e-mail extensively, every 9-14 months conducts a broad “pulse” survey and asks employees to provide two ratings (using 0-10 point scales with descriptors) one rating the employees work climate/environment and the other rating personal feelings about the work itself. Employees spend less than 4 minutes to reply by e-mail, or to be anonymous by fax or to a voice mailbox. Employees often write additional comments that give information sought in the more extensive surveys. Confidentiality is assured and rapid feedback (within 2 weeks) of survey results to participants maintains a high level of interest and participation.
5. References.
6.1 C.J. Cranny, P. Cain-Smith, and E.F. Stone, Job Satisfaction, Lexington Books, New York, 1992
6.2 R. Katz (ed), Managing Professionals in Innovative Organizations, Harper Collins, New York, 1988
GOAL CLARITY
1. Metric Definition
This metric uses an interval rating scale assessing the extent to which project performance objectives are clearly defined and understood by all participants on the project team.
2. Advantages and Limitations
While this metric may provide a semi-quantitative assessment there is a degree of subjectivity for an individual to assess the extent to which he/she really understands objectives and roles on the team.
3. How to use the metric
A member of the team would provide to all members of the team a survey form to assess understanding of the project objectives and commitments. This can be done using some interval rating scheme. It would be wise to conduct this survey several times through the life of the project to really assess the level of understanding.
4. Options and Variations
A four stage assessment can be constructed with the following as the highest stage for individual team members:
a) clear understanding of the expected product of team effort
b) personal belief that a team is the right way to develop/achieve the expected product
c) understanding whether the team is an implementation, recommendation and/or informational team
d) understanding the team’s operational ground rules and end-result boundaries
e) belief that the team has all the appropriate knowledge, functions, diversity, levels and locations, so that the expected product of the team effort can be achieved using the minimum number of people?
f) high personal commitment to achieving the objectives
g) clear understanding of personal role
h) acceptance of the recognized team leader
i) capability to draw on additional resources to keep the core team to a minimum
5. References
J.R. Katzenbach and D.K. Smith,The Wisdom of Teams, Harvard Business School Press, Boston 1993
IRI Quality Director’s Network study on self directed teams (in progress)
H.P. Dachler, and W.H. Mobley “Construct Validity of an Instrumentality-Expectancy-Task-Goal Model of Work Motivation”, Journal of Applied Psychology, 58. 397-418, 1978
PROJECT OWNERSHIP/EMPOWERMENT
1. Metric Definition
Empowerment involves management endowing the project team with the authority to make decisions on the project and to carry them out independently without constantly having to seek management permission or approval.
Project Ownership encompasses a variety of feelings and beliefs of the project team that they:
believe in the project and its goals
are committed to the project and want it to be successful
will share in the credit and reward if the project is successful
have authority and accountability for making decisions and carrying out the project
This metric has been reported to correlate positively with successful innovation from R&D. It is also likely related to employee morale on a project team.
Level 1
Projects are defined by the businesses and R&D management without the involvement of project leaders and technical people
Project team members are working on projects because they were assigned to them. They may or may not believe in the goals or be committed to achieve them* Decisions are made by R&D management and business managers; project leaders and team members are not consulted
Level 2
Some EMPOWERMENT and project ownership, but not extensive
Level 3
Extensive empowerment and project ownership, but not uniform throughout the organization
Level 4
Project leaders and technical people are involved from the beginning in defining all projects and their goals
All project teams have a high degree of belief in and commitment to their projects
Project leaders and team members have authority and accountability for making decisions and carrying out their projects; they review their decisions and progress with management and business partners on a regular basis * Project leader has financial authority and accountability for the project
2. Advantages and Limitations
This metric is relatively easy to assess. It requires an honest and objective assessment of practices in defining and carrying out projects, but it does not require a large amount of data collection, history or benchmarking.
3. How to Use the Metric
It is recommended that input be obtained from both managers and project teams to compare their perceptions. This metric would lend itself to measurement in an employee opinion survey. The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.
4. Options and Variations
In general, Empowerment is a necessary factor which precedes and contributes to Project Ownership. They might be split out into separate metrics.
5. References
Weisbord, M.R. 1990. Productive Workplaces: Organizing and Managing for Dignity, Meaning, and Community, Jossey-Bass, 121-141.
MANAGEMENT SUPPORT
1. Metric Definition
Management Support is critical to successful innovation. It is in essence showing the project teams that they are respected and trusted and providing for them all the tools they need to carry out the projects successfully.
Management Support is manifested in many ways, including:
Empowerment of project leaders and project teams
Providing appropriate input, guidance and direction on larger issues and allowing the team to attend to the details
Providing resources for the project, including people, equipment, consultants, attending technical conferences, customer contacts, etc.
Personal attention, such as attending project meetings and visiting labs
Verbal affirmation of the teams, their approaches, and their progress
Peer recognition, company and departmental awards
Defense and support of the teams in response to questions and criticism
Allowing project teams to fail without negative consequences
Low ratings in this area may point to a breakdown in relations and credibility between R&D management and the project team and, perhaps, between R&D management and business management.
Level 1
Scenario A – Micromanaging* Managers generally do not trust technical people to carry out their projects and are very critical of technical people
Managers define and direct the projects
Managers are running the show, in meetings and in the labs
Technical people are punished for project failure in performance appraisals
Scenario B – Neglect
Managers pay no attention to the projects and project teams
Managers trust the technical people implicitly and leave them to themselves
Managers rarely attend project meetings or visit the labs
Projects are understaffed
Project teams are making do with inadequate equipment and resources
Technical people are rarely rewarded in peer recognition or company awardsLevel 2
Some management support, but not extensive
Level 3
Extensive management support, but not meeting all the goals and/or not uniform throughout the organization
Level 4
Managers respect technical people and trust them to carry out their projects
Managers participate with technical people in defining projects
Managers contribute ideas, but give broad guidance and direction in carrying out the projects
Managers regularly attend project review meetings and frequently visit the labs
Project teams are given the resources they need to successfully complete their projects
Management acts as an advocate for the project team, facilitating interactions with other functions
Technical people are regularly recognized and rewarded in peer recognition and company award programs
Project failures are acknowledged, analyzed constructively, and used as a learning tool to see what could have been done better, but team members are not punished
2. Advantages and Limitations
This metric might be perceived differently by managers and employees. Although it might be evaluated simply by an honest and objective assessment of management practices, resource allocation, relationships between managers and technical people, and how project successes and failures are dealt with, it would be more meaningful to get input from project teams as well. Assessment will require little history or benchmarking.
3. How to Use the Metric
It is recommended that input be obtained from both managers and project teams to compare their perceptions. This metric would lend itself to measurement in an employee opinion survey.
The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.
4. Options and Variations
There are several aspects to Management Support, such as empowerment, resources, and recognition. They may be measured altogether or split out into separate metrics.
5. References
Farris, G.F. and Ellis, L.W. 1990. Managing Major Change in R&D, Research-Technology Management, 33 (1), Jan-Feb.
PROJECT CHAMPIONSHIP
1. Metric Definition
An effective Project Champion is a person who believes deeply, but objectively, in the project and the need for its pursuit to commercialization. This is a person who takes ownership of the project and has the authority and resources to ensure that:
a. adequate market information is gathered up-front to justify the project and define appropriate goals
b. the project team is supported and motivated
c. if the project is successful in meeting the goals, the resulting product will be taken to market.
Ideally there should be a champion associated with the sponsoring business, such as a division manager, business manager, or product manager, as well as a champion in R&D. The Project Champion can lend an air of enthusiasm, optimism, and urgency that tends to permeate the project team.
Level 1
There are no effective champions for projects* R&D people are responsible for gathering market information and setting project goals
Business people are not involved throughout the project* R&D people usually are responsible for field trials and taking products to market
Level 2
The champion for most projects is within R&D
Some business participation in projects, but not extensive
Level 3
Extensive business participation in projects. Some business champions of projects. * Not uniform throughout the organization
Level 4
All projects have an effective champion who is associated with the sponsoring business unit as well as a champion in R&D
Business people actively participate in obtaining upfront information to define and justify the project and establish its goals* Business people support and help motivate the project teams* Business people take a leadership role in arranging field trials and commercializing the product
2. Advantages and Limitations
It is critical that the champion believe strongly in the project, however, the champion also must be analytical in assessing the project and its progress, so as not to drive the project beyond when it appropriately should be terminated.This metric is relatively easy to assess. It requires an honest and objective assessment of the role of business partners in projects, but it does not require a large amount of data collection, history or benchmarking.
3. How to Use the Metric
The metric is an overall assessment of practices throughout the organization that is being measured. If practices are different in different units or projects, the metric will be an average. Levels 1 and 4 are relatively straightforward. Levels 2 and 3 are interpolations between 1 and 4.
4. Options and Variations
Business involvement typically varies throughout the stages of a project, such as project definition, technology development, product development, and product commercialization. This metric might spawn additional metrics for the different stages of each project.
5. References
Farris, G.F. and Ellis, L.W. 1990. Managing Major Change in R&D, Research – Technology Management, 33 (1), Jan-Feb.
Kanter, R.M. 1983. The Change Masters. New York: Simon and Shuster.
INFORMATION TECHNOLOGY USE IN R&D
1. Metric Definition
This set of metrics measures the extent of use and the ways in which information technology is used within R&D.
1.1 Extent of IT Use in expenditures in the R&D budget.
IT expenditures are a measure of the enhancement of R&D staff effectiveness by the use of information technology. The thrust of the metric is also how to justify this IT spending to higher management, and how to determine an optimum amount. This may be a non-linear effect – while too little use may not match the capability of your competition, too much use may involve R&D staff in activities less optimal than the use of their time directly on R&D. The form of the metric is usually to measure the amount, or ratio of the total R&D expenses of the firm, spent on information technology hardware and software.
1.2 Ways in which IT is Employed – Impact of IT on R&D
The intent of this metric is to measure how much impact information technology (IT) has on the innovation chain of your company. It is usually measured on a scale of 1 to 4. Level 1 is when IT is only used to keep track of staff times and costs of R&D activities. In level 2, IT is also used to provide tools for technical computing and/or hardware and software for the company’s products and services. In level 3, IT also enhances the effectiveness of innovation management by substituting for human effort in R&D. Level 4 IT also lets you rethink how R&D is done by enabling doing things that can not be done any other way. Where unit scores vary considerably from one unit to another, managers should consider using a median score from all the sub-units.
1.3 Extent of IT use in Managing R&D
The intent of this metric is to measure how far along you are at using IT in managing the innovation process. Level 1 is when IT is only used by R&D groups on an individual basis. In level 2, IT is integrated between the IT function and a few groups in the R&D activity. In level 3, IT is fully integrated in the process of managing technology within the R&D department. Level 4 is when IT is fully integrated in the process of managing technology with the all department managers from marketing through R&D through manufacturing/operations.
1.3 1.4 Other possible metrics that might be developed
Breadth of usage by application category vs. benchmarks; Quality of usage by application category vs. benchmarks; Percent of R&D staff using IT at benchmark levels.
2. Advantages and Limitations
The advantage of this metric is that it enables the justification of IT in R&D both at financial budget time, and in benchmarking IT use and impact in R&D against competition and strategic goals. Thus, the stakeholders most interested would be financial officers and managers, and strategic planners.
One limitation is that these are relatively untried metrics, with minimal research support as to their effectiveness. Thus, it is not yet clear how many R&D groups track IT costs. Another limitation is that it may cost more to follow than the benefit derived from it. Information technology is evolving and developing so rapidly that its dimensions are not fully appreciated and some of the measures may be difficult to make.
3. How to use the Metric
The amount of IT expenditure in R&D should be tracked over time, usually the annual budget cycle. The ratings might need to be done with input from IT professionals. The rating can be compared to other measures of R&D effectiveness to see if the change in IT expenses has enhanced effectiveness. If information is available on competitors, this too can be tracked and used to justify future IT budgets.
The ratings of extent of impact and use should also be tracked over a time period for comparison with competition and strategic needs.
4. Options and Variations
Service companies and companies with a high IT content in their operations may wish to use alternatively any of the software effectiveness metrics such as the Capability Maturity Model developed by the Software Engineering Institute at Carnegie-Mellon.
5. References
There is an IT project of the R-o-R Committee started in 1995. This is a new area with little published references. The most general reference is U. S. National Research Council, Information Technology in the Service Sector, Washington, DC: National Academy Press, 1994.
The SEI Capability Maturity Model is reviewed, and compared with Total Quality Management (TQM) and other evaluation techniques, in H. Saiedian & R. Kuzara, “SEI Capability Maturity Model’s Impact on Contractors,” Computer (IEEE), 28(1), Jan., 1995, pp. 16-26. This source has 12 additional references.
GATE EFFECTIVENESS
1. Metric Definition
In a stage-gate or milestone R&D management process, each project is subject to a design review at periodic stage gates or milestones.
1.1 Quantitative Yield
This metric is designed to measure how effective the design review process is by measuring in terms of successful yield at each gate or milestone from project initiation to project commercialization. The assumption is that the higher the yield, the more effective is the stage-gate or milestone design review process. However, a very high number on this metric indicates that the gate is not working very effectively, i.e., there is no need for a filtering process.
The metric is most accurately used by the percent of the value of R&D projects passing each gate or milestone which meet the criteria for passing the next gate or milestone. The simpler metric of percent of the number of projects may also be used, but runs the risk that it might be excessively influenced by small, easy to understand projects and misrepresent the true picture.
1.2 Income Contributors
This metric is the percent that pass through the gate that become significant income contributors. This is a more retrospective number than metric 1.2, but better reflects what the stage-gate process intends.
1.3 Costly Failures
Another metric is to document the reasons costly failures passed through the gates and were not screened out. This would identify retrospectively what criteria were missing.
2. Advantages and Limitations
The advantage of these metrics is to calibrate the stage- gate, milestone, and design review process, thus enabling diagnosis of the R&D management process. While metric 1.1 is a somewhat retrospective metric, the delay time is only one gate or milestone period long, and thus it is a more prompt feedback metric than overall success or failure in the market place indicated by metrics 1.2 and 1.3. Too low a yield at any gate or milestone indicates that too many projects are getting through for further expenditure which do not deserve to have been supported since they did not pass the subsequent gate or milestone. The limitation is that this is a subjective metric, and needs some historical data within the firm on which to base a benchmark percentage.
3. How to use the Metric
The only way to use this metric is to keep records for a period of time, and use them to evaluate trends. This is somewhat of an administrative burden, justified by its use in continuous improvement of the R&D management process.
4. Options and Variations
For the final gate or milestone before commercialization, metric 1.1 and 1.2 become the same as Metric 28: Percent or value of R&D which is commercially successful.
5. References
Van Remoortere, F., and Cotterman, R. 1993. Project Tracking System Serves as Research Management Tool, ResearchTechnology Management, 36 (2), March-April, 32- 37. This reference covers milestones in general.
Ellis, L. W. 1984. The Financial Side of Industrial Research Management. New York: Wiley. This reference, on page 105, covers the histogram approach to measuring milestone, gate and project completion.
Patterson, M. L., Accelerating Innovation: Improving the Process of Product Development, New York: Van Nostrand Reinhold, 1993. This reference introduces the measurement at each originally forecast milestone date of the percent actually completed of the originally estimated work.
NUMBER OF DEFECTS REPORTED
1. Metric Definition
This metric is intended to measure the lack of quality in the finished product or service. As such it is a retrospective metric, summing up all things that did not go right in the R&D process. This metric may take any one of the following forms: Number of defects reported at any stage by downstream operations; “bugs” found in computer programs; defects as shown by the number of change orders issued to manufacturing or operations; number of end customer complaints.
2. Advantages and Limitations
The advantage of this metric is that it tabulates the ultimate cause of customer dissatisfaction. It is important for diagnosis to identify which defects were caused by design in R&D, and which were caused by downstream operations and not attributable to R&D such as faulty scale-up, manufacturing flaws, packaging problems, etc. This has the advantage over metric 4 because it records the “bugs” identified along the technology transfer process.
Its primary limitation is that it comes rather late, after the internal or external customer has found the problem that was missed in the R&D process. Just as for any inspection system of quality, it is better to do it right the first time, because quality cannot be inspected in after the project has left R&D.
3. How to use the Metric
Like other quality metrics, this is best kept as a running average over a time period – short enough to provide as prompt a feedback as it can, but sufficiently long to smooth out the ups and downs.
4. Options and Variations.
See also Product Quality & Reliability metric : .
5. References
Lutz, Robert A. 1994. Implementing Technological Change with Cross-Functional Teams, ResearchTechnology Management, 37(2), March-April, 14-18.
CORE TECHNICAL COMPETENCY
1. Metric Definition
The understanding, definition, and maintenance of strategic Core Technical Competencies is a key to both current and future R&D performance. The strength of Core Technical Competencies may be characterized using the categories described in the Third Generation R&D Tutorial.
1.1. Scoring of Core Technical Competencies
The metric is defined by first establishing the core competencies of the the firm in the business segment(s) of interest and then rating the specific competitive position in each Core Technical Competency: Score Level Definition 5 Dominant Sets the pace and direction of technological development and recognized for such in industry to express independent technical actions and set new directions3FavorableAble to sustain technological competitiveness in general and/or leadership in technical niches Unable to set independent course. Continually in catch-up mode1WeakUnable to sustain quality of technical output versus competitors. Short-term firefighting.
1.2. Aggregate Rating of Core Technical Competencies.
To obtain an aggregate for the firm or business segment(s), these scores may be averaged over all the competencies defined.
2. Advantages and Limitations
The first advantage of such a metric is that it is considered at all. Use of the metric forces agreement within the firm on Core Technical Competencies and the competitive position in each. Understanding and acceptance can drive programs to maintain competencies or to remedy deficiencies. The process of applying the metric provides a healthy dialogue among the commercial and technical functions.
3. How to use the Metric
To rate the position of the firm with respect to each Core Technical Competency and to develop an aggregate rating of the firm over several competencies, it is first necessary to discern those technical competencies which are essential or most important to the success of a firm in selected markets or arenas of business. The prime references on core competencies from HBR and other current references in the field are recommended. There must be agreement between technology and commercial functions on these Core Technical Competencies, whether or not the firm has an advantaged position in any of them.
4. Options & Variations
For some industries and competencies, it may be possible to define an objective metric having to do with competitive technical performance, such as lead time relative to competitors on major innovations; or position along the curve of improving product performance relative to competitors ( e.g. 20% better).
5. References
Arthur D. Little, Inc., Third Generation R&D, IRI Tutorial, April 29, 1994.
Prahalad, C. K. and Gary Hamel, Core Competence of the Corporation, Harvard Business Review, May-June 1990.
DELAYED STAGE KILLS
1. Metric Definition
This metric is central to the basic notions of R&D yield, productivity, effectiveness and risk/performance abilities. It is also multi-functional in character.
This metric is defined as the failure of projects to move beyond a given stage; particularly in reference to the later stages of the product/process development activity. Since the purpose of R&D is to produce knowledge that reduces risk, then it follows that later stages should be represented by projects that have been risk-reduced. If this is so, there should be more successes and fewer failures at later stages. Conformance to this goal is captured by this common metric as expressed in its negative form.
The most frequently used form of this metric is the number of projects that are terminated either in commercial introduction or in the last phase of development.
However, this can also be generalized by examining the actual failures (or rejections) at a given stage and comparing this to a predicted level. An example might be that at a certain stage you expect that only 10% of the projects should fail, but you observe that 30% are failing at this point. This indicates a later stage kill ratio that is three times higher than expected. There is debate as to whether this metric should be measured only at the stage in question or should be made on a cumulative basis that integrates all future failures that occur beyond a particular stage. This may be more appropriate to some organizations than to others. The important concept is to establish a useful measure of a result to be avoided, i.e. making decisions to terminate later than they could have been made. Overall, these metrics are critical in describing the kind of risk can be attempted and executed well for a given R&D portfolio.
2. Advantages and Limitations
The advantages of these late(r) stage kill metrics is the focus they provide on the cost of ineffective processes. No one wants to spend time and money in significant amounts at later stages just to see it all thrown away because of an unanticipated failure. This is particularly true if it the cause of the failure should have been uncovered by the R&D process at an earlier stage.
These metrics capture the degree to which R&D is truly its risk reduction role for the corporation and thereby increasing its contribution to the value of the enterprise. They answer the question: Have you [R&D] helped me [the Business] avoid problems and used my resources wisely? Because of their value in this regard, they are useful metrics for targeting future R&D performance. However, they only represent part of a process that involves other functions besides R&D. And, given that they focus on the quality of a ‘process’ and ‘processes’ take time to change, the numbers reflected by these metrics are associated with activities that are in the past. These metrics are lagging indicators. They are a nice track record, but they may not be reflecting accurately a current level of effectiveness.
3. How to Use the Metric
The metrics should be tracked at least on a once a year basis. Because of measurement and definition problems, a baseline of two years or more of historical data is needed before accurate judgments can be made about trends and ratio efficiencies.
The metrics should be examined carefully for consistency with business strategies and the results required vs. the investments in R&D. In situations where the metrics indicate a high late(r) stage kill, there will be a difficulty in taking on high risk programs. This means that this metric should provide a guide to the level of risk that can be undertaken in executing the overall business and technology strategies. One or the other must be shifted, and variations in how R&D is conducted and managed need to be examined.
4. Options and Variations
The most common option is to apply these metrics to all stages and not just the last one. Originally, and most typically referred to as ‘late stage kills’, these metrics can be generalized to apply to all stages. However, they lose the potency and impact of having to ‘write-off R&D’ at a time period when this should not be the case. Another variation is to apply these metrics to future prospective.
COST RELATIVE TO BUDGET
1. Metric Definition
The cost effectiveness of R&D is measured by comparing the value of technology output to the cost of producing that output. Technology output is generally measured with respect to objectives, such as short-range or annual targets or long-range or strategic targets. Costs are generally measured with respect to budgets, such as annual budgets or longer-range or cumulative project budgets.
1.1 Metric -1: Annual Budget Performance
Absolute and relative variance of actual total annual project cost compared to the agreed-on annual project budget, where both are accounted on the same basis. This metric may be defined for an individual project or class of projects, for all projects from a specific R&D unit, for all projects supporting a business segment, or for the total R&D in the firm.
1.2 Metric -2: Cumulative Project Budget
Absolute and relative variance of total cumulative project cost compared to the agreed on budget for an individual project. This is especially important in the early stages of a multi-year project as an indicator of future trends; it is important toward the end of the project as a measure of total cost for comparison with total benefit.
2. Advantages and Limitations
These metrics are important to management in a firm and have the advantage that they are numerical and appear to be objective. The accuracy of the metrics is limited by the quality of the accounting processes in the firm and by the quality of the effort to assign costs accurately, including those costs which come from shared or purchased resources. Since the standard of measurement is the project budget, the quality of this metric also depends on forecasting a budget which is justified by the merits of the project and provides appropriate, affordable resourcing to project R&D. As knowledge increases and the project progresses, it may be necessary to adjust the previous forecast budget in order to make the performance versus budget a meaningful measure of effectiveness.
3. How to use the Metric
The first key to use of these metrics lies in good analysis and understanding of how manpower, support, and other costs are incurred in R&D for the project, collection of projects, or R&D unit of interest. The second key lies in establishment of project budgets which are meaningful and which reflect accurately the expectation of costs to be incurred. The third key lies in accurate assignment of costs on an ongoing basis. Then costs can be compared to budgets at appropriate times and the metrics can be computed. The fourth key to use is the avoidance of misuse: the purpose of R&D is to create value, not to demonstrate that project budgets can be forecast precisely.
4. Options & Variations
These metrics can be defined for an individual project, for a classification of projects (such as all exploratory projects), for the projects supporting a business unit, or for the total R&D effort of the firm.
5. References
Ellis, L. W. 1984. The Financial Side of Industrial Research Management. New York: Wiley.
Ellis, L. W. 1984. Viewing R and D Budgets Financially. Research Management, XXVII (3), 35-40.
DECISION GATE PROCESSES
1. Metric Definition
This metric is intended to assess the decision process itself at decision points (gates) of the stage gate innovation process. For the generalized case of a work process, such a metric is important to the management and improvement of the process. There are two important metrics: (1) an objective measurement of the degree to which a formal decision process is applied; and (2) a subjective rating of the effectiveness of the decision process.
1.1 Use of Decision Process
The use of a decision process is characterized by computing the proportion of programs (either budget % or number %) passing through the stage gate process which have been subjected to a formal decision process at the last gate encountered.
1.2 Quality of Decision Process
The quality of the decision process is rated subjectively (on a scale of 1-4) with respect to the primary dimensions of Fit, Attractiveness, and Capability. 4Parameters which define Fit (Attractiveness, Capability) are precisely defined and were the objective basis for decision at the last gate3Parameters which define Fit (Attractiveness, Capability) are well defined and were given primary consideration in the decision at the last gate.2Parameters which define Fit (Attractiveness, Capability) are poorly defined and received limited consideration in the decision at the last gate.1Parameters which define Fit (Attractiveness, Capability) are neither defined nor given significant consideration in the decision at the last gate.
2. Advantages and Limitations
These metrics are subjective measures of what is intended to be an objective decision process. They are intended to drive the use of objective decision processes at the gates of a stage-gate innovation process. It should be recognized that the best R&D decisions may not be obtained by deterministic processes. It may be that (3) in the scale above is actually the optimum, when combined with excellence in subjective technical and commercial judgement.
3. How to use the Metric
This metric is relevant to a stage-gate innovation process in which go, no-go, turn-back, abandon decisions are made at each gate. Metric 1.1 measures the use of the process and Metric 1.2 rates the quality of the decision process. Metric 1.1 may be computed by clearly identifying the formal decision process and by computing the proportion of projects subjected to it. Metric 1.2 requires a subjective evaluation by knowledgeable people.
4. Options & Variations
The best measure of the effectiveness of the decision process is the quality of subsequent results (e.g. the proportion of projects surviving the subsequent gate) but that is retrospective. The current metric is intended to be prospective and focuses on the nature of the decision process.
5. References
Van Remoortere, F., and Cotterman, R. 1993. Project Tracking System Serves as Research Management Tool, Research¥Technology Management, 36 (2), March-April, 32-37.
PROBABILITY OF SUCCESS
1. Metric Definition
This is obviously a prospective measurement. It is a good measure of the wealth and depth of the new product pipeline. It can also be used as a key criteria for passage through gates in the stage gate process.
The probability of success on an R&D program ( project ) depends on the likelihood that technical problems can be resolved and that the market opportunity can be realized. The product of the two factors is an educated guesstimate of the probability of commercial success. The probability of success times the projected peak sales is the projected, probability corrected sales estimate.
The sum of all major projected, probability corrected sales is a good prospective measure of the richness of the R&D pipeline. Major changes in the total,particularly on the downside, should be thoroughly investigated for cause and remedial action taken if warranted.
The probability of success of a program can be used as a key criteria for passage through one or all of the stage gates. The probability should remain constant or increase as the development proceeds.
2. Advantages and Limitations
Advantages of the measurement are:
Among the best prognosticators of R&D’s future success
Invaluable in analysis of portfolio balance
Measured over time, provides a good measure of the organizations predictive ability
Can provide warning alarm for problems and fears not yet articulated
Disadvantages/Limitations
Qualitative and very subjective
Subject to gamesmanship to push favorite projects
Requires solid market knowledge and forecasting ability
Subject to usual technical overconfidence
Early market forecasts and become “engraved in stone”
3. How to use this metric
An informal panel of program or project members and other lab experts should review the program and especially the technical issues and obstacles impeding it’s progress. Each panel member should rate the likelihood of project success ( including meeting all product attributes ) on a scale of 1 to 10 where 10= very simple to complete and 1= insurmountable obstacles exist. Marketing personnel should provide regular updates on the probability of commercial success using the same numerical system. The product of the two factors is the overall success probability. Significant changes, particularly drops, should be investigated for underlying causes. The sum of the probabilities times projected peak sales across the R&D portfolio is an excellent prospective of the firms new product prospects.
4. Options
The projected, probability corrected sales forecast spread over 5-10 years is a valuable measure of the R&D new product pipeline. Gaps are easy to spot and remedial adjustments in the portfolio can be made. Probability trends on a program can be used as an indication of enthusiasm or lack of it. Calculations can be refined using net present value calculations, but the rough nature of the estimates usually negates the effort.
5. References
Merrifield, D. Bruce, 1981, Selecting Projects for Commercial Success, Research Management, 24 (6), 13-18. This is an early source article for the technical probability of commercial success.
Ellis, L.W., 1984. The Financial Side of Industrial Research Management,, New York, Wiley. Chapter 6 discusses the probability distribution of outcomes.
TECHNOLOGY PLANNING
1. Metric Definition
This metric employs a four stage/interval scale for an assessment of key elements in the technology planning process and in the technology plan itself.
Technology Planning/Plans: Self Assessment
The following will be put in matrix format: criteria vs. levels of performance 1-4:
Planning Processes:
involvement:
business planning and technology planning are either non-existent or remain superficial and are not used as a basis for action.
Business planning and technology planning are carried out as separate processes with little or no involvement of technology in business planning.
Technology function is asked for input into business plans and business plans provide basis for technology plans; technology is often in a responsive mode.
Technology function is an integral part of business planning significantly impacting the plans and technology planning is an important part of the business planning process. Senior professionals have opportunity to be involved and provide input.
timing:
technology planning is done sporadically and is not built upon prvious plans.
technology planning is done periodically responding to organizational needs (demands for planning or for budget preparation)
technology planning is done regularly on an annual basis and results reviewed regularly. Plans are iterative — building on last year’s plan.
technology plans are “evergreen” and are used regularly to guide action and decision making and are modified periodically based on results and external events.
internal communications:
technology plans are not communicated to the R&D technology organization (non-existent, or incompletely formulated)
technology plans and related business plans are communicated to and understood by less than half of the technical organization. Some familiarity by the members of the business team.
technology plans and related business plans are communicated to and understood by more than half of the technical organization; members of the organization understand how their programs are supportive.
major parts of the technical organization are integrally involved in the planning process; communication/understanding/doing are are all interrelated.
external communications:
Little if any communications of needs and opportunities outside the immediate organization.
What cooperative efforts and discussions take place with those outside the immediate organization are not done within the framework of the technology plan or critical needs of the the business.
members of the technical organization meet sporadically with technical resources outside their organization to discuss needs spelled out in their plan.
members of the technical organization under the leadership of the technology director interact regularly with others outside the organization (corp. other R&D units, Universities, other external research sources) to look for opportunities to address the issues in the plan. Continuing relations/cooperative efforts are built upon past successes. (addition: where R&D is separate from the business/operating unit — may want to have gate keeper/formal liaison).
learning/improving (Quality)
No learning takes place based on results of the previous plan and projects.
Some lessons are simply acknowledged; some actions taken in response
The planning process and/or results of individual programs/projects are reviewed in a semi-formal manner; some actions taken within the organization to to improve the process and/or execution of future projects.
The planning process and results are reviewed and improvements/responses to learnings are carried throughout the organization.
Elements of the Plan itself
business issues
technology programs are not explicitly related to key business needs/strategies
technology programs are related to business needs in only vague terms
it is clear how results from technology programs contribute to business success
it is clear how results from technology programs not only contribute to business success but how some results can provide a real change in the game.
nature of the markets, customers
little if any reference to markets and customer needs
a few customer needs are mentioned
the “voice of the customer” is heard in the plan; market structure/environment are well understood
customer needs are spelled out but future needs in the marketplace are anticipated
Competitors
little if any reference to competitors
some competitors and their position in the market are recognized
Competitive intelligence is reflected in the plan with an understanding of most competitive threats, in-kind and not-in kind
. technical program strategies recognize the thrusts of competitors — including their technological strengths and weaknesses (addition: include recognition of potential end runs from non-traditional competitors)
human resources
people’s skills/competencies are not mentioned in the plan — other than number of budget people for individual projects
people needs are addressed — but not in connection with skills and competencies
skills and competencies needed are mentioned but with no analysis.
the skills and competencies needed to carry out the technology plan are addressed in the plan — both near and longer term. Needs to fill gaps are discussed, and a plan to fill gaps
contingencies/alternative plans
no alternatives to the planned programs are presented
any alternatives or contingency plans are not realistic and offer no real choices
limted cut and dried alternatives are presented
alternatives and contingencies offer options in the case of changes in resources or major changes in economic or market conditions.
other elements of the plan
none of the following are addressed to any extent: capital productivity costs product quality opportunities for growth in the business environmental challenges maintenance/development of (core) competencies opportunities for breakthroughs
some of these elements are addressed
most of these are addressed where they are important to the business
None are overlooked and impact on technology programs are seen.
Core competency considerations
Core competencies (technology) are not considered part of the planning process.
Core competencies are discussed but are not central to the planning process.
The planning process defines core competencies required and the plan to develop and maintain.
Core competencies that are in place are understood and are an important consideration in the planning process. Competencies that are required are clearly defined with a clear plan of action to acquire the competencies and to prune those not needed.
Time Frame of Plan
essentially one year
1-3 years
3-7 years
7-10+ years (may also be framed around the planning cycle)
Content/documentation of the Plan
plan covers the relevant details and consists of prose and data. No effort made to use visual planning charts.
plan covers the relevant details with back up prose and data. Visual planning charts are included in the plan.
plan covers the relevant details with back up prose and data. Executive summary defines clearly the most important issues.
Plan covers the relevant details with back up prose and data. Plan is summarized in less than 15 slides using visual planning charts which easily and clearly define needs and actions.
Patents/intellectual property
no mention of value and impact of patent estate
brief mention of key patents currently held
patent estate is presented along with key patents of competitors.
value of present patents is understood and analyzed along with plan to build estate in key areas; key patents of competitors are also included.
Other areas to consider in the planning process:
a) training and skills needed to be developed or improved to do the planning
b) Continuous improvement of the planning process (applying TQM to this process)
c) basis for alignment/resource allocation for all parts of the organization.
d) assuring that plan delivers results — accountability; relation to performance management.
Other areas to be considered as elements of the plan (some may be in other criteria):
a) explicit connection between technology development and sustainable competitive advantage.
b) balancing the portfolio and aligning with other business initiatives
c) identifying and nuturing critical technical capabilities.
d) meeting needs of both external and internal customers.
e) understanding evolution of markets and changes in technology needs.
f) understanding relation of technology and product life cycles. (S curves and technolical limits)
g) metrics for measuring implementation of the plan and success of the planning process.
h) time-sequencing of steps for implementation (technology road maps?)
2. Advantages and Limitations
This assessment matrix can provide some common ground for assessing the quality of a process and the result of the planning process. Since a common matrix is being used by a number of firms it provides a way to make some external comparisons. The assessment factors/elements however should be modified to reflect the process and plans for the individual firm.
3. How to use the metric
Those doing the planning and developing plans should use the matrix as a self assessment tool and then focus on areas for improvement.
4. References
“Integrating Technology and Business Planning in IRI Companies” 73 minute video, self assessment matrix, Slides used by speakers in the video, written summary, and bibliography; available from Industrial Research Institute, 1550 M St. NW Suite 1100, Washington D.C. 20005-1708
P.S. Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37
ENVIRONMENTAL MANAGEMENT IN R&D
1. Metric Definition
This metric uses a stage/interval self assessment/ appraisal framework to determine the quality of environmental management in the R&D function.
“Reactive”
a) policies and strategies: environmental policy not developed or well-communicated; environmental problems and opportunities given limited attention by management and staff.
b) scope of activities: focus solely on regulatory compliance by the R&D facility; no environment-related R&D or employee awareness activities
c) level of participation: limited to a few (safety and environmental group)
d) Management processes and systems: administrative procedures in place only for compliance management
“Participative”
a) policy developed and publicized; management encourages participation and contribution
b) good citizen initiatives underway; awareness and training programs in place
c) entire staff involved in one way or another in citizen initiatives; limited involvement of management
d) Total Quality of Continuous Improvement Teams deployed
“Active”
a) strong ownership of policy by staff; management ensures environmental planning and review for R&D projects
b) projects subject to environmental planning and analysis
c) entire staff involved in environmental planning and review of R&D activities
d) environmental check-lists used for project screening; stage-gate system used to guide developments
” Innovative-Leader”
a) policy encourages active search for environmental innovation and influences corporate direction; strong commitment of corporation to R&D initiatives
b) large portion of portfolio aimed at cleaner technologies
c) all units and functions in the corporation feel challenged to contribute or support innovation initiatives
d) corporate, business, and R&D plans are aligned on environmental strategies and priorities; senior management provides oversight with periodic reviews
2. Advantages and Limitations
Advantages and limitations are similar to other “quality transformation grids” or assessment matrices.
3. How to use this metric.
The matrix is to be used to find areas for improvement and to assess the job being done in comparison with others.
INFORMATION USE IN R&D
1. Metric definition
This set of metrics measures the extent of use and the ways in which information technology is used within R&D.
1.1 Extent of Information Technology Use in expenditures in R&D budget
IT expenditures are a measure of the enhancement of R&D staff effectiveness by the use of Information Technology. TYhe trust of this metric is also how to to justify this IT spending to higher management and how to determine an optimum amount. The form of the metric is usually to measure the amount or ratio of the amount spent of IT hardware, software, and effort to total R&D expense.
1.2 Ways in which Information Technology is employed- Impact of IT on R&D
Four stages may be developed:
IT is only used to keep track of staff times and activities
IT is also used to provide tools for technical computing and/or hardware and software for the company’s products and services
IT also enhances the effectiveness of innovation management by substituting for human effort in R&D
IT enables the rethinking of how R&D is done and doing things that cannot be done in any other way; IT may also allow entirely new output from R&D.
1.3 An alternative to 1.2
A 9 box grid may be created:
Columns are Processes Impacted: Technical Computing; Mgt. of Technology; Integration of R&D with rest of Company
Rows are Type of IT impact: Tools to improve efficiency / effectiveness of R&D; Substitutes for old or existing ways of doing R&D; Tools that enable R&D to do things not possible before.
1.4 Extent of IT use in Managing R&D
The intent of this metric is to measure how far along an organization is at using IT in managing the innovation process. Four stages can be seen:
IT is only used by R&D groups on an individual basis
IT is integrated between the IT function and a few groups in the R&D activity
IT is fully integrated in the process of managing technology within R&D
IT is fully integrated in the process of managing technology across the firm.
1.5 Other possible metrics that might be developed
Breadth of usage by application category vs. benchmarks
Quality of usage by application category vs. benchmarks
Percent of R&D staff using IT at benchmark levels
2. Advantages and Limitations
The advantage of this metric is that it enables the justification of IT in R&D both at financial budget time, and in benchmarking IT use and impact in R&D against competition and strategic goals.
A limitation is that these are relatively untried metrics, with minimal research support as to their effectiveness.
Information Technology is evolving and developing so rapidly that its dimensions are not fully appreciated and some of the measures may be difficult to make.
3. How to use the metric
The amount of IT expenditure should be tracked over time, usually at the annual budget cycle. This can be compared to other measures of R&D effectiveness to see if the change in IT expenses has enhanced effectiveness.
The ratings of extent of impact and use should also be tracked over a time period for comparison with competition and strategic needs.
4. Options and Variations.
Service companies and companies with high IT content in their operations may wish to use alternately any of the software effectiveness metrics such as the Cpability Maturity Model developed by the Software Engineering Institute at Carnegie- Mellon.
5. References
The most general reference is U.S. National Research Council, Information Technology in the Service Sector, National Academy Press, Washington D.C. 1994
The SEI Capability Maturity Model is reviewed and compared with Total Quality Management and other evaluation techniques: H. Saiedian and R. Kuzara, “SEI Capability Maturity Model’s Impact on Contractors,” Computer (IEEE), 28(1), Jan.,1995 , 16-26; 12 references
IDEA GENERATION AND CREATIVITY
1. Metric Description
This metric uses an interval rating scale assessing the state of creativity and innovation within an organization. Four stages of development/performance can be given:
a. managerial control systems and organization discourage individual and organizational creativity
b. Ideas are encouraged and creativity is values in assessing performance of individuals, but no formal idea management tools are used. Creativity training is not generally available or promoted. Risk taking is discouraged directly or indirectly. Innovation takes place within well known and understood arenas.
c. Champions for new ideas are sought and supported. Creativity skills are taught and formal mechanisms are used to obtain new ideas from employees. Idea awards are presented to individuals and teams. Higher risk innovation for the creation of new opportunities is valued, but the technology management for existing businesses is typically a preferred route for advancement and status.
d. Innovative and entrepreneurial behavior among employees is encouraged and rewarded. Model behavior is highly visible. Funding programs are available to test new ideas outside the mainstream. The organization demands that scouting time be used to generate new approaches and ideas. Organizational status is higher for creation of new opportunity than for maintaining existing business. External sources are integrated into new idea processes. Idea banks and support systems are commonly used.
2. Advantages and Limitations
Assessing idea generation and creativity within an organization is not straightforward and most likely other measures will need to be developed.
3. How to use the metric
Members of the organization use this self assessment tool and then seek areas for improvement.
4. Options and Variations
Organizations may also want to study the extent to which “creativity tools” are used: Brainstorming, brain writing, lateral thinking (random input, escaping from thought patterns, building on provocations), metaphoric thinking, forced visual associations, guided imagery, six hats thinking, criteria setting
5. References
G.F. Farris and L.W. Ellis, “Managing for Change in R&D”, Research-Technology Management, 33(1), Jan.-Feb. 1990. 6.2 various works by DiBono
PEOPLE DEVELOPMENT
1. Metric Definition
1.1 This metric uses an interval rating scale for assessing the recruitment, development, evaluation, and rewarding of R&D personnel : (ref. 1)
Level 1
a) recruit only for specific openings
b) recruit only from local area
c) no training or development programs
d) rewards based on who you know; favors most senior people
e) no support to first-line supervision in evaluation process
Level 2
a) recruit locally and regionally
b) training programs unrelated to strategy needs
c) limited communication on reward system
d) no significant difference in monetary rewards for different performance levels e) no meaningful rewards other than salary increases
Level 3
a) active nationwide college recruiting
b) career development programs
c) training at all levels
d) three-year personnel plan
e) realistic appraisals
f) appraisal training
g) written evaluations, annual reviews
h) significant differences in rewards for top performers
Level 4
a) recruiting based on skill mix, competency analysis ,and long term staff development planning
b) all management levels involved in selection
c) effective dual ladders
d) recruitment and development recognize need for global technology management
e) planned balance of roles
f) incentives for entrepreneurial behavior
g) interfunctional and international career opportunities
h) mix of individual and team rewards
i) personnel development accomplishments a key factor in evaluations of managers.
1.2 Measure of the inventory of ideas in the pool
A listing is maintained of ideas coming from various sources within the firm. The organization develops some standards on number of ideas contributed in the last three months and for the year. A measure of the number of backlog ideas is also developed.
1.3 Measure of the amount of effort devoted to idea generation
An assessment is made of the typical % of time devoted to idea generation by members of the R&D staff.
2. Advantages and Limitations
The advantages and limitations are the same for the use of any “quality transformation grid.
3. How to use the metric
The grid descriptions should be tailored to the particular organization; then members of the organization make the assessment to find areas where improvements are needed.
4. References
P.S. Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37
C.J. Cranny, P. Cain-Smith, and E.F. Stone, Job Satisfaction, Lexington Books, New York, 1992
R. Katz (ed), Managing Professionals in Innovative Organizations, Harper Collins, New York, 1988
L.W. Ellis and S. Honig-Haftel, “Reward Strategies for R&D”, Research-Technology Management, 35(2) March- April, 1992 16-20.
INTELLECTUAL PROPERTY MANAGEMENT
1. Metric Definition
This metric uses a four stage interval rating scale to assess the state or stage of intellectual property management within an organization: (ref. 1)
Level 1
a) ignored
Level 2
a) rewards for patents
b) intellectural property issues left to legal
Level 3
a) selective patenting based on evaluation of pros and cons of disclosure
b) in-licensing if needed and out-licensing if asked
c) trade secrets defended in court
Level 4
a) intellectual property opportunities are part of business strategy, project selection, and project management criteria
b) in-licensing to maintain focus, speed external comparison, and learning opportunities
c) technical personnel rotate through intellectual property assignment
d) out-licensing based on business and technical assessments
e) comprehensive trade-secret policies
2. Advantages and Limitations
The advantages and limitations are the same for the use of any “quality transformation grid.
3. How to use the metric
The grid descriptions should be tailored to the particular organization; then members of the organization make the assessment to find areas where improvements are needed.
4. References
Adler, D William McDonald, F MacDonald, “Strategic Management of Technical Functions”, Sloan Management Review, Winter 1992, 19-37
Griliches “Patent Statistics as Economic Indicators: A Survey” Journal of Economic Literature, XXVIII, December 1990, 1161-1707
Narin, “Technology Indicators in Strategic Planning”, Science and Public Policy, December 1992, 369
R&D CLIMATE
1. Metric Definition
This metric provides a way to score the overall climate in an R&D organization — especially to see the extent to which it is characterized by “Third Generation R&D”: If the index is put into four stages similar to other metrics we can outline stage 1 and stage 4 (ref. 1)
Stage 1
No explicit strategic framework for technology management
R&D a second class citizen
R&D isolated
R&D strategy is not linked to division strategy
Management is not tolerant of failure
Business planning does not include R&D
R&D funding of programs varies with earnings forecasts
Priorities are set according to near-term business conditions
R&D performance measured according to product successes
R&D has poorly defined projects
Progress is evaluated only when things go wrong
“Fires” constantly cause unplanned changes to programs
Manufacturing becomes involved only during late stages
Marketing input only at instigation of project
Funding is a function only of internal needs/requirements
Sales and Marketing planning is the province of GM and marketing managers
All projects are managed by individuals
R&D projects are evaluated by R&D on an individual basis
Projects are never terminated
Stage 4
Corporate-wide strategic framework for technology management
R&D is respected and a partnership exists between it and all other areas of the organization
R&D is fully integrated
R&D and business strategy are integrated corporate-wide
Management is tolerant of failure
Division and corporate business includes R&D participation
R&D funding varies with technological maturity and competitive impact
Priorities are set according to contribution to division or corporate strategic objectives
R&D performance is measured according to division or corporate business objectives and technological expectations
All R&D projects are well defined
Progress is evaluated regularly and whenever events warrant
“Fires” are dealt with without interfering with planned R&D programs
Manufacturing included in project planning
Marketing participates in project reviews
External competitive environment also considered in funding decisions
R&D staff have input to sales projections and marketing plans
Large complex projects are “managed” by multifunctional teams
R&D projects are evaluated as part of a portfolio taking into account risk/reward, time horizon etc.
Projects are continually weeded out
2. Advantages and Limitations
This metric allows for a quantitative comparison between R&D organizations. Some aspects may indeed by subjective.
3. How to use the metric
The scoring is probably best done by a number of different individuals within the organization.
4. Options and Variations
This scoring system has been developed by AD Little; an assessment scheme for R&D decision making has also been developed by Strategic Decisions Group and could be used instead.
5. References
Roussel, K.N. Saad, and T.J. Erickson,Third Generation R&D, Harvard Business School Press, 1991
