Innovation Performance Assessment Levels- Valutrics
Improving Companies performance implies having to embed innovation in a continually changing host. For a successful outcome, organizations must have an accurate, although not necessarily fully detailed, picture of where, in terms of overall innovation capability evolution, the company is at the start. Moreover, this has to be an insight in the context of the industry sector in which the company exists, the markets in which it operates and the overall state of its evolution. There are five different levels of initial assessment that can be undertaken. The approach for companies will vary; sometimes some but not necessarily all of them are used, and often only those that are felt to be most relevant. Largely depending on the specific circumstances, the time and resources available, as well as the initial scope, some are looked at first and the rest later as part of a second cycle of improvement. There are also two major risks in selecting the initial assessment approach, both of which can have detrimental impact. Looking at too few means that you end up still guessing and making stabs in the dark over missing information, whereas looking at too many at once can overwhelm both the company and the evaluation team. Either way, these levels of assessment are key to providing the initial data points from which current status can be reliably determined, changes can be introduced and progress can be measured. They are:
1 Overall corporate performance
2 Innovation activity performance
3 Sector and firm evolution
4 Opportunity identification, clarification and validation
5 Innovation environment.
Each has a different mechanism for evaluation, and each provides a different insight. However, when taken in combination, and of course ideally all together, they provide a unique view of where an organization is and more specifically where, given the appropriate enhancements underpinned by an understanding of industry context, it could be.
Overall corporate performance
Firms, and particularly their executives and investors, like to know where they stand in relation to their peers. Growth in share price, market capitalization and P/E ratios are all recognized means of comparing one company with another, but, as these can all include an element of M&A activity, sales performance, market conditions and other elements such as analyst sentiment, unfortunately none give a reliable indication of real corporate growth derived from innovation.
Effectively measuring how innovative a company is has been something that many have been grappling with for some time. In a bid to provide meaningful insights and, particularly, to understand relative performance, identify the leaders and hence question what it is that they do that makes them so innovative, a variety of approaches have been taken over recent years. Perhaps the most well known and the most indicative metric so far adopted has been 3M’s internal target of generating 30 per cent of revenue from products launched in the previous four years, a goal recently increased to 35 per cent. Taken in combination with the company’s more short-term driver of 10 per cent sales from products introduced in only the last year, again recently increased to 15 per cent, as mentioned in Part 1, this has certainly proven to provide both a very good motivation for that particular organization and an accurate representation of 3M’s collective innovation performance. However, unfortunately few other organizations actually track and even less publicly declare their performance against this or a similar measure. Thus although arguably an ideal metric, it cannot, without consistent inside information on every competitor, be used for any credible cross-industry comparison.
From the available public information there have been two main approaches, each with differing philosophies that have therefore been used for assessment of comparative corporate innovation performance. Especially in the USA, and increasingly across the high-tech arena, there has been a focus on innovative output which, given the broad range of end products, components and services delivered by business, has tended towards a league table based on the number of US patents granted to organizations in any one year. By using data from the US Patent and Trademarks Office, the definitive intellectual property registry today, consultants, academics, journalists and even companies themselves have all been able to produce rankings that, by implication, correlate a higher number of patents granted with better innovation performance. Hence, as a consequence, large multinationals tend to dominate. Alternative output measures, such as the number of new products launched in any one period, do not include any innovation threshold, and there is therefore no distinction between an enhancement and a wholly new product.
Although potentially more representative, this cannot therefore be seen as a reliable indicator. However, by implication, a patent is only granted if it relates to something that is both original and new and thereby provides a suitable common measure. In addition, although some may argue that patents granted elsewhere by the UK or European patent offices should also be included, given both the predominance of the USPTO as the place to file intellectual property and the fact that most companies submit multiple applications across several regions, the number of US patents granted is the most appropriate. MIT’s Technology Review uses an extra level of refinement in its annual patent scorecard by also taking into account how often a patent is cited as prior art in other applications as a means of trying to determine the worth of patent portfolios. The results, however, are largely the same as straight patents granted rankings, and are thus dominated by large firms and also open to subjective interpretation. Consequently, as a reliable and indicative innovation output metric, the number of US patents granted per annum, although not perfect, is one of the best available.
The other major approach has been to measure innovation input, most commonly seen as annual R&D expenditure. Again ultimately biased to larger organizations, drawing from company annual reports, this provides a separate ranking of innovation performance. In a bid to represent input more accurately, the same R&D scoreboard also includes a measure of ‘R&D intensity’ or, more simply, the percentage of annual revenue invested in R&D. Although better in terms of providing a relative rather than absolute metric, and thus being far more reliable for benchmarking purposes, this does nevertheless suffer from a sector bias. Pharmaceutical companies, by the very nature of their business, invest more of their revenue in R&D than in their electronics and automotive counterparts.
By implication, neither of these two approaches therefore provides a fully reliable indication of how well an organization actually uses its available resources. An innovative company is one that delivers more from less, and hence, to measure innovation performance, organizations should clearly take some account of the linkage between output and input. The most effective high-level measure for innovation performance is therefore to provide a ratio that represents this linkage. To be useful to as many organizations as possible, this has to be independent not only of company size, but also of revenue. Moreover, as far as possible, it also has to provide a reliable indication of relative performance between firms.
The approach that has been shown to be most effective has been to take the key reliable metrics for innovation output and input in combination. A measure of the ratio of output/input is more valid, and for this, output can be considered to be the average number of patents granted per employee and input to be the percentage of revenue invested in R&D. This provides a score, the innovation index, which successfully takes into account company size (number of employees) and revenue as well as the R&D investment and hard innovation output. Thus smaller companies are considered on a level field with larger ones. Moreover, as a ratio it gives a more accurate indication of relative performance. If one company has an innovation index twice that of another, then it can reliably be considered to be twice as effective at delivering innovation.
Although not perfect and, as with the US favoured patent approach, more applicable to products than services, the innovation index metric does provide a more reliable and consistent overall measure for comparative use. It is not necessarily the ideal top level metric for a firm’s balanced scorecard, but it does provide cross-industry insight and is therefore highly useful for setting one organization in the context of the rest of the sector in which it exists. It gives an idea of where top-level performance should eventually be if a firm is to be either up with the pack or leading its sector. When taken in combination with other internal assessments, it also gives a long-term target to aim for.
Innovation activity performance
Within any organization there are a host of measures that can be used to assess performance of the component activities that impact and influence innovation and growth. These cover a broad range in scope, from input to output, as well as in level, from business unit to individual task. This spread in itself makes the selection of the most appropriate indicator a challenge. However, to make matters worse, measures also vary from sector to sector and market to market. Potential candidates include:
• Number of new product launches
• Percentage of revenue from new products
• Market share
• Customer base growth
• Time to market
• Time to profit
• Average time between launches
• Average launch delay
• Change in product profitability
• Return on sales
• Market penetration
• Market ranking
• Installation time
• Number of employee suggestions
• Number of patents filed
• Number of patents granted
• Number of licences awarded
• Number of design awards won
• Percentage of projects killed
• Number of business cases evaluated
• Average improvement in service delivery
• Customer satisfaction
• Customer churn rate
• Resource utilization to plan
• Budget utilization to plan
• Employee satisfaction
• Average number of training days per employee
• Employee retention.
All are, in principle, relevant. However, measuring each one would turn into a logistical nightmare, take forever, most likely lead to information overload, and certainly delay the introduction of any improvement. Far more effective is to choose a few that are most relevant to the organization, its sector and its market, and for which reliable external data of either industry average or best practice are available. This means that an effective and pertinent view of current performance can be obtained in as short a time as possible.
For a consumer product company operating in four major markets, the internal metrics assessed for innovation performance were therefore agreed to be:
• Return on sales
• Number of successful new product launches
• Average time to market
• Market share in each core segment.
For a high-tech firm developing and licensing new instrumentation technology into the medical sector, the corresponding metrics were:
• Number of patents granted per annum
• Number of successful licences per annum
• Average time to royalty income
• Overall five-year return on innovation.
By contrast, for a financial services organization providing traditional and on-line banking to a consumer retail market within one region, the best measures were found to be:
• Number of new services introduced
• Customer churn
• Annual increase in operating margin
• Sector market share.
Lastly, for a data communications company providing high-speed connectivity to the global business market, the most appropriate innovation measures for evaluation were:
• Billed revenue from new services
• Average installation time
• Average launch delay
• Market share by sector.
Each organization clearly had different scope and therefore different core activities that contributed to its ability to create and deliver real innovation to its customers. As such, each set of these different measures was the most appropriate for gaining the necessary insight into individual performance. A consumer product company should be competent at quickly launching new products that increase market share, whereas a financial services organization should be effective at growing its customer base through the delivery of higher value services. These aims are different, and hence the specific capabilities required to achieve them can also be somewhat dissimilar. However, comparing like with like enables quick and effective identification of where any individual firm is, and is not, performing the applicable innovation activities to, or even above, industry and market expectations.
Sector and Firm evolution
One of the easiest but also one of the most insightful assessments of any organization is to look at where it, its competitors and the sector as a whole are in the overall evolution of innovation capability. Using the key steps in the evolution of innovation practice outlined in Parts 1 and 2, we can create a summary of all the elements that can contribute to the capability of an organization to innovate in terms of strategic, product, market, organizational and process focus useful representation not only of where an organization is generally but also of the arenas in which it is ahead or behind the competition and sector best practice.
For a financial services organization perceived internally to be class leading at innovation, the assessment showed that although it was ahead of the sector average in many areas, the movement of its lead competitor into personalization as apposed to mass customization of new products could be a major source of differentiation going forward. Similarly, in an IT services company, it was clear that the company in question was operating at an early stage of innovation capability evolution in all but two areas, the fact that this is largely typical of the sector as a whole indicated that, at the time, no major competitive advantage was either being gained or lost by any of the players. The consequence of this is that, if the company itself did not make an appropriate step forward, its competition could well do so. Whilst, for example, a move solely from using a stage-gate to a fuzzy-gate process may not deliver immediate competitive advantage by itself, taken in combination with an organizational change to introducing matrix-based teams and increasing involvement of key suppliers in the heart of the innovation process, the impact on its ability to launch more and better products more frequently could prove pivotal. However, the fact that a lead competitor was seen to be more advanced in terms of its capability in leveraging the significance of its brand did indicate a specific opportunity and threat that would have to be addressed going forward. This simple analysis therefore both highlights current status and identifies clear opportunities for improved performance.
Opportunity identification
The fourth key area for innovation assessment goes back to the initial perceptions offered by individuals across the organization. Whilst these are often highly subjective, influenced by personal circumstances, relationships and attitudes, and frequently symptomatic of under-performance, they may nevertheless provide accurate self-diagnosis. They therefore need to be captured and understood, but at the same time questioned and, if necessary, investigated. This is a lost cause if driven by the statements themselves, for, to be effective in terms of objective evaluation, any such clarification and validation of perspectives internal to the organization has to occur within a consistent framework.
Undertaking a combination of individual one-on-one interviews and team-based focus group type workshops is often the most effective and efficient approach for this.
Handing out questionnaires to be filled in by a sample selection of the organization is always an obvious route, but one that rarely delivers any specific insight. As with countless employee and customer satisfaction questionnaires, unless specifically focused on one or two fields for detailed and statistically reliable analysis, the survey approach to asking questions rarely provides an organization with anything other than the answers it is expecting.
A combination of closed questions, lack of opportunity to express opinions and fear of follow-up all combine to make the results from this type of approach largely superficial and predictable. Far more effective is the use of confidential interviews and open group based discussions facilitated by an objective and ideally independent moderator. The former provides the opportunity for more in depth exploration of issues relevant to the individual and the associated areas of input. Although requiring a willingness from the interviewee to talk openly and often taking an hour or so for each session, the insights gained from such an approach are often the most relevant in determining key areas that need to be addressed to improve performance. By contrast, group-based discussions enable the working of issues and opinions towards a common and agreed perspective. Often through the discussion of recent examples of success and failure to provide focus and relevance, a typical two hour workshop where individuals from across the organization are encouraged to provide their perspectives has multiple benefits.
As well as enabling more collective insights to the challenges and opportunities ahead, through effective facilitation, such group discussions can be pivotal in creating the basis for cross- disciplinary cooperation, providing stimulus for change and engaging key sources of influence.
In a packaging company where successfully delivering new products into an increasingly competitive market place was an acknowledged core problem, such questioning validated a number of initial perspectives, including:
• Ideas primarily reactive to competition
• Unwillingness to take risks
• Difficulty in terminating projects
• Little success in commercializing projects
• Poor communication across new product development
• Poor interaction between R&D and marketing
• Confusion over roles in projects
• Late application of commercial perspective.
It also identified a number of ‘new’ issues, including:
• No clear innovation strategy
• No clear ownership of product development
• No clear project selection procedure
• No idea collation across the organization
• Haphazard idea generation
• Little engagement of the organization in innovation
• Poor and absent decision-making
• Too much political influence.
At the same time several initial perceptions expressed by some were shown to be unsubstantiated, including:
• No process being used for innovation
• Functional isolation and protection
• Insufficient manufacturing input
• Limited internal testing and evaluation
• Localized innovation within R&D
• No clear launch responsibility.
Together these provided a good start in identifying what key issues had to be addressed within the organization. In addition, the parallel use of a short, reliable diagnostic questionnaire focused specifically on the single subject of the management of ideas across the company provided effective supportive information . Through a controlled survey of perceptions within a defined population, a collective view of current company performance was obtained that, as well as confirming points from interviews and workshops, could also be used at an executive level to summarize the main issues.
Innovation Environment
The environment within which innovation takes place requires greater up-front investment in time, a detailed evaluation of the skills, structure and culture of a defined group using a combination of techniques can provide highly useful contextual information in advance of the introduction of any change. More than just an indication of how a firm is performing at a moment in time, this can also highlight the potentially greater, as yet underutilized, capacity of the organization to innovate.
At the heart of the organization, the available skill-set is most directly associated with the process of undertaking innovation activities, but the organization’s structure and the prevailing culture both provide compound influence:
• The available skill-set provides the core means by which innovation can occur. The interaction between individuals and groups of different disciplines and levels of development is the fundamental element underpinning the core processes and the interdisciplinary working that promotes innovation within an organization.
• The structure within which the skill set operates directly affects the means by which the skill set can interact both inside and outside the organization. The more decentralized and informal the structure and the fewer the number of structural barriers the greater individual and group autonomy, but at the same time, maintenance of corporate coherence to an overall innovation strategy is potentially more difficult.
• The culture in which both the structure and skills exist determines how people behave, how they relate and therefore how they interact. It is the culture that characterizes a significant aspect of an organization’s capacity to innovate, as it helps to define the organization’s style as well as its values and practices .
Whereas the proficiency of an organization in performing innovation activities and maintaining core processes and linking mechanisms can be assessed through analysis of commonly available numeric data, measuring this softer side of an organization’s ability is less straightforward, for it is influenced by such issues as group dynamics as well as individual and corporate comfort zones.