In this article
Idea Management Software
KWOKKA by UBT™ collects ideas from employees across your organization and gives each one an owner and a defined path.
Book a WalkthroughKey takeaways
Agree the idea evaluation criteria before the ideas arrive, with the people who will be judged by them.
Sort ideas into core, adjacent and breakthrough, and start with core.
Keep scoring simple: three or four criteria, scored alone by at least three people.
Settle a disagreement with a small test that takes two hours or costs 20 dollars at most.
Give every idea one of three answers, with the reason: do it, test it more, or put it on the shelf.
Idea Evaluation Criteria: Agree Them Before the Ideas Arrive
Definition
Idea evaluation criteria
The questions an organization asks of every idea to decide what happens to it. They are agreed in advance, applied to every idea in the same way, and published where ideas are submitted.
Coming up with ideas is the easy part of innovation. A team can fill a wall with them in an hour. A wall of ideas is content. It becomes change only when someone decides which ideas move, in what order and on what grounds.
A wall of ideas, before any decisions.
Most advice on how to evaluate ideas begins with a scoring template, which is the simplest part. Three questions matter more, and each is settled before the first idea is scored: who agreed the idea evaluation criteria, who scores, and how a disagreement is resolved. This article covers each, with a plan for a short meeting.
If Innovation Is Everything, It Is Nothing
If innovation is everything, it’s nothing.
Shannon Phillips, Co-Founder and Innovation Strategist
When every idea is treated as equally important, no one knows where to start. Teams then choose by other means: the idea of the most senior person, the idea argued for most often, or the idea that arrived last. The people whose ideas were passed over see how the choice was made, and share less the next time.
Criteria written after the ideas arrive do not help, because each one reads as an argument for a particular idea. The weakness is long established: a paper by Messerle, Binz and Roth, presented at the International Conference on Engineering Design in 2013, noted that in business practice product ideas are often judged against criteria that are partly vague, incomplete or changing.
People trust criteria under three conditions:
They helped to set them. The people whose ideas will be judged took part, or were represented.
They apply to everyone. A director’s idea meets the same criteria as an idea from the front line.
They come with a reason. Every decision is explained by the criteria, including a decision to wait.
Sort Ideas First: Core, Adjacent and Breakthrough
The most useful step comes before any scoring: sort the ideas by type. It is also the first step in how to prioritize ideas, because an improvement to a weekly report and a proposal for a new line of business cannot be compared on one scale.
We use three categories. They come from Bansi Nagji and Geoff Tuff, whose article Managing Your Innovation Portfolio (Harvard Business Review, May 2012) divided innovation work into core, adjacent and transformational initiatives. We call the third category breakthrough.
Core
Improve What Exists
Changes to current products, services and ways of working, for customers and teams already served.
Adjacent
Extend What Works
Something the organization does well, taken to new customers or put to a new use.
Breakthrough
Create What Is New
A new offer or way of operating, for a need the organization does not serve today.
Our advice is to start with core. Core ideas cost the least and return value soonest. They show people that the process works, and they give many employees an early result they can see. The research behind the categories points the same way:
70%
of innovation resources, on average, went to core offerings at the high-performing firms the authors studied
10%
went to transformational initiatives, the category we call breakthrough
Source: Bansi Nagji and Geoff Tuff, Managing Your Innovation Portfolio, Harvard Business Review, May 2012
The authors add that the right balance differs by industry and by company, and that the largest long-term returns in their study came from the transformational share. Judge each category by criteria that suit it: financial measures that fit a core idea, they warn, can end a transformational idea before it has been explored.
Who Sets the Idea Selection Criteria, and Who Scores
Idea selection criteria are often written by one person. Such criteria can be sound and still lack support, because the people judged by them had no part in them. Agree them in a small group that includes:
The decision maker, who commits to deciding by the criteria.
People who submit ideas, including the front line.
People who would carry the ideas out, who know what the work costs in time.
One person who holds the constraints, such as finance, safety or operations.
Agree the scoring rules at the same time. Use at least three scorers for each idea. Each scores alone before any discussion, so that the first opinion spoken does not set the others. No one scores an idea they raised. The decision maker is named in advance and decides after seeing the scores.
A 45-Minute Meeting That Sets the Criteria
One short meeting is enough to agree the criteria and the rules around them. Invite five to seven people from the groups above, and ask each in advance to consider what would have to be true of an idea for the organization to act on it this year.
1
State the capacity: 5 minutes
The decision maker says how many ideas the team can work on at one time.
2
Write alone: 10 minutes
Each person writes what must be true of an idea before it goes ahead, one point to a note.
3
Reduce to three or four: 10 minutes
Group the notes and keep the points most people raised. Write each as a question two people would answer the same way.
4
Settle the rules: 10 minutes
Agree who scores, who decides, how a tie is broken and what happens when scorers disagree.
5
Check against past ideas: 10 minutes
Apply the criteria to one past idea that worked and one that did not. If they would have chosen badly, change them now.
A criterion is useful when it can be answered from evidence:
Instead of
Has high impact
Is feasible
Fits our strategy
Try this
Removes a problem that a named team meets every week
A first test can start within 30 days with the people we have
Supports one of this year’s stated priorities
In practice. Publish the idea evaluation criteria and the scoring rules where ideas are submitted. Review them once a year, and never during an evaluation.
Keep the Idea Scoring Matrix Simple
If you score ideas at all, keep the matrix simple. An idea scoring matrix lists the ideas in rows and the criteria in columns. It is a quick way to compare many ideas, and it does not replace judgment. Weighted formulas with decimal totals suggest a precision the estimates behind them do not have.
A three-point scale. Yes, partly and no are enough. Allow “not known”, which shows where a test is needed.
The spread as well as the total. When three scorers give one idea a yes, a partly and a no, the disagreement is the finding.
One matrix for each category. A breakthrough idea scored beside core ideas will lose on cost and certainty.
The common scoring methods (weighted scoring, impact and effort, ICE and RICE) each sort a list in a different way. We compare them, with a free template for each, in a separate guide.
Free templates
How each of the four scores, when it fits, what to watch for, and a one-page template for each.
Treat the result as a first filter: the score orders the discussion, and the named decision maker decides. In our innovation process this is the work of the first stage, Qualify, which decides whether an idea is worth pursuing.
Idea management software
Idea Management Software That Takes Ideas to Results
KWOKKA by UBT™ collects ideas from employees across your organization, helps your team decide which ones to pursue, gives each one an owner and a defined path, and records what it delivered.
When People Disagree, Run a Small Test
A split among the scorers usually means that people hold different assumptions about the same idea. Further discussion rarely changes those assumptions. A small test replaces them with evidence.
Key idea
A small test settles a disagreement faster than a debate.
Unbounded Thinking’s 2 Hour, $20 Test is our rule for a first test: if the test cannot be run within two hours or for 20 dollars, make it smaller. Name the assumption on which the scorers differ, write down the result that would count as a yes, and run the smallest test that could show it.
Agree a rule for ties in the criteria meeting. One workable rule: when two ideas score the same and there is capacity for one, the idea with the smaller first test goes first.
Three Answers: Do It, Test It More, or Put It on the Shelf
Every idea that is evaluated receives one of three answers, and the person who raised it is told which one and why.
Do it
The Idea Goes Ahead
The criteria are met. The idea has an owner, a first step and a date.
Test it more
The Evidence Is Not In
A criterion could not be answered, or the scorers disagreed. A small test is agreed, with a date.
On the shelf
Not Now, With a Reason
The criteria are not met today. The reason is recorded, with what would have to change.
The reason matters most for the third answer: people keep sharing ideas when they can see how decisions are made. Our article on how to respond to employee suggestions covers the reply itself.
How to Evaluate Ideas: The Checklist
Put together, these practices turn the question of how to evaluate ideas into a routine with four steps:
1
Agree
Criteria and scoring rules, before the ideas arrive
2
Sort
Core, adjacent and breakthrough
3
Score
Alone first, then discussed
4
Answer
Do it, test it more, or the shelf
Idea evaluation checklist
Three or four criteria, agreed by the people judged by them
Criteria and scoring rules published before ideas are collected
Ideas sorted into core, adjacent and breakthrough
At least three scorers for each idea, scoring alone first
A small test for every split among the scorers
One of three answers for every idea, with the reason
For a single session, see how to run a brainstorming session. How well these practices hold depends on how your organization already decides about ideas. The Innovation Readiness Assessment asks your whole team how the organization handles process, people, projects and performance, and shows where the gaps are. For a first look, one person can take the free Innovation Readiness Snapshot.
Limited-time Q4 offer
Innovation Process Certification at 50% Off
Your team can build this on a real problem in our two-day Innovation Process Certification, for up to 10 people. Alberta organizations that book by 30 November pay half our medium business price.
Frequently Asked Questions
What are idea evaluation criteria?
Idea evaluation criteria are the questions an organization asks of every idea to decide what happens to it. They work best when they are few, agreed in advance and applied to every idea in the same way.
How do you evaluate ideas?
Agree three or four criteria before the ideas arrive, sort the ideas into core, adjacent and breakthrough, and have at least three people score each idea alone. Settle a disagreement with a small test, and give every idea one of three answers with the reason.
Who should set the idea selection criteria?
The people who will be judged by them, together with the person who decides. Five to seven people is enough: the decision maker, people who submit ideas, people who would carry them out, and one person who holds the constraints.
What is an idea scoring matrix?
An idea scoring matrix is a table with the ideas in rows and the criteria in columns, used to compare many ideas quickly. Keep it to three or four criteria and a simple scale, and treat the result as a first filter.
How do you prioritize ideas when there are too many?
State how many ideas the team can work on at one time, then sort the ideas into core, adjacent and breakthrough. Start with core ideas, which cost the least and return value soonest.
What should happen when evaluators disagree about an idea?
A disagreement usually means an assumption has not been tested. Name the assumption, agree the result that would count as a yes, and run a small test. If it needs more than two hours or 20 dollars, make it smaller.
About the author
Shannon Phillips
Co-Founder and Innovation Strategist
Designs structured innovation systems for enterprise organizations and helps leadership teams set priorities that reflect how the organization operates.
Related Articles
Innovation Execution
What to Do After a Brainstorming Session
Psychological Safety
How to Respond to Employee Ideas So People Keep Sharing
Innovation Execution