Reading the evidence well
Research methods, interpretation and better choices.
A half day on reading the evidence well: why data is never raw, collecting data and judging quality, reading data honestly, noise and bias, cost benefit analysis, unintended consequences and sensemaking.
- Level
- Explore
- Learning time
- 6 hours
- Where
- Complexity, decisions and innovation
- Price
- £129
When you step up to a higher level, what you paid in the last 12 months for lower-level courses inside the bigger course comes off its price, in proportion to how much of each is inside it, whichever topic it's from. Courses at the same or a lower level don't give credit against each other. A Focus course or above always costs at least £15. Read the full terms
Certificate of Completion, or Certificate of Completion and Assessment with a short assessment
Overview and Outcomes.
Who It's For
Leaders, board members and analysts in health, housing, local government and the voluntary sector who rely on data
Overview
Every figure that reaches a board paper was shaped by human choices. This first half of From data to decisions begins with Lisa Gitelman's argument that raw data is an oxymoron and Cheryl Strauss Einhorn's data types, then collecting data through Joseph McGrath's three-horned dilemma and two Welsh cases. Quality follows with Deming's standard and the Labour Force Survey, then reading data honestly with Shewhart on variation, Gigerenzer's natural frequencies and Kahneman's work on noise. The final section brings in cost benefit analysis, the 2020 A Level grading algorithm, Goodhart's and Campbell's laws, and Weick, Snowden and Rosenhead on sensemaking under uncertainty.
Aims
What it sets out to do is help leaders question any number before acting on it: choosing methods that follow the decision, judging how much weight a figure can bear, reading data together without noise and bias, and noticing when a measure starts to distort the work it was meant to see.
In summary, you will be able to:
- Use the five questions for any number to judge how much weight a figure can bear before you act on it.
- Distinguish common cause from special cause variation, and apply decision hygiene so that a team reads data independently before combining views.
- Design a small basket of measures for learning that starts with purpose and adapts when measures distort behaviour.
What you'll develop
Across this Explore, you will develop the ability to:
- Explain how human choices at each of the six steps of the data journey, from first question to final decision, shape the figures you are given.
- Plan data collection by working through the six steps from decision to method, starting with the decision the data must inform.
- Apply Deming's four tests of accurate, valid, precise and reliable to a figure you use, and identify where it is weakest.
- Distinguish common cause from special cause variation, and explain why reacting to every movement as a signal, which Deming called tampering, makes a stable process worse.
- Describe the steps of a cost benefit analysis and how social cost benefit analysis adapts them for public decisions.
- Apply Deming's four tests of accurate, valid, precise and reliable to a figure you use in your own work.
- Explain why measures change behaviour, using Merton's causes of unintended consequences and Goodhart's and Campbell's laws.
- Distinguish prediction problems from sensemaking problems, and explain how sensemaking differs from the gather, analyse, decide model.
Built from 9 modules.
6 hours of learning.
- 01Foundations: Why Data Is Never Just the NumbersUnderstanding Data and Data Analysis38 min
- 02Collecting Data: Method Follows the QuestionUnderstanding Data and Data Analysis38 min
- 03Types, Sources and Quality of DataUnderstanding Data and Data Analysis38 min
- 04Reading Data HonestlyUnderstanding Data and Data Analysis38 min
- 05The Same Data Point, Seen DifferentlyUnderstanding Data and Data Analysis39 min
- 06Cost Benefit and Market Analysis: Valuing Your OptionsSolving Complex Problems and Decision Making38 min
- 07Understanding Data: The Evidence Base for Good DecisionsSolving Complex Problems and Decision Making39 min
- 08Determining Outcomes and Unintended ConsequencesUnderstanding Data and Data Analysis39 min
- 09Sensemaking with Data under UncertaintyUnderstanding Data and Data Analysis39 min