11. Measurement and learning

Measure the decision, not just the activity

A measurement plan is useful when it tells the team whether to continue, revise or stop. Start with the decision and the proposed mechanism. If clearer eligibility information is intended to reduce unsuitable enquiries, total enquiries alone is ambiguous: fewer could reflect better self-selection or lower discovery. Choose observations that distinguish those explanations.

Separate outputs, audience responses, service use and mission outcomes. Publishing a page is an output. An enquiry is a response. Attendance is service use. Whether advice helps someone understand or act on a problem concerns a later outcome that needs its own evidence. Do not relabel the nearest available metric as impact because it is easy to count.

Define every measure before collecting it

A measure needs a numerator, denominator if relevant, time window, source, owner and interpretation limit. 'Conversion' is incomplete: enquiry among visitors, booking among enquirers and attendance among bookings answer different questions. A changed tracking method can alter a rate without any change in people's experience.

Use a small set: a primary response or service measure, a mechanism check and a balancing measure. For a proposed booking change, the primary measure might be attended appointments among eligible bookings, if eligibility can be defined appropriately. A mechanism check could examine whether the next step is understood. A balancing measure could track handling burden or whether access becomes worse for another group. The course does not require live collection.

Keep counts beside percentages. A rate based on a small denominator can move sharply after a few events. Rounding can disguise that fragility. Explain missing observations and differences between cohorts. If the website count excludes offline referrals, do not use it to describe everyone served.

Compare before-and-after carefully

The fictional baseline has 24 bookings and 15 attendances, giving 62.5% attendance among bookings. A later fictional month has 30 bookings and 21 attendances, giving 70%. The change is 7.5 percentage points, while the attendance count rises by six. Neither fact proves that a booking intervention caused the difference.

Other explanations include seasonal demand, referral changes, eligibility mix, staff availability, chance and recording differences. List the most plausible alternatives before celebrating. A before-and-after comparison can inform a decision, but its causal strength depends on the design and context. Reporting uncertainty does not make the work useless; it prevents the organisation from learning the wrong lesson.

Do not borrow a benchmark to hide this problem. A sector average may use a different denominator, service type or tracking method. Even a valid comparison does not identify which intervention would work locally. Use benchmarks to ask questions, not to manufacture a target or causal conclusion.

Understand what a comparison group can add

A comparison group can help estimate what might have happened without the change. Random assignment can reduce systematic differences between groups if implemented appropriately, but it does not solve every problem. Small groups, contamination, missing outcomes and an intervention delivered inconsistently can still undermine interpretation. Ethical and operational suitability comes before a neat experiment.

The supplied miniature test has 20 bookings assigned to route A with 12 attendances and 20 to route B with 15 attendances. The observed rates are 60% and 75%, a 15-percentage-point difference. The sample is small. Do not claim a reliable effect or statistical significance from these counts alone. You can describe the observation, limitations and a proportionate next decision without advanced software.

If assignment is not random, explain how groups may differ. People choosing an evening route may differ from those choosing daytime appointments in ways related to attendance. Comparing their rates does not isolate the effect of appointment time. Matching or adjustment may help in some settings but requires expertise and cannot recover information that was never observed.

Distinguish attribution and incrementality

Attribution assigns credit for an observed action to touchpoints under a chosen rule. Incrementality asks what additional result happened because of the intervention compared with what would otherwise have happened. A last-click report can assign all credit to a search result even when a partner explained the service first. A channel receiving credit is not necessarily creating additional benefit.

For a small organisation, a proportionate design may be a carefully documented staged change with explicit rival explanations rather than a complex experiment. State the causal limit and keep the decision modest. A rigorous test design using fictional data is an equally valid course route; an actual live test is not required for completion.

Set decision rules before seeing the result

A rule should include evidence quality, practical importance, capacity and harm, not just a positive number. For example, continue a bounded test only if information is accurate, operations can support it and the result is promising enough to justify further learning. Stop immediately for a consequential misleading claim or an unacceptable access harm, even if attendance rises.

Specify what to do with an inconclusive result. 'No clear difference' does not establish equivalence or prove failure. The team might keep the cheaper safer route, collect more suitable evidence or change the hypothesis. The right next step depends on the cost of uncertainty and the resources available.

Do not revise the success definition after observing the numbers without documenting the change. Exploratory findings can be valuable, but they should be labelled exploratory. A learning log records the original prediction, observed fictional result, competing explanations, decision and next uncertainty. This makes a disappointing test useful rather than something to hide.

Your measurement plan should show how evidence changes action. If every possible result leads to the same recommendation, the plan may be a reporting exercise rather than a test. A useful plan includes a plausible stop, a revision and a continuation path, with an owner responsible for each.

Interpret a result that does not settle the question

Imagine the miniature test had produced 12 attendances in each group of 20. You could report equal observed rates in those groups. You could not conclude that the routes are equally effective in every future cohort. The design may be too small, implementation may vary and the outcome may not capture the mechanism you intended to change. A decision can still be made, but its basis might be cost, accessibility or reversibility rather than a proven effect difference.

Conversely, the supplied 12-versus-15 observation does not automatically justify expansion. Ask whether the three extra attendances would be practically important if the effect were real, whether the new route increases staff burden and whether it serves the intended people. A statistically sophisticated analysis would not remove the need for these judgements. Nor should a small team invent statistical certainty to make its plan look advanced.

Prepare a result table with four possible patterns: promising response with no identified harm; little apparent change; better response with unacceptable burden; and unusable evidence. For each, write a different action. The unusable-evidence route matters: broken tracking or unclear denominators should lead to repair of the measurement process, not a performance conclusion. Keep the original prediction visible beside the result. This helps distinguish learning from retrospective storytelling and protects the organisation from repeatedly selecting whichever metric happened to look favourable after the work was done.

If a measure cannot be collected safely or consistently, choose a narrower question rather than collecting more information by default. A smaller, clearly defined observation with an honest limit is more useful than a large dataset whose meaning or permission basis is unclear.

Study allocation and completion

Allocate 1.5 hours to this lesson and worked example, 1.5 to critical reading, 2 to the lab, 3 to the assignment, 1 to critique and 1 to reflection and revision: 10 hours in total. These are planning allocations, not measured duration or a timed assessment. Read in short sessions if helpful. All resources are plain text; a notebook, calculator or optional free spreadsheet is sufficient. No video, AI account or paid service is needed.

For critical reading, use the linked selection if accessible and the internal reading comparison. Record its question, argument, evidence type, assumptions and one reason it might not transfer to your organisation. A missing external reading does not prevent completion: compare the two original internal positions and state that you have not appraised the external source. Keep a source log with title, URL, access date if accessed, claim supported and limitation. Links are references, not permission to copy or upload readings to AI.

Complete the lab manually first. The supplied simulated AI output is authored teaching material, not a transcript, real research or verified model performance. Mark each claim supported, unsupported or contradicted. Optional live AI may generate alternatives from these fictional inputs only; record tool, date, instructions, checks and revisions. Never supply personal, confidential or unauthorised material. If not using AI, compare the supplied output against your manual work and label that route accurately.

For critique, use the rubric and annotated example, alone or with a consenting peer; no peer is required. For reflection, revise one consequential decision and record the old choice, new choice, evidence and unresolved uncertainty. Keep work locally. Self-completion records learning activity, not professional competence or an independently assessed award. Version: 1 November 2026; report suspected errors through the course host's available contact route, or retain a correction note if no route exists.

Practical lab — 2 hours

Calculate both monthly rates, their percentage-point difference and the difference in counts. Then calculate the A/B observation. List at least four rival explanations for the before/after result and three limitations of the miniature test. Draft continue, revise and stop rules before reading the simulated AI conclusion. No live tracking or statistical service is required.

Portfolio assignment — 3 hours

Produce an 800-word measurement plan and test protocol. Include hypothesis, primary measure, mechanism and balancing checks, denominators, data source, permission boundaries, assignment/comparison plan, uncertainty, capacity limits and pre-specified decisions. Use the fictional case rather than collecting live information. Carry the protocol into the capstone strategy.

Critique — 1 hour

Use the rubric and annotated example, alone or with a consenting peer. Challenge your preferred option, a missing owner and a possible harmful consequence.

Reflection and revision — 1 hour

Change one consequential choice and keep a revision note. State what evidence would reverse the revised decision. No live contacts, spend, AI account or publication are required.

Self-review weights: evidence and source appraisal 25%; critical reasoning 25%; contextual fit 20%; feasibility and evaluation 20%; clarity and reflective judgement 10%. Score each 0–4 and sum(weight × score / 4). These are illustrative learning criteria, not validated academic grades.

Evidence: preserve counts, denominators and design provenance. Reasoning: separate observed difference from causal effect. Context: include mission and exclusion checks. Feasibility: propose proportionate data collection and a safe stop. Reflection: explain an inconclusive result without claiming equivalence.

0 means absent or misleading; 1 asserts without support; 2 supplies partial relevant support; 3 links evidence, alternatives and feasible action; 4 tests a strong contrary account and defines a reversal condition. Invented evidence, privacy breaches and unsupported public promises require correction before self-completion regardless of the score.

Weak: 'Attendance rose 40%, proving the campaign worked.' The count increased from 15 to 21, but causation is not established. Competent: 'Attendance among bookings rose from 62.5% to 70%, a 7.5-point change.' Strong: 'The later month has six additional attendances and a 7.5-point higher booking-attendance rate. Referral mix and timing may explain some or all of it. The small A/B observation is promising but insufficient for a reliable-effect claim. Continue only a safe bounded learning step, with handling capacity and access checked.' Annotation: correct arithmetic is necessary, but causal restraint and decision relevance distinguish the stronger analysis.