Methodology
What we build for CELPIP® practice, how we check it, and what we do not claim.
Content: generated, then gated
Every item is written by an AI model from the published format — the component, part or task type, question counts and time limits — and never from the provider's own materials. No test-provider items, passages, audio or sample answers are copied, paraphrased or used as training data, including free samples.
Before an item can be used it passes independent gates: an independent solver must reproduce the answer key; every wrong option must have a defensible reason to be wrong and only one option may be right; a difficulty target is assigned; a reviewer pass checks naturalness, Canadian context and ambiguity. Items enter the bank only when all gates agree. After publication, response statistics keep watching: an item that strong candidates get wrong, or that does not separate strong from weak, is quarantined automatically and replaced.
Strategy lessons are made the same way. An AI session writes each one from the published structure; a separate AI editor session, which never sees the writer's instructions, checks it against our rules: no test-provider material, no wording to reuse, no number that is not in the published format, and nothing about scoring beyond the published dimension names. A lesson that fails gets one rewrite, and it is published only when the editor passes it.
Vocabulary sets go through the same writer and editor. Each entry is a word or a short collocation with its meaning, a usage note where one helps, and a prompt to use it in a sentence of your own — never an example sentence to copy. The editor also checks that every meaning and usage note is accurate, and that no amount, deadline or rule is stated as fact, because those differ between provinces and change.
The counts on our public pages — practice tests, task sets, lessons, vocabulary topics, and the word ranges of our Writing tasks — are counted from this content bank each time the site is built, not typed in.
Difficulty: built to a published profile
Every question is written to a target level, and every practice test follows the same plan: in each component the targets average out as close as whole levels allow to the published 2024 mean level — Listening 8.22, Reading 7.21, Writing 7.91, Speaking 7.76 (source: CELPIP Data Report of 2024 Test Takers (Paragon Research Reports, PDF)). A target is how a question is written, not a measurement. Because all our tests share one plan, we do not label any test or task set as easier or harder. Where difficulty is shown, it is this one line:
Built to the published 2024 difficulty profile · Measured: not yet measured
Measured difficulty will come from learners' answers. Once a task set has at least 30 responses from learners taking it for the first time, it will show how they did, with that sample size, and how it compares with the other sets of the same task. It will be a comparison, never a CLB level. Until a set reaches that number, it says “not yet measured”.
AI scoring is an estimate
Writing and Speaking are scored on the published performance-standard dimensions — Content/Coherence, Vocabulary, Readability, Task Fulfillment for Writing and Content/Coherence, Vocabulary, Listenability, Task Fulfillment for Speaking. Level-by-level band descriptions are not published, so the level layer is our own inference and is labelled that way on every screen: “Estimated level, based on the published CELPIP Performance Standards dimensions. Our scoring is not yet equated to the score scale the test itself uses, so read it as a range. Not a CELPIP score.”
Each response is scored in several independent passes and the median is reported as a range, for example CLB 8 (7–9). Deterministic checks — word count, time used, format — are done in code, not by a model. A wide spread is escalated to more passes; if it stays wide, the range is shown wide.
What we measure
- Difficulty deviation: median estimated level on our tests against the published mean per component.
- Scoring consistency: each response in our internal set is scored from scratch several times, each time the way the product scores it (several passes, a second model when they disagree, the median); we report the spread between the highest and lowest estimate — what you would see if you submitted the same response again.
- Raw pass spread: the same responses scored in single passes, before anything is combined; spread between the highest and lowest pass. Stricter than what you see, and published next to it so the combining step is not hiding anything. The pass count is published with the figure.
- Answer-key accuracy: share of items where an independent solver agrees with the key.
- Reported items: flags per 1,000 answered questions on the live site, how each flag ended, and how long it took (see below).
- Cohort accuracy: test takers who choose to share the level they receive after their test; we compute the mean absolute error of our estimate. This cannot be rushed — it waits for real tests.
Both scoring figures come from the same run: the same model, prompt and scoring rules as the product. So far that run was carried out in separate AI agent sessions, one per scoring pass, not through the production scoring service itself; a re-measurement through the production service is planned, and each figure says which of the two it came from.
The trust panel reads these values from the evaluation output. If a value is missing, the panel shows “not yet measured”.
Live-site figures
Some figures can only come from real use. We read them from the production database with a read-only export that returns totals only — counts and averages, never a row about a person, an account, an email address or anything a learner wrote. The date of each export is shown with the figures, which are updated when a new export is published. No figure is published from fewer than 30 of whatever it counts — tests, flags, responses or test takers; until then it says “not yet measured” with the count so far.
- Practice volume: submitted practice tests, and the Listening and Reading questions in them that were answered.
- Flag rate: questions flagged by learners per 1,000 answered questions. Published from 1,000 answered questions; until then only the counts are shown.
- Flags resolved: the share of flags that have an outcome — the answer key confirmed, the question found suspect or ambiguous, or already taken out of scoring — and the average time from flag to outcome. Each flagged question is re-solved by an independent AI solver; no person reviews it.
- Questions taken out of scoring: questions new tests no longer score, after the AI check or because the response statistics condemned them, and text corrections applied to our published tests.
- Writing and Speaking scoring: responses sent for scoring and how many were scored; the share that were delayed (for example by the daily spending limit) or failed at least once; and how many needed extra passes because the first ones disagreed.
- Cohort accuracy: computed from test takers who chose to share their result, and published only from 30 of them, as a total.
Market review, September 2026
In September 2026 we looked at active preparation products (web, iOS and Android) and read public reviews on app stores and review sites, noting prices, stated claims and complaint themes. What follows is what we took from it, in words: we do not publish counts from it, because the product-by-product record behind them is not complete. No product is named in a negative context, and quotes are anonymised. Limits: reviews are self-selected, store ratings can be gamed, and this is one point in time.
What people say, and what we do about it
“Practice was easier than the test.”
Every question is written to a target level, and each section’s targets are set to average out at the published 2024 mean. Whether the finished tests really land there is a trust-panel metric — not yet measured, and the panel says so.
“The AI score changed every time.”
Writing and Speaking scoring runs several independent passes and reports a range. We measured how far repeated scorings of the same response drift apart, and the trust panel shows it.
“Just a number, no idea what to fix.”
Your report names the one pattern holding your level down. For Writing and Speaking, feedback quotes your own sentence and shows it revised.
“The timer disappeared, the mic dropped.”
The clock never leaves the screen. Speaking moves from preparation to recording to saving on its own. The exam flow is covered by automated end-to-end tests at phone and desktop widths.
“I paid and could not use it.”
Your first full test is free, so you see the microphone and the clock work on your device before you pay anything. Quotas are printed on the pricing page.
What we do not claim
- We are not affiliated with the test provider and do not use its materials.
- Our estimate is not a CELPIP® score and cannot predict one for an individual.
- We do not sell memorised answers; the provider warns that non-original responses can void a score.
- Figures we cite from the provider or from IRCC are quoted with their source; where a link is still being verified we say so.