Mark ở vũ trụ khác · Another Universe Papers That Changed How We Learn #9 · Memory · AI in education

The AI that guesses when you will forget

Settles & Meeder · "A Trainable Spaced Repetition Model for Language Learning" · ACL 2016, Berlin

PaperBurr Settles and Brendan Meeder, ACL 2016, pp. 1848–1858
Questioncan a machine guess when a person is about to forget a word?
Data12.9 million learning traces over 2 weeks from a language-learning app
LabelSolid, moderately: good at predicting, not shown to make people learn better

Source: Settles and Meeder (2016), Proceedings of the 54th Annual Meeting of the ACL, pp. 1848–1858, §4.1 (data). The label is Mark's assessment.

One-minute recap

Essence

A 130-year-old ideaMemory fades along Ebbinghaus's forgetting curve (1885). Reviewing just before you forget makes it last longer.
A word's "half-life"Each word in your head has a half-life: after that much time, your chance of recalling it is 50%.
Learned from 12.9 million tracesThe model learns to predict from real data, with 45% less error than the old box system.
Measured who came backTrials with millions of learners measured whether they returned the next day, not test scores.
Better guesses, not proven better learningThe paper shows the machine predicts better. It does not show you learn the language better.

Source: Settles and Meeder (2016), §1 (Ebbinghaus), §3.3 (half-life), §4.1 and Table 2 (12.9 million traces, error), §4.3–4.4 (user experiments). The last card is Mark's reading of the paper itself.

Part 1

The paper's question

If a machine knew exactly when you were about to forget a word, it could remind you to review right then. This 2016 paper teaches a machine to make that guess, from the data of millions of real learners.

1885The forgetting curveEbbinghaus: memory fades over time; spreading practice out saves effort.
→
1960s–1980sHand-set schedulesPimsleur, the Leitner box: review gaps fixed in advance by people.
→
2016Learned schedulesSettles and Meeder: let data set the half-life of each word, for each person.

Source: Settles and Meeder (2016), §1 (Ebbinghaus 1885, algorithms from the 1960s–80s), §3.1–3.3.

Part 2

The story of the discovery

Who did it, where, why, and what was known before.

1885 · Ebbinghaus tests himselfAs the paper retells it: after a day of cramming he could recite 12-syllable sequences; spreading practice over 3 days took half as many repetitions for comparable results. More in Paper #1: The forgetting curve.
1967 and 1972 · The first schedulesPimsleur fixed a listening schedule: 5 seconds, 25 seconds, 2 minutes, 10 minutes, 1 hour... up to 2 years. Leitner used boxes of cards: right answers move a card to a box reviewed less often, wrong answers move it back.
2012 · A language app launchesBy the time of the paper, more than 150 million people had enrolled in a course on this app. At first it used a Leitner-like variant to show which words needed review.
Learners complainThe Leitner-based "strength" meters did not reflect what they had actually learned. That is why the team started the research.
Two weeks of dataBurr Settles (Duolingo) and Brendan Meeder collected 12.9 million "one person, one word, one session" traces and compared ways of predicting recall.
8/2016 · BerlinThe paper was presented at ACL, the main conference of computational linguistics. The team released the data and code.

Source: Settles and Meeder (2016): §1 (Ebbinghaus), §2 (150 million learners), §3.1–3.2 and Appendix A.2 (Pimsleur 1967, Leitner 1972, schedules), §3.2 (learner complaints), §4.1 (2 weeks, 12.9 million); first page (ACL, Berlin, 7–12 August 2016); repository github.com/duolingo/halflife-regression.

The Leitner box system: five boxes reviewed after 1, 2, 4, 8, 16 days. A right answer moves the card to the next box, reviewed less often. A wrong answer moves it back to a box reviewed more often. Leitner boxes: review gaps set in advance 1 day 2 days 4 days 8 days 16 days right → next box wrong → back to a busier box every word, every learner, one shared rule, even if "pantalon" is easy and "manteau" is hard
Before the paper: hand-picked rules. The boxes adapt to each card, but the gaps are still set by people: double when right, shorten when wrong. The easy and hard words are the paper's own example.Source: Settles and Meeder (2016), §3.2, Figure 3 and Appendix A.2.
Part 3

What the paper says, in 3 sentences

1. Each word has a "half-life"Recall of a word drops along Ebbinghaus's formula; after exactly one half-life, it is 50%.
2. Data sets the half-lifeInstead of hand-picked rules, the model learns from how often you have seen a word, got it right, got it wrong, and how hard the word is.
3. Better predictions, more learners returningError falls by more than 45% versus the boxes; the final version was rolled out to all learners.

Source: Settles and Meeder (2016), abstract, §3.3–3.4, Table 2, §4.4.

The half-life formula

half-life regression · p = 2^(−Δ/h)§3.3, equations (1) and (2)
Forgetting curve p equals 2 to the power minus delta over h. With a half-life h of 1 day, recall is 50 percent after 1 day and 25 percent after 2 days. With a half-life of 4 days, the curve is much flatter: recall reaches 50 percent only after 4 days. 100%50%0 01248 days h = 1 day: 50% left after 1 day h = 4 days: 50% left after 4 days p = 2^(−Δ/h): Δ is the time since the last review
Every correct review stretches the half-life. The model estimates the half-life as 2 to the power of a weighted sum (times seen, times right, times wrong, which word). The weights are not hand-picked; they are learned from 12.9 million traces. The curves are drawn from the formula, not from data.Source: Settles and Meeder (2016), §3.3, equation (1) p = 2^(−Δ/h) and (2) ĥ = 2^(Θ·x); §3.4 (features). Illustration drawn by Mark from the formula.

Result 1: predicting recall

mean absolute error · lower is betterTable 2
Mean absolute error, lower is better. Learned half-life 0.128. Constant guess at the average 0.175. Logistic regression 0.211. Leitner boxes 0.235. Pimsleur 0.445. half-life (HLR)0.128 constant guess 0.8590.175 logistic regression0.211 Leitner boxes0.235 Pimsleur0.445 mean absolute error between predicted and observed recall rate, lower is better
45% less than the boxes. But a detail few people mention: at ranking "which word is more likely forgotten" (AUC, where 0.5 is a coin flip), the new model scores 0.538 while the Leitner boxes score 0.542, the best in the table. The paper explains that the data are very noisy and 85.9% of words are recalled correctly on average.Source: Settles and Meeder (2016), Table 2 and §4.1, footnote 7.

Result 2: trials with real users

daily retention · A/B test§4.3–4.4, Table 4
ExperimentScaleCame back next day: any activityNew lessonsPractice
I. Half-life model vs Leitner boxes6 weeks, just under 1 million learners+0.3%+0.3%−7.3%
II. Lean half-life model vs full model2 weeks, 3.3 million learners+12.0%+1.7%+9.5%

Source: Settles and Meeder (2016), Table 4, §4.3 (6 weeks, "just under 1 million students"), §4.4 (2 weeks, 3.3 million). The −7.3%, +12.0%, +1.7%, +9.5% cells are statistically significant (p < 0.001).

Read this table carefully: 12% compared with what?

The abstract says the model "improved daily student engagement by 12%". That number comes from experiment II: the lean version (without word-specific features) versus the full version. Not versus the old boxes.

Versus the boxes (experiment I), the share of learners returning to practice fell by 7.3%. The authors read this as good news: many people used to practise only to "keep the tree gold"; with more accurate meters they took new lessons instead of pointless practice.

And "came back the next day" measures engagement, not a language test score.

Part 4

How this paper has been received

Each row has a source and lookup date.

ItemFinding
Venue, year54th ACL conference, Berlin, August 2016, pp. 1848–1858, peer reviewed. DOI 10.18653/v1/P16-1174.
CitationsGoogle Scholar: not checked (blocked). The aggregator liner.com shows 176 (seen on 15/10/2026, update date unknown).
Replicated?Data and code are public. A personal GitHub project reran it and found about 46% less error than Leitner, and also found Leitner ranks slightly better. No peer-reviewed replication of the user trials found.
Used as a benchmarkTabibian et al. (PNAS 2019) used this data; Ye et al. (KDD 2022) compared against models of this kind on 220 million logs from another app.
Main criticisms(1) Ranking is close to a coin flip, behind the boxes; (2) it measured coming back, not learning; (3) the 12% is versus the full model itself; (4) the authors worked for the company and tested on its own product.
AwardsNo source found, none listed.
VerdictSOLID, moderately: solid for what it measured (prediction), cautious for "helps you learn better".

Source: first page of the paper; liner.com (checked 15/10/2026); github.com/HyunsikParker/half-life (personal project, not peer reviewed); Tabibian et al., PNAS 116(10):3988–3993 (2019); Ye, Su, Cao, KDD 2022, pp. 4381–4390. Criticisms (1)–(3) come from Tables 2 and 4 of the paper itself; (4) is Mark's remark.

Part 5

Fact check: facts and myths

14 things often said about the paper and about "the app that knows when you forget". Tap each row for the reason.

SOLIDPublished at ACL 2016, pp. 1848–1858

First page of the paper and the official repository README.

SOLIDThe model builds on Ebbinghaus's forgetting curve

§3.3: "Central to the theory of memory is the Ebbinghaus model", formula p = 2^(−Δ/h).

SOLIDThe data are 12.9 million learning traces over 2 weeks

§4.1. Each trace is one person, one word, one session.

SOLIDPrediction error is over 45% lower than the Leitner boxes

Table 2: 0.128 vs 0.235. A rerun on the public data found about 46% (GitHub personal project, secondary source).

SOLIDVersus the boxes, learners returning to practice fell 7.3%

Table 4, experiment I. The authors say learners stopped practising just to "keep the tree gold".

SOLIDDropping word-specific features worked better

§4.4: word features overfitted, some words decayed very fast however much people practised; the lean version won on all 3 measures.

SOLIDSpaced practice beats cramming for retention

Meta-analysis by Cepeda et al. (2006), Psychological Bulletin: 317 experiments in 184 articles. The paper itself cites it.

SOLIDData and code are public

github.com/duolingo/halflife-regression; 361 MB data file on Harvard Dataverse.

CAUTION"The machine predicts the exact moment you forget a word"

It predicts the average recall rate well. For ranking which word is more likely forgotten, AUC is only 0.538 (0.5 is a coin flip), behind the boxes at 0.542 (Table 2).

CAUTION"The new algorithm made learners 12% more engaged than the old one"

12% is experiment II, lean versus full model. Versus the old boxes, any activity rose just 0.3% (Table 4).

CAUTION"34 hours on the app equals a university semester of Spanish"

The paper repeats this (§2). The original study by Vesselinov and Grego (2012) was commissioned by the company and not peer reviewed; only 90 of 156 people stayed to the end (TechCrunch 2013 and critiques, secondary sources).

CAUTION"Newer models are far better"

Ye et al. (KDD 2022) report 12.6% over the previous best methods on 220 million logs; the 2023 extension cuts error by 64%. But this is reported by the team behind that app, without a large independent comparison.

FALSE"The paper proves the app makes you better at languages"

The paper measured two things: prediction error and next-day return rate. There was no language test in the study.

FALSE"The app uses chatbot-style AI to guess when you forget"

The model is a regression: a few counters (seen, right, wrong) plus word features, fitted by gradient methods (§3.3–3.4, Appendix A.3). No large language model is involved.

Source: as noted in each row; full list under Sources.

Part 6

What the latest research says

2006 · The foundation holdsCepeda et al. pooled 317 experiments: spaced practice beats cramming, and the longer you need to remember, the longer the best gap between reviews.
2019 · PNAS: optimising the scheduleTabibian et al. used data from the same app and report that learners who followed the algorithm's schedule remembered better than those on rule-based schedules.
2020 · Word difficultyZaidi et al. (Cambridge): word complexity is a highly informative feature that a neural network can learn.
2022–2023 · 220 million logsYe et al. (KDD 2022, IEEE TKDE 2023) built a least-effort review schedule for a vocabulary app; they report 12.6% over earlier methods, then a 64% cut in prediction error.

Source: Cepeda et al., Psychological Bulletin 132(3):354–380 (2006); Tabibian et al., PNAS 116(10):3988–3993 (2019); Zaidi et al., arXiv 2004.11327 (2020); Ye, Su, Cao, KDD 2022, pp. 4381–4390 and IEEE TKDE 2023. Papers after 2006 were read in abstract only.

In short: machines keep getting better at predicting when you forget. Whether learners actually get better, measured by an independent test, is still rarely answered.

Mark's reading of the studies above.

Part 7

Mark scores the paper

Scores are Mark's assessment based on the sources listed, not data from a study.

Method
4/5
Real-world use
5/5
Evidence of better learning
2/5
Data transparency
5/5
Strengths
  • Real, very large, public data
  • Links an 1885 theory to machine learning
  • Tested on real users, reporting an unfavourable result too (−7.3%)
Weaknesses
  • No test of learning
  • Word ranking close to a coin flip
  • The abstract's 12% is easy to misread

An honest, useful paper: machines predict forgetting better. But "predicts well" is only halfway to "learns well".

Part 8

Lessons: use spaced-repetition apps the right way

Usable from tomorrow, with or without an app.

Review when remindedA reminder means the half-life is running out. Review then, rather than piling it all into the weekend.
Recall before flippingSay the answer in your head before you look. Tapping multiple choice too fast makes the app misjudge you.
No app: boxes 1-2-4-8Four paper boxes. Right moves a card to a less frequent box, wrong sends it back to the 1-day box. That is Leitner.
A streak is not a test scoreStreaks help you come back. Whether you really know the words needs a check outside the app.

Source: Leitner boxes from Settles and Meeder (2016), §3.2; "review before forgetting" from Cepeda et al. (2006). The habits are Mark's suggestions.

Self-check after 1 week: 3 steps

1. Write out 20 words you learned last week, hide the meanings.

2. Without the app, write each meaning and one sentence per word. Count the correct sentences.

3. Record the number. Repeat next week with 20 new words. If the number does not rise, change how you review, not just how many minutes.

Every word has a half-life. Your job is to meet it again before the half-life runs out.Mark's summary of the paper's formula, not a quotation.

Want to learn how to use AI for studying (reviewing, making cards, self-testing)? Visit Mark Học AI (markhocai.com) to read or learn.

Part 9

Further reading

Settles and Meeder (2016), "A Trainable Spaced Repetition Model for Language Learning"The original, 11 pages; §3.3 and Table 4 give the gist.
Cepeda et al. (2006), "Distributed practice in verbal recall tasks"The meta-analysis on spaced practice behind every spaced-repetition app.
Papers That Changed How We Learn #1: The forgetting curve (Ebbinghaus, 1885)The first person to measure how fast he forgot.
Giải mã siêu trí nhớ (Mark Ở Vũ Trụ Khác, book #4)Memory techniques to pair with smarter review.
Papers That Changed How We Learn #10: Does AI help or harm learningWhen students had a chatbot during practice: better homework, worse exams.
Part 10

Sources

The original paper was read in full (the PDF in the official code repository on GitHub). The working environment blocks Google Scholar, the conference site and most journal sites; sources read only through search results or the press are marked.

  1. Settles, B. and Meeder, B. (2016). "A Trainable Spaced Repetition Model for Language Learning". Proceedings of the 54th Annual Meeting of the ACL, pp. 1848–1858, Berlin. DOI 10.18653/v1/P16-1174. aclanthology.org · code and data
  2. Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., Rohrer, D. (2006). "Distributed practice in verbal recall tasks: A review and quantitative synthesis". Psychological Bulletin 132(3):354–380. PubMed
  3. Tabibian, B. et al. (2019). "Enhancing human learning via spaced repetition optimization". PNAS 116(10):3988–3993. DOI 10.1073/pnas.1815156116.
  4. Ye, J., Su, J., Cao, Y. (2022). "A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling". KDD 2022, pp. 4381–4390; extended in IEEE TKDE 2023.
  5. Zaidi, A. et al. (2020). "Adaptive Forgetting Curves for Spaced Repetition Language Learning". arXiv 2004.11327.
  6. Vesselinov, R. and Grego, J. (2012). "Duolingo effectiveness study". Technical report, Queens College, CUNY (not peer reviewed).
  7. Secondary sources: liner.com (citation count, checked 15/10/2026); github.com/HyunsikParker/half-life (rerun, personal project); TechCrunch 17/1/2013 and critiques of the 34-hour study.