Skip to main content

Research basis

Every design decision that rests on published research is listed here with the reference and where it is used in the code. A change that relies on a new study adds it to this page in the same commit. The improvement loop in ralph/ works through these items one at a time.

Each entry gives the claim we rely on and how far we rely on it. Most of these studies used other learners, other material or other models than ours, so applying them here is an inference, and the evaluation suite is how we check it.

Grounded question answering​

DecisionWhereReference
Rank retrieved captures by Okapi BM25 before packing the context, instead of packing them in time orderstudy.bm25_rank, study.answer_from (bm25-v1)Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval 3(4), 333–389. doi:10.1561/1500000019
Put the most relevant source firststudy.answer_fromLiu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P. (2024). Lost in the middle: how language models use long contexts. Transactions of the ACL 12, 157–173. doi:10.1162/tacl_a_00638
Check every [n] citation: it must point at a source that was sent, and substantive sentences should carry one; report citation_warnings and citation_coveragestudy.check_citations, study ask --jsonGao, T., Yen, H., Yu, J. and Chen, D. (2023). Enabling large language models to generate text with citations. Proceedings of EMNLP 2023, 6465–6488. doi:10.18653/v1/2023.emnlp-main.398

Learning science​

DecisionWhereReference
Review cards by testing yourself, on a spaced schedule; not by rereading notesreview.py, study methodDunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J. and Willingham, D. T. (2013). Improving students' learning with effective learning techniques. Psychological Science in the Public Interest 14(1), 4–58. doi:10.1177/1529100612453266
Spacing helps recall across many studies; the best gap grows with how long you need to rememberreview.pyCepeda, N. J., Pashler, H., Vul, E., Wixted, J. T. and Rohrer, D. (2006). Distributed practice in verbal recall tasks: a review and quantitative synthesis. Psychological Bulletin 132(3), 354–380. doi:10.1037/0033-2909.132.3.354
Schedule each card with the FSRS memory model (difficulty, stability, retrievability), pass = Good, fail = Again: FSRS-6 default parameters in spaced.py (CLI and app, target recall chosen by a workload simulation against a weekly minutes budget), FSRS-5 in review.py and the Studiospaced.py, review.pyYe, J., Su, J. and Cao, Y. (2022). A stochastic shortest path algorithm for optimizing spaced repetition scheduling. Proceedings of KDD 2022, 4381–4390. doi:10.1145/3534678.3539081. Formulas and default weights: open-spaced-repetition, "The Algorithm", FSRS-5 section (awesome-fsrs wiki).
Open each lecture's notes with three questions to answer from memory, answers after the noteslecture_kit.recall_firstRoediger, H. L. and Karpicke, J. D. (2006). Test-enhanced learning: taking memory tests improves long-term retention. Psychological Science 17(3), 249–255. doi:10.1111/j.1467-9280.2006.01693.x
A failed attempt before study still helps: ask before showing the noteslecture_kit.recall_firstRichland, L. E., Kornell, N. and Kao, L. S. (2009). The pretesting effect: do unsuccessful retrieval attempts enhance learning? Journal of Experimental Psychology: Applied 15(3), 243–257. doi:10.1037/a0016496

Evaluation​

DecisionWhereReference
Report every pass rate with a 95% Wilson score interval, not the normal interval, because our suites are small and often at 0% or 100%tests/evals/stats.py, summarize.py (ci95)Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158), 209–212. doi:10.1080/01621459.1927.10502953 · Brown, L. D., Cai, T. T. and DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science 16(2), 101–133. doi:10.1214/ss/1009213286
Compare two runs case by case with McNemar's exact test on the discordant pairstests/evals/compare_runs.py, run.sh (compare.md)McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157. doi:10.1007/BF02295996 · Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation 10(7), 1895–1923. doi:10.1162/089976698300017197
Validate the LLM judge against hand labels before trusting it as a gate; report agreement (Cohen's kappa) and intervals, not accuracy alonetests/evals/judge_bench.pyZheng, L., Chiang, W.-L., Sheng, Y. and others (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685 · Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 37–46. doi:10.1177/001316446002000104
Bootstrap interval for AUROC, seeded for reproducible reportstests/evals/stats.pyEfron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall. doi:10.1201/9780429246593
Judge benchmark includes partial claims (one true and one false atomic fact) and multi-step derived claims, so a judge that checks only the first fact, or only word overlap, is caughttests/evals/judge_pairs.jsonMin, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L. and Hajishirzi, H. (2023). FActScore: fine-grained atomic evaluation of factual precision in long form text generation. Proceedings of EMNLP 2023, 12076–12100. doi:10.18653/v1/2023.emnlp-main.741

Transcripts and slides​

DecisionWhereReference
Normalise reference and hypothesis the same way before word error rate: case, punctuation, contractions, spelled-out numbers, and only true disfluencies (uh, um, hmm) removedtranscript_ref.normalize (normalized in transcript_eval.json)Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C. and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of ICML 2023, PMLR 202, 28492–28518 (text normaliser, appendix). arXiv:2212.04356
Confirm a keyframe repeat with local structural similarity, pooled by counting changed grid cells, not by the meanlecture_kit.is_repeat, changed_cells; ADR-005; eval/keyframe_bench.pyWang, Z., Bovik, A. C., Sheikh, H. R. and Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600–612. doi:10.1109/TIP.2003.819861

Lecture Lens Studio (in-browser model)​

DecisionWhereReference
Check Studio answers' [n] citations the same way as study ask (shared fixtures pin Python and JS together), and remove [slide n] / [deck n] citations to sources the model was never given, counting them as citations_removedstudio/src/core/citations.js, notes.js, ask.js (cite-v1)Gao, T., Yen, H., Yu, J. and Chen, D. (2023). Enabling large language models to generate text with citations. Proceedings of EMNLP 2023, 6465–6488. doi:10.18653/v1/2023.emnlp-main.398
Rejected: a lexical screen that flags answer sentences whose words are absent from the sources. On the 96 labelled judge pairs its precision was about 0.5 (chance) at every threshold: wrong numbers and swapped facts reuse the source's words, correct paraphrases avoid themstudio/scripts/support_bench.mjs (kept as the record)Honovich, O., Aharoni, R., Herzig, J. and others (2022). TRUE: re-evaluating factual consistency evaluation. Proceedings of NAACL 2022, 3905–3920. doi:10.18653/v1/2022.naacl-main.287 · Maynez, J., Narayan, S., Bohnet, B. and McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. Proceedings of ACL 2020, 1906–1919. doi:10.18653/v1/2020.acl-main.173
Studio retrieval packs the best BM25 passages first and, when one is too long, keeps going for shorter ones below it instead of stopping; budgets use characters per token measured by the model's own counterstudio/src/core/ask.js, engines/budget.jsRobertson and Zaragoza (2009); Liu et al. (2024), as above
Studio notes open with the same three recall questions as the oMLX path; the vault note gets the notes without themstudio/src/core/recall.js, lecture_kit.strip_recall_firstRoediger and Karpicke (2006); Richland, Kornell and Kao (2009), as above
Spaced review in the Studio with the same FSRS-5 schedule and the same reviews.jsonl as ./lecture review, reproduced to 1e-9 on golden logs generated from Pythonstudio/src/core/review.js, Review tabYe, Su and Cao (2022); Dunlosky et al. (2013), as above
Compare two generators' outputs (Nano vs the oMLX model) case by case: a paired bootstrap interval for the mean difference and an exact sign test for numbers, Wilson intervals and McNemar for pass/fail checkstests/evals/compare_outputs.py, stats.sign_testDixon, W. J. and Mood, A. M. (1946). The statistical sign test. Journal of the American Statistical Association 41(236), 557–566. doi:10.1080/01621459.1946.10501898 · Efron and Tibshirani (1993); McNemar (1947), as above