RememberMore was a direct response to Graham Nuthall's research. A 'wicked problem' that I had the opportunity to revisited in September 2019. With the benefit of the growing interest in research-informed pedagogy, a further ten years classroom experience and a very supportive department environment, I set about a series of classroom trials that would led to the development of RememberMore. A solution more than ten years in the making.
ResearchEd Surrey: Surrey 2020
Research Informed. Experience led.
RememberMore draws heavily on Cognitive Science, specifically on the direct and indirect learning gains of Retrieval Practice and Successive Relearning (retrieval practice and spacing), Memory and Forgetting, Attention and Flow, Confidence-based assessment and Metacognition and more) and educator expertise, educator and learning feedback.
First we will review the direct and indirect benefits of retrieval practice or the 'testing effect' as we move to underline the benefits of Successive Relearning.
"Does retrieval practice improve Learner learning in school and classroom settings? Based on our literature review, our response for researchers and educators is an unequivocal 'yes.'" (Agarwal, et al, 2020)
“Retrieval practice consistently benefits Learner learning.” Agarwal, et al, (2020).
"If the eventual goal is to be able to retrieve that knowledge from memory, perhaps practicing retrieval of that information would be a better way to learn. Indeed, retrieval practice is one of the most effective ways of solidifying new knowledge, although this fact is underappreciated by most learners (and teachers)," McDermott (2020).
Retrieval practice not only enhances students’ memory performance in a future test (Roediger & Karpicke, 2006) but also helps them make more accurate predictions about future performances (Ariel & Dunlosky, 2011; King, Zechmeister, & Shaughnessy, 1980).
Adesope, Trevisan, and Sundarajan (2017) concluded in their meta-analysis that "practice tests are more beneficial for learning than restudying and all other comparison conditions," such as restudying (e.g., reviewing notes and homework) and filler activities such as resting, solving puzzles, and so forth. The average effect size was 0.61 (which is considered high to very high). These benefits persisting across a wide array of educational levels, settings, and testing formats and procedures.
These meta reviews consolidate years of research. "Retrieval practice is strongly supported by over 100 years of research and is one of only two learning techniques rated by Dunlosky et al. (2013) as having ‘high utility’ for classroom practice."
"The benefit of retrieval practice is one of the most robust findings in cognitive psychology," Roediger & Karpicke, 2006; Storm, Bjork & Storm (2010).
"Retrieval Practice alone can provide improved recall performance by as much as 10-20%. When combined with Spaced Retrieval, the effect is multiplied," Dobson, (2013).
"Practicing retrieval one time doubled long-term retention, and repeated retrieval produced a 400% improvement in retention relative to studying once." Karpicke and Roediger (2010).
“The act of retrieving information from memory actually alters the retrieved memory by elaborating on the existing memory trace and/or creating additional retrieval routes. One consequence of these changes is that the probability of successful retrieval in the future is increased, making testing a potent mechanism for enhancing long term retention.” Roediger, H. L., & Butler, A.C. (2013).
"The process of retrieval itself alters knowledge in anticipation of demands we may encounter in the future. Retrieval is therefore not only a tool for assessing learning but also a tool for enhancing learning," (Roediger & Karpicke, 2006a).
Retrieval practice strengthens our memory and long-term learning with research showing that the active process of retrieving information, improves learners complex thinking and application skills, their organisation of knowledge and their ability to transfer of knowledge to new concepts. Retrieval practice is not merely memorisation – it increases understanding.
Based on Rowland’s (2014) meta-analysis, retrieval practice effects become more robust as initial retrieval success increases, especially when initial retrieval is greater than 75%. Second, across 159 studies, the overall effect size of retrieval practice relative to repeated study was g - 0.50, and 81% of comparisons favoured retrieval practice over repeated study.
Not only does retrieval practice helps us retain learning, it help us figure out what we do know. There is a degree of dialogic thinking required when you assess your own level of confidence. This crucial benefit of retrieval practice is called metacognition, or awareness of what we know and don’t know. Add to that the know benefits of prediction, Crouch, et al., (2004).
Studies using an fMRI showed that retrieval practice, compared with re-study, involved more monitoring and working memory-related brain activity (Liu, Liang, Li, & Reder, 2014). Is this where the learning reside? This would suggest that students may benefit from practice tests prior to an exam, which can not only improve their exam performance, but also allow for better metacognitive monitoring based on their subjective experience during the practice test.
Practicing retrieval one time doubled long-term retention relative to reading the text once (34% vs 15%), and engaging in repeated retrieval increased retention to 80%.
Long-term retention after studying once, practicing retrieval once (followed by rereading), or practicing repeated retrieval. Practicing retrieval one time doubled long-term retention, and repeated retrieval produced a 400% improvement in retention relative to studying once" Karpicke and Roediger (2010).
This is particular important as learners will report re-reading as more effective - when in fact learners who use retrieval practice remember over 50% more than those learners who adopt re-reading strategies. See Karpicke and Blunt (2011).
All students were given a final test 4 days after the study session. In the intervening time, however, about half the students were asked to retrieve the studied material 2 days after the initial study session, with no instructor feedback given on this initial test.
Performance on the final test differed markedly as a function of whether a prior retrieval attempt had occurred. For essays, the students who had taken the initial test outperformed those who had not (0.38 and 0.16 recall probabilities, respectively). Similarly, for diagrams, the students taking an initial test outperformed those who had not (0.37 and 0.20 recall probabilities, respectively). This difference on a final test as a function of whether or not an initial test occurred is referred to as the testing effect, Glover (1989).
Students’ metacognitive judgements of learning (predicted recall) opposite to the pattern of students’ actual long-term retention.
The more times students repeatedly read the material, the better they believed they had learned it. However, students’ actual learning exhibited the opposite pattern. The more times students practiced actively retrieving the material, the better they retained it in the long term.
What is deeply alarming is that 93% of students students incorrectly endorsed massed study as being more effective for learning than spaced study (McCabe, 2015).
Practicing retrieval produces greater gains in meaningful learning than "Study," "Repeated Study" and elaborative studying with "Concept Mapping" across a range of testing conditions (verbatim and inference short-answer questions). What is more, this was in stark contrast to what the students predicted, Karpicke and Blunt (2011).
The two experiments reported here establish the potency of a successive relearning intervention for enhancing student learning by demonstrating meaningful improvements in course exam performance and on long-term retention tests. (Rawson et al., 2013)
Learners who used a successive relearning strategy over a 56-day period could remember more than 75% of the information after a year and 60% of the information after five years, (Janes, Dunlosky, Rawson and Jasnow, 2020).
What is more, student predict that four periods of reading and re-reading (red) would positively predict performance and that retrieval or testing (green) will be far less effective. When quite simply - the reverse is true. Simply, students may not know what is best for their learning.
Even when they do know the benefits, even when they have experienced the benefits, 58% of students, who knew better, indicated that they would not practice active retrieval.
"The effects of retrieval practice were largely the same for children with varying levels of reading comprehension and processing speed scores," (Karpicke, Blunt and Smith, 2016).
A signficant body of research focused on the benefits of the retrieval and the testing effect. Subsquentially, interest arose in the scheduling and spacing of retrieval. Here, at RememberMore we are particularly interested in optimising these learning and retention gains.
Successive Relearning (retrieval + spacing), Spacing and Scheduling
It is well known that spaced practice, in which time or other events occur between repetitions, promotes better learning and long-term retention than does massed practice, in which repetitions occur in immediate succession without intervening events. To start that review, you when need to start with the research of (Bahrick, 1979). Of particular interest here, spacing has larger and longer-lasting effects on retention when practice trials for a given item are distributed across days versus within a session (Cepeda et al., 2006; Kornell 2009).
"When you want to learn something that that's going remain with you, that you can retain for the long term, you know one exposure, you know one encounter with the information is probably going to be insufficient... one shot, one encounter with a set of materials isn't likely to promote long term retention so we know we in order to retain something well it's probably the case that we need to re-encounter, revisit, review the information again and again." Dr Sean Kang (Spacing Study Sessions Enhances Learning).
Rawson and Dunlosky (2011, 2012a) demonstrate undergraduates who successively relearned key concept definitions showed relatively high levels of retention on cued recall tests administered in the lab 1 and 4 months after relearning (up to 68% and 49% respectively, depending on the particular schedule of relearning, as compared with around 11% in a baseline control condition).
Relearning potency - is more that just a dosage effect. Three relearning sessions boosted 1-week retention to 80% even after a 3-week delay (76%). By comparison, the most effective combination of the two initial learning conditions only yielded 38% retention after 1 week.
Items correctly recalled three times during initial learning, which yielded 26% retention 1 week later. By comparison, items correctly recalled a total of three times across initial learning plus two relearning sessions; this schedule yielded 68% retention 1 week later. Thus, correctly recalling items one time in three sessions versus three times in one session yielded a 262% increase in retention test performance (d 2.17) Rawson et al., (2018).
More recently, in two applied research experiments, Successive Relearning boosted students’ learning of course content by at least 10%, Janes et al., (2020).
The advantage of successive relearning over single-session learning (i.e., relearning potency) was substantial. 50, 70, 80 markers for relearning, once, twice and thrice times, are useful if approximate. Notably, longer lasting retention or durability, 1 week, and 3 weeks later, was reported, Rawson et al., (2018).
Of greatest interest, did successive relearning enhance performance on actual course measures? "The answer is decisively yes: Successive relearning improved performance by more than a letter grade based on the grading metric used by the instructor (84% versus 72%) for course exam questions tapping practiced concepts versus baseline control concepts, Rawson et al., (2018). Importantly, a similar effect was obtained when successive relearning was unsupervised [t(22)=2.01, p=0.029, d=0.55], establishing the potency of successive relearning even when implemented under conditions with potentially lower fidelity than when administered in the lab.
Durability and efficiency: With both short-term and long-term goals in mind, Rawson and Dunlosky (2011) offer the following recommendation. Practicing to three correct recalls during initial learning, followed by three subsequent relearning sessions.
With three recalls during initial learning, learners correctly recalled 80% of the content 2 days after a second relearning session and and 77% 1 week after a third relearning session, and 57% recall after 4 – 6 weeks.
With respect to efficiency, the 3+3 schedule is preferable to schedules involving more relearning sessions as additional relearning session increased recall by only 2%.
Learning gains increased across items when quizzed once (g ! 0.444), twice (g ! 0.601) and three times or more (g ! 0.642) Unlimited attempts (g ! 0.762)
Spacing and Scheduling
Cepeda et al., (2006) research summary of 254 studies involving more than 14,000 participants found that students scored about 10% higher on tests after spaced study than after massed study.
Several studies have also reported positive outcomes from administering summative assessments that are shorter and more frequent rather than longer and less frequent (e.g., one exam per week rather than only two or three exams per semester), not only for learning outcomes but also on students’ ratings of factors such as course satisfaction and preference for more frequent testing (Dunlosky, 2013).
In the experiment (Karpicke & Bauernschmidt, 2011), students learned a list of foreign language words (e.g., Swahili vocabulary words like "mashua — boat") across cycles of study and recall trials.
Merely studying the words once without ever recalling them produced extremely poor performance (average recall was 1%). Practicing until each translation was recalled once was much better.
Massed retrieval — repeating the translations three times immediately — produced no additional gain in learning. Repeated retrieval enhanced learning only when the repetitions were spaced, and indeed, the effects of repeated spaced retrieval were very large. In a single experiment, simple changes that incorporated spaced retrieval practice took performance from nearly total forgetting to nearly 75% one week after an initial learning experience.
Another way of looking at it is that repeated retrieval with long intervals between each test produced a 200% improvement in final recall relative to repeated retrieval with three massed tests, Karpicke & Bauernschmidt (2011).
There is a wealth of research highlighting the benefits of spacing over massed cramming. Using spacing instead of cramming, can result in a 10% to 30% difference in final test results - in the long term. This finding has been found throughout a range of tasks, including remembering key words, random facts or solving maths problems.
Learning gains increased across items when quizzed once (g = 0.444), twice (g = 0.601) and three times or more (g = 0.642) Unlimited attempts (g = 0.762) Yang et al (2021). Overall, hence an effect size of g = 0.499 can be roughly translated into a change in GPA of about 0.30 – 0.35 points. This is equivalent to moving from the 50th to the 69th percentile of a normal distribution. Although g = 0.499 is only a medium-sized effect according to conventional descriptors, it is a notably large effect by the standards of educational interventions.
Where g = Hedges Effect Size. Small 0.2, Medium 0.5 and Large 0.8.
Research has shown that the longer you need to remember information, the more powerful the "Testing Effect" is and the less effective simply reading the content is.
Research has shown that there may be an optimum gap to leave depending on when you’ll need to retrieve the information. RememberMore ensures that you inform that gap.
I know you are going to ask next... What is the recommended spacing gap between study or retrieval practice sessions? "If you want to know the optimal distribution of your study time, you need to decide how long you wish to remember something." Cepeda N, Vul E, Rohrer D, et al., (2008).
A gap of between 10% and 20% of the “total time you want to retain the information” can only be given as a guide. For example: If the test is in a month, you should review the information around once a week. If the test is in a week, review the information daily.
The idea behind expanding retrieval is intuitive, and for many years it was recommended as the optimal spaced repeated retrievals schedule. Whilst expanding schedules (0-1-5-9) are far superior to massed schedules (0-0-0-0), the surprising finding in several recent studies has been that equally spaced schedules produce retention that is the same as or sometimes better than retention produced by expanding schedules , Karpicke and Roediger, (2007a).
Miyatsu et al., (2018) make two recommendation for flashcards. First, students should keep studying and testing themselves even after they get an item correct, contrary to "the conventional wisdom that once you get an item correct, you should stop studying it." Of course, RememberMore ensures that this happens. Strengthening their learners memory for the target information, and avoiding the pitfall of inaccurate metacognition (i.e., not discriminating between learned and unlearned information). Second, students should space out their studying of a given flash card. Check.
Ten Benefits of Testing
Testing in schools and in training, are usually employed diagnostically. Undertaken for purposes of assessment, to assign learners grades, or rank or both. Yet tests can serve other purposes in educational settings that greatly improve performance:
Benefit 1: The Testing Effect: Retrieval Aids Later Retention
Benefit 2: Testing Identifies Gaps in Knowledge
Benefit 3: Testing Causes Students to Learn More from the Next Study Episode
Benefit 4: Testing Produces Better Organisation of Knowledge
Benefit 5: Testing Improves Transfer of Knowledge to New Contexts
Benefit 6: Testing can Facilitate Retrieval of Material That was not Tested
Benefit 7: Testing Improves Metacognitive Monitoring
Benefit 8: Testing Prevents Interference from Prior Material when Learning New Material
Benefit 9: Testing Provides Feedback to Instructors
Benefit 10: Frequent Testing Encourages Students to Study
Roediger H, Putnam A, Smith M, et al. (2011) Ten benefits of testing and their applications to educational practice. In: Mestre J (ed.)
Why Forgetting is important
“It is natural to think that learning consists of building up knowledge or skills in our memories and that forgetting is losing some of what was built up. The relationship between learning and forgetting, however, is not so simple and is, in some respects, quite the opposite. One of the “important peculiarities” of human learning is that certain conditions that produce forgetting — that is, decrease our ability to access what we have stored in our memories — actually create opportunities to enhance our level of learning,” Bjork and Bjork (2019).
In fact, “the act of retrieving information from memory actually alters the retrieved memory by elaborating on the existing memory trace and/or creating additional retrieval routes. One consequence of these changes is that the probability of successful retrieval in the future is increased, making testing a potent mechanism for enhancing long term retention,” (Howard, 2014).
“People usually believe that forgetting happens over time; if you don’t use a memory, you lose it. This may be hard to believe, but sometimes the memory isn’t gone—it’s just hard to get to. So, more important than the passage of time or disuse is the quality of the cues you have to get to the memory.” Dan Willingham
“Just understanding the power of retrieval practice is crucial… we carry around a flawed mental model of how we learn and remember.” Robert Bjork
A knowledge of forgetting is almost as important to effective retrieval practice as a knowledge of learning. “In very short order we lose something like 70 percent of what we’ve just heard or read. After that, forgetting begins to slow… but the lesson is clear: a central challenge to improving the way we learn is finding a way to interrupt the process of forgetting,” (Brown et al., 2014, p. 28).
Attention and Flow: Simple and clean by design
RememberMore is designed and built on the fundamental principle of memory - that "memory is as thinking does." Based on the tenets of ‘flow’ and the two forms of effort:
Concentration on the task - (simple and clean)
Deliberate control of attention - (minimising distraction)
The most important design criterion for learning is that the most relevant pieces of information are attentionally focused upon and thus processed deeply. RememberMore harnesses and directs attention, first externally, then internally and finally metacognitively, whilst at the same time minimising all other distractions or competition for it.
The construction of RememberMore also utilises question cueing, permitted shorter, simpler questions, reducing error rates, improving productivity, concentrates and directs attention. Shorter questions also more accurate self-assessment.
Simply: RememberMore is optimised for learning.
(Furthermore ‘notes’ extend the learning opportunities with ‘Key Questions’ icon signposting key content or Threshold Concepts.)
"Memory is the residue of thought," Daniel Willingham (2003).
Why ask learners to successfully recall a card more than once?
"Restudying an item that one can already retrieve correctly can have enormous memory benefits." Bjork and Kornell (2008). It also address the fundamental learning flaw - "stability bias," that learners "overestimating remembering and underestimating learning."
Why ask learners to successfully recall a card after it has been learnt?
When making study decisions, it is natural to focus on items that one has struggled with, but it is important to realize that memory is anything but stable (Kornell, 2012). Items that were answered correctly can be forgotten, and items that were not answered correctly at one point may have been learned during subsequent studying.
Confidence-based assessment (self-assessment) and metacognition
Metacognition and self-regulation approaches that encourage learners to think about their own learning (often by teaching them specific strategies for planning, monitoring and evaluating their learning) have consistently high levels of impact with learners. However, as Rivers (2020) summaries,
Without support, learners lack metacognitive awareness of testing as a tool to enhance memory but do recognise that testing can be used as a monitoring tool.
Learners can accurately monitor their learning while using practice testing when judgements are made in contexts that are representative of those encountered during a criterion test.
In educational contexts, learners report using less effective strategies equally or more often than practice testing.
Learners tend to test themselves only under conditions that encourage retrieval success, and rarely use a strategy involving repeated successful retrieval even when it would lead to improved retention.
It is clear, that metacognition and Successive Relearning are intertwined.
Learners are frequently, overconfident regarding their learning status, and test performance provides diagnostic feedback to inform them about the gap between their anticipated and actual learning level (Szpunar, Jing, & Schacter, 2014), which then motivates them to expend more effort to narrow the perceived gap. In addition, frequent tests may induce high test expectancy (i.e., expecting to be tested subsequently), and test expectancy is an important motivator driving students to commit more effort to prepare for subsequent tests (Agarwal & Roediger, 2011; Szpunar, McDermott, & Roediger, 2007; Yang, Chew, Sun, & Shanks, 2019).
Confidence is the hall-mark of metacognitive judgements and the most commonly used for investigating metacognition. These predictions are usually made on a numeric confidence scale expressed as percentages; e.g. 0–20–40–60–80–100% (with the highest classification accuracies at the terminal ends of the scale) or binary (yes/no). Subjects’ predictions are good (much better than in the standard JOL procedure, in which the cue and target are presented together at study), Putnam & Roediger, (2013) as they are in RememberMore.
Confidence-based assessment is not a new idea. Underwood (1966) demonstrated that individuals predicted their own learning with considerable success, more recently Siedlecka et al., (2016) reported confidence accuracies percentages above 70%, "response accuracy always correlated with confidence." With Dougherty and colleagues (2005) reporting that both retrospective judgement about how confident they were that their answer was correct, and a Judgement of Learning about how confident they would be able to recall the target word in a later test, are both highly correlated. Finally (Chen et al., 2019) reported improved coefficients r=.655 between judgements post testing, significantly higher than the correlation for re-study trials, r=.338, t=8.773, p<.001. With cured recall, superior to multiple choice or recognition recall.
Also that JOLs decreased with increasing study time, Hoffmann-Biencourt et al. (2010).
As Rivers (2020) summaries, research shows that testing can improve both the absolute and relative accuracy of metacognitive judgements. Absolute accuracy refers to the degree to which judgement correspond with performance, whereas relative accuracy refers to the degree to which learners can discriminate between material that is better or less well learned.
Furthermore, self-testing, as promoted by RememberMore, produces higher performance and that learners’ metacognitive control becomes more effective with experience, Kornell and Son (2009). On self assessment, the two meta-analyses (Graham et al., 2015; Sanchez et al., 2017) demonstrated a positive association between self-assessment and learning, on average, “students who engaged in self-grading performed better (g = 0.34) on subsequent tests than did students who did not.”
We use it within RememberMore to encourage learners to think about what and how they are learning (metacognition) and to take advantage of the Hypercorrection Effect (errors endorsed with higher confidence are more likely to be corrected on a the final test than are errors endorsed with lower confidence) in that RememberMore always provide the retrieval prompt answer. It is also important to note that "...confidence-weighted practice tests led to greater benefits in the ability of test-takers to answer new but related questions than did standard multiple-choice practice tests," Sparck et al., (2016).
There are essentially, two types of metacognitive judgements. Both prospective and retrospective judgements correlate with actual performance accuracy (Chua et al., 2009) with confidence offering a "significant effect of the confidence judgement on final test performance." RememberMore employs prospective thinking (during pre-reveal) and Retrospective Confidence Judgements (RCJs) to adapt the Successive Relearning spacing. Learners who make RCJs prior to their restudy decisions are more accurate at identifying items in need of being restudied, nudging learners them "toward better utilizing of valid information when deciding which items are in need of further study." (Dougherty, et al., 2005; Hines, et al., 2009; Wattier & Collins, 2011; Robey, et al., 2017).
Retrospective confidence judgements (RCJs) are better predictors of memory recall than Judgements of Learning.
We know that Retrieval Practice is a powerful memory modifier. Thinking about something changes it. Together with Sadler's (2006) observation that "The act of assessing one’s own judgement of learning is one of the most effective ways to deepen a memory trace," we adopted Confidence-based assessment knowing that the connection of confidence and knowledge provides an acceleration of learning and improves student performance, (Adams and Ewen, 2009).
Also be aware that learners (younger, male and low attaining) tend to be overconfident in predicting their own learning - what is referred to as the “stability bias.” Learners tend to terminate their encoding before materials are sufficiently committed to memory and therefore 20% over learning is recommended. Note that with RememberMore, cards will always be retained and reviewed, even once "coded" as memorised. There is no advantage to "memorising" cards faster.
As Dylan William states “the best person to mark a test is the person who just took it.” (https://www.tes.com/news/memories-are-made-3)
RememberMore aligns with the evidence that strongly suggests that self-assessment is most beneficial, in terms of both achievement (performance) and self-regulated learning (progress), when it is used formatively.
We concluded that confidence may be a good marker for accuracy with cued recall. (Luna and Martín-Luengo, 2011). The Confidence-Accuracy relationship tends to be much stronger in studies of recall memory. Leading to the conclusion “that confidence may be a good marker for accuracy with cued recall.”
Research suggests we are more likely to be slightly under-confident in the knowledge we are unconfident of, and a little over-confident of the knowledge we are confident of. One of the metacogniition benefits of retrieval practice is that it improves the calibration of students’ judgments and reduce overconfidence (Hacker and Bol, 2019) and engaging in practice testing (as compared to restudying) can also improve relative monitoring accuracy.
Interesting in the workplace, Confidence-based assessment may have even greater impact.
"The fusion of knowledge and confidence that leads to behavioral outcomes and empowers people to act. People who are confidently correct will take actions that are productive," Bruno, (1995).
One key benefit of Confidence-based assessment is that a "critical factor in retrieval-based learning is initial retrieval success." Confidence-based assessment always presents the retrieval target.
Metacognitive monitoring appears to be an independent skill that can be cultivated separately from knowledge-based learning (Swanson,1990) and may have profound implications for education.
Self assessment (and self regulation)
As previously highlighted self-regulation approaches that encourage learners to think about their own learning have consistently high levels of impact with learners. The logic is that, like self-regulation, self-evaluation of the quality attributes of one’s own work draws on metacognitive competencies. It is also seen as a potential way for teachers to reduce their own assessment workload, making students more responsible for tracking their progress and feedback provision (Sadler & Good, 2006).
Self-assessment practices can be grouped into three major types: self-ratings, self-estimates of performance, and criteria or rubric based assessments. RememberMore uses confidence based assessments, a form of self-ratings. A number of studies have shown that students who engage in self-assessment experience positive gains in their learning. Most studies report positive effects of having students self-assess,the median effect lies between .40 and .45, a moderate effect consistent with values reported in Black and Wiliam (1998).
Interleaving simple mixes topics instead of focusing or blocking the same topic. This is a long-term strategy. It is more effective when those topics are related.
Rohrer and Taylor (2007) reported greater performance for those students who learned through interleaving (63% accuracy) compared to blocking (20% accuracy).
Both spacing and interleaving strategies lead to slower and more error-prone learning in the short-term however, they boost memory retention when implemented over a longer time span. But this is also one of the reasons students, in spite on being told their test scores, still report they prefer that massed, re-reading type strategies.
Using these two strategies RememberMore enhances inductive learning (making patterns and connections) and later memory retrieval. Spacing benefits memory recall whilst interleaving enhances inductive learning and later memory retrieval.
Butler et al., (2017) found that interleaving or viability applied to retrieval practice with "superior transfer of knowledge to new examples," suggesting that repeatedly retrieving and applying knowledge to different examples is a powerful method for acquiring knowledge that will transfer to a variety of new contexts.
Recent research shows that the mnemonic benefits of taking a test are not limited to the specific questions or facts that were tested; retrieval practice also improves transfer of knowledge to new contexts.
Motivation and Improved Attendance
Numerous studies have provided supporting evidence for the motivation explanation. Schrank (2016) found that class quizzes increase attendance; Heiner, Banet, and Wieman (2014) reported that a preannounced quiz encourages students to read the assigned textbook material and prepare better before class; Yang et al. (2017) showed that frequent tests drive learners to allocate more time to learning; Szpunar et al. (2013) observed that learners make more notes when they are frequently tested; Jing et al. (2016) found that frequent tests reduce task-unrelated thoughts (i.e., mind wandering) while watching lecture videos; and Weinstein et al. (2014) found that frequent tests induce high test expectancy which in turn boosts test performance.
The concept of elaboration among cognitive psychologists is broad and can mean a lot of different things, however elaborative interrogation is essentially asking ‘how’ and ‘why’ questions, elaborating on the opening question. It is constructing meaning of the subject matter by explaining, elaborating, making inferences, connecting new facts and ideas to prior knowledge.
The purpose of elaboration is primarily to improve our memory for the new information. An additional benefit is that it begins to organize our knowledge in a more coherent way.
In one of the largest and most comprehensive meta analysis of undergraduate STEM education published, Freeman et al., (2014) reported an average examination scores improved by about 6% in "active learning sections," (elaboration) and failure rates reduced from 34% to 22% where elaboration was established practice.
Across several instructional conditions and settings, Nesbit and Adesope (2006) found the use of concept maps was associated with increased knowledge retention. With (Butler, Godbole, & Marsh, 2013) show further elaborating on the correct answer in the feedback can be important for later application of that knowledge. In fact, activation of elaborative information which would occur to a greater extent during retrieval practice over restudying — i is one mechanism that is used to explain the broader concept of and effectiveness of Successive Learning.
Before we go on, not all elaboration techniques are equal.
Elaborative information tends to increases performance only when it helped specify the relevance of "target information." In this situation, RememberMore tags / cues provide excellent opportunities for elaboration as does making connections between the cues, between prompts or any presented information within RememberMore, particularly with Classroom. RememberMore "Notes" add yet a further layer.
Equally important is giving learners the opportunity to explain the prompts themselves. Doing so encourages the learners to infer the missing information, to synthesize the presented information. Those learners encouraged to self-explain outperformed those who read and reread material (Griffin, Wiley, & Thiede, 2008).
The Potentiating Effect
Most educators would probably assume that giving students a test on material before they had learned it would have little impact on student learning beyond providing teachers with insight into their students’ knowledge base. You may be surprised to learn of potentiating impact of pretesting. Simply, pretesting has the potential to increase the efficiency and utility of future learning that follows. Research suggests that tests can be valuable learning events, even if learners cannot answer test questions correctly.
Wissman, Rawson, and Pyc (2011) reported that retrieval practice of one set of material may facilitate learning of later material, which may be related or unrelated.
Similarly Karpicke (2009) reported the benefits of active retrieval improves students’ encoding when they restudy material.
"Pretrieval" practice - asking questions before teaching has a beneficial effect for learning. Most often attributed to learners being "primed" or "selectively attending" to pre-tested material - during the taught phase.
"Studies have shown that prequestions - asking students questions before they learn something - benefit memory retention." (Carpenter, Rahman and Perkins, 2018)
Cues or tags
The diagnostic value of retrieval cues — the degree to which cues help people recover particular target knowledge to the exclusion of competing candidates — is the critical factor for all learning.
Retrieval may enhance learning because it improves the match between a cue and particular desired knowledge, or it may enhance learning by constraining the size of the search set— the set of potentially recoverable candidates that comes to mind in the context of a cue (Karpicke & Blunt, 2011b; Karpicke & Zaromb, 2010).
Carpenter (2009) items recalled from weak cues were retained better over time, such that this advantage was eliminated or reversed at the time of the final test. Of note, is that diminishing cues conditions consistently outperformed the study-only condition and the accumulating cue condition, Finley et al. (2011). RememberMore is currently exploring how best to apply this empirical research.
Simply, the design of RememberMore promotes efficient prompt design with retrievals tags enabling shortened retrieval prompts, reducing error rates. Furthermore, tags then act as a knowledge "filter," learners now free-recall from a redacted knowledge base, increasing retrieval rates.
Pre-sights, feedback and rewards
Learning pre-sights provide the learner feed-forward about what they are about to review. Insights provide the learner with immediate feedback about the information they have just reviewed and deeper insights provide feedback and progress over time.
"Retrieval practice is often effective even without feedback (i.e. giving the correct answer), but feedback enhances the benefits of testing," Roediger and Butler, (2011).
Kornell and Son (2009) found that across experience with multiple study-test cycles, learners learned to self-test themselves more and did so at a faster rate when there was feedback on the tests at the end of each cycle.
Finally, rewards. Learners’ mnemonic performance is enhanced when incentives are present. RememberMore employs rewards, carefully scheduled and carefully constructed to complement each other and to deter ‘learner retirement.’ Also note that learners are more responsive to increased costs than increased benefits.
Social comparisons can also incentivize learners at recall with Fraundorf and Benjamin (2015) suggesting that learners are sensitive to how their experience with materials compares with others’.
Reducing test anxiety
Test anxiety is quite common. This nervousness can negatively affect student performance on those tests and can even impact their feelings about school and education. Retrieval practice had quite the positive effect on test anxiety experienced by middle and high school students. Data collected from 1408 students between the ages of 11-18 showed 72% of students reported retrieval practice made them less nervous for tests and exams, 81% of students reported they felt the same amount of test anxiety or less in the class that used retrieval practice relative to their other classes.
“This finding suggests that experiencing retrieval practice makes students less anxious regarding upcoming tests and exams for classes in which retrieval practice was implemented." Agarwal et al. (2014). Similarly Sullivan, (2017) found that percentage elevated to 90% for College students.
Furthermore, in another study, it was shown the retrieval practice was “had the most potential to create memories that were resilient to stress.,” Whereby retrieval practice appeared to create more distinct routes to the information to be retrieved, even if cortisol impairs one or more pathway, another may still be viable for retrieval.
Back to those re-reading strategies. Increased stress made re-reading strategies even more vulnerable. Studies showed that these learners experienced even great performance impairment. Up to 32% worse.
Students enjoy Retrieval Practice
In perhaps the most extensive study of this effect, Agarwal et al. (2014) surveyed 1408 middle and high school students who had experienced classroom-based retrieval practice programs (e.g., like the one implemented by McDaniel et al., 2011). The key findings from their survey were that 92% of students viewed the classroom retrieval practice activities positively, believing that retrieval practice helped them learn, and 72% said that frequent retrieval practice helped them feel less nervous about classroom exams.
"Students perceived retrieval practice as helpful for retaining and retrieving information over time, for monitoring their comprehension, and for cueing important information for subsequent restudy," Cogliano, et al., (2018).
This concurs with what learners tell us about learning with Classroom and to a greater degree RememberMore.
If "Successive Relearning" aids retention, expectation of any kind of test, enhances the processing of studied material. is an important motivator driving students to commit more effort to prepare for subsequent tests (Agarwal & Roediger, 2011; Szpunar, McDermott, & Roediger, 2007; Yang, Chew, Sun, & Shanks, 2019). Simple, test performance is higher when students expect a final cumulative test compared to when they do not, even when the students are not provided with an opportunity to restudy the material, Szpunar, McDermott, and Roediger (2007). Why? Two theories are presented: By reducing learners’ mind-wandering during studying and second, by reducing interference from previously studied information Weinstein et al., (2014).
Other researchers have similarly found increased performance for learners expecting either a multiple-choice or essay test (compared to students with no test expectation). The research also suggests that telling students the type of test they should expect (e.g., multiple-choice or essay) and the target learning goal (e.g., memory, comprehension, or application) can benefit their learning.
Expecting an upcoming test can be helpful to learners, actually gaining experience with the test leads, to the greatest adoption of optimal learning strategies, Colak, (2015).
Low stakes - 'the poker chip'
The more with teach with both classroom.remembermore.app and RememberMore, the more I am convinced that success drives motivation. That in turn, this motivation then drives success. The success-motivation-success cycle. What is more, I am convinced that the RememberMore approach protect learners (sometimes) fragile self-confidence. Here is how.
"If class was a poke game and you give each learner only one poker chip. Then each time you ask a question of the class you are asking learners to go all-in. Unsurprisingly, many learners choose to hold on to their chip, even those who are relatively confident, and look down. A few will risk, though only a few." -- My thanks to Chris Smith for this!
With low-stakes RememberMore, learners get both the prompt and the desired response, and they get to hold onto their chip. The knowledge become "sticky," they learn, and eventually remember, more. With a growing security, a deepening knowledge, their willingness to risk that chip become more and more likely.
The importance of prior knowledge for learning
It is not a question of whether or not prior knowledge is important for learning, it is a question of how important it is, and how important it is in facilitating the acquisition and consolidation of new learning. It is "all the knowledge one has before learning about a particular topic."
Here in the UK, setting aside data that describe the socio-economic circumstances of children, prior attainment at the end of primary school is found to be the most powerful available predictor of Secondary School attainment.
What about in classrooms? Prior knowledge has long been considered the most important factor influencing learning and learner achievement, with both "the amount and quality of prior knowledge positively influencing both knowledge acquisition and the capacity to apply higher-order cognitive problem-solving skills," Hailikari et al., (2008); Dochy et al., (1999); Chen, et al., (2018). With Dochy et al. (1999) go as estimating that between 30-60% of the variance in learning outcomes is explained by prior knowledge. Finally, given our focus on Successive Relearning, Van Kesteren, et al., (2018) demonstrated how prior knowledge structures are suggested to facilitate many stages of memory processing such as encoding, consolidation, and retrieval and reduce the interference of competing memories.
Cognitive research continues to uphold the importance of content knowledge in understanding academic material and the world around us. It is the foundation that enables us to make sense of what we learn. A seminal study by Recht and Leslie (1988) has shown that content knowledge is a better predictor of a learner's understanding of a text than reading ability; learners who are familiar with the relevant content of an article, understand it better than do their peers who are presumed better readers.
In the broader conversation of a "modern" education, we are often reminded that our learners future success, within an increasingly automated, artificially intelligent world, will demand more non-routine, creative thinking and refined reasoning skills than said declarative or substantive knowledge. However, this proscribed dichotomy is arguably misleading: with what do we make connections or challenge assumptions, if not with the thoughts, ideas and knowledge stored in our memory.
Value knowledge. Define what needs to be learnt, design the learning, and signpost exits routes from it. Without question, an essential factor in developing an integrated knowledge framework is aligning the educators expectations of learner prior knowledge, the learners actual knowledge base and what is to be learnt. Defining the parameters for substantive or declarative knowledge is important for both the educator and learner. Challenge your learners in your classes to acquire it, to use it and share it. And when they do, notice it and celebrate it. This is no easy task, though ignoring or undermining the importance of content knowledge in the hope that it will be assimilated is not a fair strategy either.
Vocabulary and simple definition acquisition is possibly one of the most obvious use cases for RememberMore (all the memorable when mixed with other content and knowledge). Why is vocabulary acquisition so important? Word knowledge is crucial to reading comprehension and determines how well students will be able to comprehend the texts they read. Now I accept that comprehension is far more than recognising words and remembering their meanings, however, if a student does not know the meanings of a sufficient proportion of the words in the text, comprehension is impossible.
Vocabulary experts agree that adequate reading comprehension depends on a person already knowing between 90 and 95 percent of the words in a text (Hirsch, 2003). Knowing at least 90 percent of the words enables the reader to get the main idea from the reading and guess correctly what many of the unfamiliar words mean, which will help them learn new words. Readers who do not recognise at least 90 percent of the words will not only have difficulty comprehending the text, but they will miss out on the opportunity to learn new words.
RememberMore supports revision
We foresee RememberMore being adopted as a revision tool. How might RememberMore support effective revision practice in addition to adaptive, spaced retrieval? RememberMore:
is mobile, accessible, encourages learners to build retrieval / revision into their practice.
tags break down learning into manageable (and potentially, interleaved) chunks
enable revision as a moments notice, offer the opportunity for shorter, more frequent, revisions periods
Calibration has been deﬁned as the degree of fit between a person’s judgement of performance and his or her actual performance (Keren, 1991). As such, calibration reﬂects a metacognitive monitoring process that provides information about the status of one’s knowledge or strategies at a cognitive level. Based on this information, control at a metacognitive level can be exerted to regulate one’s knowledge or strategies (though often this is under utlised by learners). Therefore, greater accuracy in a person’s judgments of performance, being well calibrated, creates greater potential for self-regulation.
Incentives to be accurate, significantly improved calibration accuracy only among lower-achieving students, Schraw et al. (1993).
Cognitive Load Cognitive Load Theory was recently described by British educationalist Dylan Wiliam as ‘the single most important thing for teachers to know,’ (Wiliam 2017). Cognitive load theory is built upon two commonly accepted ideas. The first, that there is a limit to how much new information the human brain can process at one time. The second, there are no known limits to how much stored information can be processed at one time.
"The implications of working memory limitations on instructional design can hardly be overestimated... Anything beyond the simplest cognitive activities appear to overwhelm working memory. Prima facie, any instructional design that flouts or merely ignores working memory limitations inevitably is deficient," Sweller, van Merrienboer & Paas (1998).
The aim of cognitive load research is therefore to develop instructional techniques and recommendations that fit within the characteristics of working memory, in order to maximise learning. Cognitive Load Theory supports explicit models of instruction, because such models tend to accord with how human brains learn most effectively (Kirschner, Sweller & Clark 2006). Explicit instruction involves teachers clearly showing students what to do and how to do it (rather than having learners discovering or constructing information for themselves). It will come as no surprise then, then RememberMore is built, or at least designed with Cognitive Load Theory very much in mind.
For the most part, my interest in Retrieval Practice and Successive Relearning has been from a Cognitive Science and behavioural perspective that has long emphasised the role of retrieval and in memory updating, however more recently the neural mechanisms and basis behind these process or processes are currently being explored by Neuroscience researchers.