Cognitive Science

Testing in schools and in training, are usually employed diagnostically. Undertaken for purposes of assessment, to assign learners grades, or rank or both. Yet tests can serve other purposes in educational settings that greatly improve performance:

Benefit 1: The Testing Effect: Retrieval Aids Later Retention

Benefit 2: Testing Identifies Gaps in Knowledge

Benefit 3: Testing Causes Students to Learn More from the Next Study Episode

Benefit 4: Testing Produces Better Organisation of Knowledge

Benefit 5: Testing Improves Transfer of Knowledge to New Contexts

Benefit 6: Testing can Facilitate Retrieval of Material That was not Tested

Benefit 7: Testing Improves Metacognitive Monitoring

Benefit 8: Testing Prevents Interference from Prior Material when Learning New Material

Benefit 9: Testing Provides Feedback to Instructors

Benefit 10: Frequent Testing Encourages Students to Study

Roediger H, Putnam A, Smith M, et al. (2011) Ten benefits of testing and their applications to educational practice. In: Mestre J (ed.)

“It is natural to think that learning consists of building up knowledge or skills in our memories and that forgetting is losing some of what was built up. The relationship between learning and forgetting, however, is not so simple and is, in some respects, quite the opposite. One of the “important peculiarities” of human learning is that certain conditions that produce forgetting — that is, decrease our ability to access what we have stored in our memories — actually create opportunities to enhance our level of learning,” Bjork and Bjork (2019).

In fact, “the act of retrieving information from memory actually alters the retrieved memory by elaborating on the existing memory trace and/or creating additional retrieval routes. One consequence of these changes is that the probability of successful retrieval in the future is increased, making testing a potent mechanism for enhancing long term retention,” (Howard, 2014).

“People usually believe that forgetting happens over time; if you don’t use a memory, you lose it. This may be hard to believe, but sometimes the memory isn’t gone—it’s just hard to get to. So, more important than the passage of time or disuse is the quality of the cues you have to get to the memory.” Dan Willingham

“Just understanding the power of retrieval practice is crucial… we carry around a flawed mental model of how we learn and remember.” Robert Bjork

A knowledge of forgetting is almost as important to effective retrieval practice as a knowledge of learning. “In very short order we lose something like 70 percent of what we’ve just heard or read. After that, forgetting begins to slow… but the lesson is clear: a central challenge to improving the way we learn is finding a way to interrupt the process of forgetting,” (Brown et al., 2014, p. 28).

Numerous studies have provided supporting evidence for the motivation explanation. Schrank (2016) found that class quizzes increase attendance; Heiner, Banet, and Wieman (2014) reported that a preannounced quiz encourages students to read the assigned textbook material and prepare better before class; Yang et al. (2017) showed that frequent tests drive learners to allocate more time to learning; Szpunar et al. (2013) observed that learners make more notes when they are frequently tested; Jing et al. (2016) found that frequent tests reduce task-unrelated thoughts (i.e., mind wandering) while watching lecture videos; and Weinstein et al. (2014) found that frequent tests induce high test expectancy which in turn boosts test performance.

"The impact of achievement on self-concept is greater than the impact of self-concept on achievement."

"Lack of motivation is a logical response to repeated failure." Caroline Spalding

Pekrun et al., (2017) reported a "reciprocal effects model of emotion and achievement," where positive emotions (enjoyment and pride) positively predicted their subsequent end-of-the- year math grades, and grades, in turn, positively predicted the development of positive emotions. Where Math-related negative emotions (anger, anxiety, shame, hopelessness, and boredom) were negative predictors of subsequent math grades, and grades, in turn, were a negative predictor for the development of negative emotions.

These findings were consistent for seven discrete emotions, four time intervals, two different measures of achievement (grades, test scores), while controlling for students’ gender, intelligence, and critical demographic background variables.

Remember, "emotions indeed have an influence on adolescents’ achievement, over and above the effects of general cognitive ability and prior accomplishments," Pekrun et al., (2017).

These findings go beyond correlational evidence for achievement and emotion, disentangling the directional effects underlying the emotion-achievement link. It is why 'Reorder' is such an important feature in RememberMore. With "success is expected to generally increase perceived control, thus enhancing positive emotions, and failure is expected to decrease control, leading to negative emotions." Pekrun, et al., (2017).

Emotions have effects on "adolescent" students’ academic achievement and that these effects are not merely a by-product of prior performance. More likely, they represent a true causal influence of students’ emotion experiences. Therefore: who do we strengthen adolescents’ positive emotions (and minimize their negative emotions).

Pekrun, et al., (2017) outline that the results imply reciprocity. Providing students with opportunities to experience success and mastery over competition goals may help to promote positive emotions (and prevent negative emotions) Pekrun, et al., (2014) .

Kriegbaum, et al., (2018) review the relative importance of intelligence and motivation as predictors of school achievement: A meta-analysis of 74 studies (N = 80,145). They reported average correlations between intelligence (r = 0.44) and motivation (r = 0.27) with school achievement and between intelligence and motivation (r = 0.17). A path model showed that 24% of variance in school achievement was explained overall, of this overall explained variance in school achievement, 66.6% was uniquely explained by intelligence and 16.6% uniquely by motivation. Thus, the results show that both intelligence and motivation contribute substantially to the prediction of school achievement.

On success-rates with RememberMore look to keep success rates. When assessing students’ individual work and oral responses to class discussions, Rosenshine found that the most effective fourth-grade math teachers had a student success rate of 82%. On the other hand, the least effective math teachers only had success rates of 73%. So aim for the upper end and above of this band and make use of the 'Reorder' feature in Classroom.

RememberMore is designed and built on the fundamental principle of memory - that "memory is as thinking does." Based on the tenets of ‘flow’ and the two forms of effort:

• Concentration on the task - (simple and clean)

• Deliberate control of attention - (minimising distraction)

The most important design criterion for learning is that the most relevant pieces of information are attentionally focused upon and thus processed deeply. RememberMore harnesses and directs attention, first externally, then internally and finally metacognitively, whilst at the same time minimising all other distractions or competition for it.

The construction of RememberMore also utilises question cueing, permitted shorter, simpler questions, reducing error rates, improving productivity, concentrates and directs attention. Shorter questions also more accurate self-assessment.

Simply: RememberMore is optimised for learning.

(Furthermore ‘notes’ extend the learning opportunities with ‘Key Questions’ icon signposting key content or Threshold Concepts.)

"Memory is the residue of thought," Daniel Willingham (2003).

Why ask learners to successfully recall a card more than once?

"Restudying an item that one can already retrieve correctly can have enormous memory benefits." Bjork and Kornell (2008). It also address the fundamental learning flaw - "stability bias," that learners "overestimating remembering and underestimating learning."

Why ask learners to successfully recall a card after it has been learnt?

When making study decisions, it is natural to focus on items that one has struggled with, but it is important to realize that memory is anything but stable (Kornell, 2012). Items that were answered correctly can be forgotten, and items that were not answered correctly at one point may have been learned during subsequent studying.

Cognitive load theory (CLT) is an instructional theory based on human cognitive architecture and evolutionary psychology (Sweller et al., 2011; Sweller et al., 2019). It can be described by five fundamental principles (Sweller & Sweller, 2006).

Cognitive Load Cognitive Load Theory was recently described by British educationalist Dylan Wiliam as the single most important thing for teachers to know,’ (Wiliam 2017).

Cognitive load theory is built upon two commonly accepted ideas. The first, that there is a limit to how much new information the human brain can process at one time. The second, there are no known limits to how much stored information can be processed at one time.

"The implications of working memory limitations on instructional design can hardly be overestimated... Anything beyond the simplest cognitive activities appear to overwhelm working memory. Prima facie, any instructional design that flouts or merely ignores working memory limitations inevitably is deficient," Sweller, van Merrienboer & Paas (1998).

Most cognitive load effects are caused by an instructional procedure that overwhelms working memory during learning. Learning is enhanced when compared with an alternative instructional procedure that reduces working memory load. For example, presenting information in a split-attention format requires the use of more working memory resources than presenting the same material in a physically integrated format, leading to the split-attention effect. If intrinsic cognitive load is low because element interactivity is low, the effect will not be obtained because even under split-attention conditions, total cognitive load does not exceed working memory resources.

The aim of cognitive load research is therefore to develop instructional techniques and recommendations that fit within the characteristics of working memory, in order to maximise learning. Cognitive Load Theory supports explicit models of instruction, because such models tend to accord with how human brains learn most effectively (Kirschner, Sweller & Clark 2006).

Explicit instruction involves teachers clearly showing students what to do and how to do it (rather than having learners discovering or constructing information for themselves). It will come as no surprise then, that RememberMore is built, or at least designed with Cognitive Load Theory very much in mind, minimising Split Attention and Redundancy Effects.

Metacognition and self-regulation approaches that encourage learners to think about their own learning (often by teaching them specific strategies for planning, monitoring and evaluating their learning) have consistently high levels of impact with learners. However, as Rivers (2020) summarises,

"Without support, learners lack metacognitive awareness of testing as a tool to enhance memory but do recognise that testing can be used as a monitoring tool."

Learners can accurately monitor their learning while using practice testing when judgements are made in contexts that are representative of those encountered during a criterion test. That this accuracy improves as learners get older.

In educational contexts, learners report using less effective strategies equally or more often than practice testing. Learners tend to test themselves only under conditions that encourage retrieval success, and rarely use a strategy involving repeated successful retrieval even when it would lead to improved retention.

Testing can improve both the absolute and relative accuracy of metacognitive judgements. Absolute accuracy refers to the degree to which judgement corresponds with performance, whereas relative accuracy refers to the degree to which learners can discriminate between material that is better or less well learned. This was explored using RememberMore and reported under my Our Story.

Couchman et al., (2014) Real-time metacognitive monitoring – measured by confidence ratings for each individual question – accurately predicted performance and were a much better decisional guide than retrospective judgements.

It is clear that metacognition and Successive Relearning are intertwined.

Learners are frequently, overconfident regarding their learning status, and test performance provides diagnostic feedback to inform them about the gap between their anticipated and actual learning level (Szpunar, Jing, & Schacter, 2014), which then motivates them to expend more effort to narrow the perceived gap. In addition, frequent tests may induce high test expectancy (i.e., expecting to be tested subsequently), and test expectancy is an important motivator driving students to commit more effort to prepare for subsequent tests (Agarwal & Roediger, 2011; Szpunar, McDermott, & Roediger, 2007; Yang, Chew, Sun, & Shanks, 2019).

Confidence is the hall-mark of metacognitive judgements and the most commonly used for investigating metacognition. These predictions are usually made on a numeric confidence scale expressed as percentages; e.g. 0–20–40–60–80–100% (with the highest classification accuracies at the terminal ends of the scale) or binary (yes/no). Subjects’ predictions are good (much better than in the standard JOL procedure, in which the cue and target are presented together at study), Putnam & Roediger, (2013) as they are in RememberMore.

Confidence-based assessment is not a new idea. Underwood (1966) demonstrated that individuals predicted their own learning with considerable success, more recently Siedlecka et al., (2016) reported confidence accuracy percentages above 70%, "response accuracy always correlated with confidence." With Dougherty and colleagues (2005) reporting that both retrospective judgement about how confident they were that their answer was correct, and a Judgement of Learning about how confident they would be able to recall the target word in a later test, are both highly correlated. Finally (Chen et al., 2019) reported improved coefficients r=.655 between judgements post testing, significantly higher than the correlation for re-study trials, r=.338, t=8.773, p<.001. With cured recall, superior to multiple choice or recognition recall.

Also that JOLs decreased with increasing study time, Hoffmann-Biencourt et al., (2010).

Furthermore, self-testing, as promoted by RememberMore, produces higher performance and that learners’ metacognitive control becomes more effective with experience, Kornell and Son (2009). On self assessment, the two meta-analyses (Graham et al., 2015; Sanchez et al., 2017) demonstrated a positive association between self-assessment and learning, on average, “students who engaged in self-grading performed better (g = 0.34) on subsequent tests than did students who did not.”

Four reasons to encourage self-assessment with RememberMore

• Increases learning (fills gaps in knowledge) and academic performance (improved accuracy of performance) and transfer to new questions

• Improve self-regulation (of which learning strategy are effective and which to employ)

• Enhance self-efficacy and agency (motivation-success-motivation)

• Gains can be achieved without supervision (RememberMore is personalised and adaptive)

We prompt self-assessment with RememberMore to encourage learners to think about what and how they are learning (meta-cognition) and to take advantage of the Hypercorrection Effect (errors endorsed with higher confidence are more likely to be corrected on a the final test than are errors endorsed with lower confidence) in that RememberMore always provide the retrieval prompt answer. It is also important to note that "...confidence-weighted practice tests led to greater benefits in the ability of test-takers to answer new but related questions than did standard multiple-choice practice tests," Sparck et al., (2016).

There are essentially two types of metacognitive judgements. Both prospective and retrospective judgements correlate with actual performance accuracy (Chua et al., 2009) with confidence offering a "significant effect of the confidence judgement on final test performance." RememberMore employs prospective thinking (during pre-reveal) and Retrospective Confidence Judgements (RCJs) to adapt the Successive Relearning spacing. Learners who make RCJs prior to their restudy decisions are more accurate at identifying items in need of being restudied, nudging learners "toward better utilizing of valid information when deciding which items are in need of further study." (Dougherty, et al., 2005; Hines, et al., 2009; Wattier & Collins, 2011; Robey, et al., 2017).

Retrospective confidence judgements (RCJs) are better predictors of memory recall than Judgements of Learning.

We know that retrieval practice is a powerful memory modifier. Thinking about something changes it. Together with Sadler's (2006) observation that "The act of assessing one’s own judgement of learning is one of the most effective ways to deepen a memory trace," we adopted Confidence-based assessment knowing that the connection of confidence and knowledge provides an acceleration of learning and improves student performance, (Adams and Ewen, 2009).

Also be aware that learners (younger, male and low attaining) tend to be overconfident in predicting their own learning - what is referred to as the “stability bias.” Learners tend to terminate their encoding before materials are sufficiently committed to memory and therefore 20% over learning is recommended. Note that with RememberMore, cards will always be retained and reviewed, even once "coded" as memorised. There is no advantage to "memorising" cards faster.

As Dylan William states “the best person to mark a test is the person who just took it.” (https://www.tes.com/news/memories-are-made-3)

RememberMore aligns with the evidence that strongly suggests that self-assessment is most beneficial, in terms of both achievement (performance) and self-regulated learning (progress), when it is used formatively.

We concluded that confidence may be a good marker for accuracy with cued recall. (Luna and Martín-Luengo, 2011). The Confidence-Accuracy relationship tends to be much stronger in studies of recall memory. Leading to the conclusion “that confidence may be a good marker for accuracy with cued recall.”

Research suggests we are more likely to be slightly under-confident in the knowledge we are unconfident of, and a little over-confident of the knowledge we are confident of. One of the metacognition benefits of retrieval practice is that it improves the calibration of students’ judgments and reduces overconfidence (Hacker and Bol, 2019) and engaging in practice testing (as compared to restudying) can also improve relative monitoring accuracy.

Interestingly, in the workplace, Confidence-based assessment may have an even greater impact.

"The fusion of knowledge and confidence that leads to behavioral outcomes and empowers people to act. People who are confidently correct will take actions that are productive," Bruno, (1995).

One key benefit of Confidence-based assessment is that a "critical factor in retrieval-based learning is initial retrieval success." Confidence-based assessment always presents the retrieval target.

Metacognitive monitoring appears to be an independent skill that can be cultivated separately from knowledge-based learning (Swanson, 1990) and may have profound implications for education.

Barenberg and Dutke (2019) study examined the potential of retrieval practise during learning to improve the accuracy of confidence judgements in future retrieval. In the final test, the proportion of correct answers and the proportion of confident answers were higher with retrieval practice than compared to the control condition. Moreover, the confidence judgments were more accurate and less biased.

Barenberg and Dutke (2019) conclude that the confidence judgments can "stimulate the learners to reflect their understanding of learning topics and the quality of knowledge they acquired." What is more, that this reflection can help them to identify learning topics that need further clarification and help them to develop the accuracy of their confidence judgments.

Judgments of Knowing, Learning, and Memory - Do I Know It. As most other metacognitive judgments, feelings of knowing have behavioral consequences and predict how much time and effort people will - or will not invest in a memory search (Dunlosky & Metcalfe, 2009; Koriat, 2007). Judgments of Learning, Will I Remember It? Are moderately predictive of people’s actual performance Dunlosky & Metcalfe, (2009). RememberMore takes the positive elements of each, trusting the learning with both the cue and the response.

An interesting area of study is combines response latency and confidence. Ackerman and Koriat (2011) report "When the test format was easy (Experiment 1), 2nd graders were as accurate as 5th graders in monitoring the accuracy of their answers, and the latency of their responses was no less predictive of accuracy. When the task was more difficult, age differences emerged. Nevertheless, in all experiments and for both age groups, response latency was found to have added value for predicting accuracy over and above that of confidence.

How reliable is teacher assessment?

"Over 40 years of empirical research shows that teachers' ability to estimate student learning is rather poor (Urhahne & Wijnia, 2021), Nevertheless, it can be enriched by the learners' own ability to estimate their learning gains."

“Students’ predictions were almost always higher than the grade they earned and this was particularly true for low-performing students,” Miller and Geraci (2011). Develop metacognitive accuracy to improve academic outcomes. Thus, there may be good pedagogical reasons to ask students to try to accurately predict their exam performance. In addition, a somewhat unexpected positive outcome from the their research was that students actually reported enjoying predicting their grades.

As previously highlighted self-regulation approaches that encourage learners to think about their own learning have consistently high levels of impact with learners. The logic is that, like self-regulation, self-evaluation of the quality attributes of one’s own work draws on metacognitive competencies. It is also seen as a potential way for teachers to reduce their own assessment workload, making students more responsible for tracking their progress and feedback provision (Sadler & Good, 2006).

Self-assessment practices can be grouped into three major types: self-ratings, self-estimates of performance, and criteria or rubric based assessments. RememberMore uses confidence based assessments, a form of self-ratings. A number of studies have shown that students who engage in self-assessment experience positive gains in their learning. Most studies report positive effects of having students self-assess,the median effect lies between .40 and .45, a moderate effect consistent with values reported in Black and Wiliam (1998).

Promoting metacogitive practice (knowledge / skills) is an area of interest, particularly when connect with meso cycles (cycles within a scheme of learning, 8-12 lessons in English), assessment and feedback. More on this area soon once the terms data has been reviewed.

Interleaving simple mixes topics instead of focusing or blocking the same topic. This is a long-term strategy. It is more effective when those topics are related.

Rohrer and Taylor (2007) reported greater performance for those students who learned through interleaving (63% accuracy) compared to blocking (20% accuracy).

Both spacing and interleaving strategies lead to slower and more error-prone learning in the short-term however, they boost memory retention when implemented over a longer time span. But this is also one of the reasons students, in spite on being told their test scores, still report they prefer that massed, re-reading type strategies.

Using these two strategies RememberMore enhances inductive learning (making patterns and connections) and later memory retrieval. Spacing benefits memory recall whilst interleaving enhances inductive learning and later memory retrieval.

Butler et al., (2017) found that interleaving or viability applied to retrieval practice with "superior transfer of knowledge to new examples," suggesting that repeatedly retrieving and applying knowledge to different examples is a powerful method for acquiring knowledge that will transfer to a variety of new contexts.

Recent research shows that the mnemonic benefits of taking a test are not limited to the specific questions or facts that were tested; retrieval practice also improves transfer of knowledge to new contexts.

“When students must learn to distinguish among similar concepts (or terms or principles or kinds of problems), the findings reviewed here suggest that the exposures to each of the concepts should be interleaved rather than blocked.” (Rohrer, 2012)

Five days after the last lesson, all the students went through a review session. Then, a surprise test was given one day or one month later. The results from the next day showed a 25% better performance for the problems trained with interleaving. The results from the next month, though, showed an amazing 76% advantage for the problems trained with interleaving versus the ones trained with blocked practice.

“Students might balk because interleaving increases the difficulty of a question or problem. A group of questions or problems are easier when all relate to the same topic or concept. By contrast, answering a set of interleaved biology questions might require students to consult material presented in previous chapters, and an interleaved mathematics assignment prevents students from simply repeating the same procedure throughout the assignment. Students will therefore make more errors, and work more slowly, when assignments are interleaved. This in itself is not problematic because the aim of classroom instruction is ultimate mastery, not error-free learning. Still, some students might be unwilling to make the extra effort. In this scenario, interleaving is like bad-tasting cough syrup—ineffective because children refuse to use it,” (Rohrer, 2012).

The great majority of the participants in the present study, as well as those in prior studies, judged that they had learned more effectively with blocked than with interleaved study. Thus, a bit of practical advice to learners and educators seems warranted: If your intuition tells you to block, you should probably interleave. Birnbaum M, Kornell N, Bjork E, et al. (2013)

“Interleaving boosts inductive category learning and later transfer.”

“Interleaving can help you see the links, similarities and differences between ideas more easily.” (Rohrer, 2012).

Interleaving improves the brain’s ability to tell apart, or discriminate, between concepts. With blocking, once you know what solution to use, or movement to execute, the hard part is over. With interleaving, each practice attempt is different from the last, so rote responses don’t work. Instead, your brain must continuously focus on searching for different solutions. That process can improve your ability to learn critical features of skills and concepts, which then better enables you to select and execute the correct response.

A second explanation is that interleaving strengthens memory associations. With blocking, a single strategy, temporarily held in short-term memory, is sufficient. That’s not the case with interleaving—the correct solution changes from one practice attempt to the next. As a result, your brain is continually engaged at retrieving different responses and bringing them into short-term memory. Repeating that process can reinforce neural connections between different tasks and correct responses, which enhances learning.

The concept of elaboration among cognitive psychologists is broad and can mean a lot of different things, however elaborative interrogation is essentially asking ‘how’ and ‘why’ questions, elaborating on the opening question. It is constructing meaning of the subject matter by explaining, elaborating, making inferences, connecting new facts and ideas to prior knowledge.

The purpose of elaboration is primarily to improve our memory for the new information. An additional benefit is that it begins to organize our knowledge in a more coherent way.

In one of the largest and most comprehensive meta analysis of undergraduate STEM education published, Freeman et al., (2014) reported an average examination scores improved by about 6% in "active learning sections," (elaboration) and failure rates reduced from 34% to 22% where elaboration was established practice.

Across several instructional conditions and settings, Nesbit and Adesope (2006) found the use of concept maps was associated with increased knowledge retention. With (Butler, Godbole, & Marsh, 2013) show further elaborating on the correct answer in the feedback can be important for later application of that knowledge. In fact, activation of elaborative information which would occur to a greater extent during retrieval practice over restudying — i is one mechanism that is used to explain the broader concept of and effectiveness of Successive Learning.

Before we go on, not all elaboration techniques are equal.

Elaborative information tends to increases performance only when it helped specify the relevance of "target information." In this situation, RememberMore tags / cues provide excellent opportunities for elaboration as does making connections between the cues, between prompts or any presented information within RememberMore, particularly with Classroom. RememberMore "Notes" add yet a further layer.

Equally important is giving learners the opportunity to explain the prompts themselves. Doing so encourages the learners to infer the missing information, to synthesize the presented information. Those learners encouraged to self-explain outperformed those who read and reread material (Griffin, Wiley, & Thiede, 2008).

Familial with post-testing gains, most educators would probably assume that giving students a test on material before they had learned it, would have little impact on student learning beyond providing teachers with insight into their students knowledge base. You may be surprised to learn of potentiating impact of pretesting. Simply, that pretesting (and pre-questions) have the potential to increase the efficiency and utility of the future learning that follows.

Research suggests that tests can be valuable learning events, even if learners cannot answer test questions correctly. That prior tests enhance the encoding of not-yet-taught and not-yet retrieved items. Even when the rate of success obtained in the pre-test was very low (in Richland et al.’s study, participants got as many as 95% of the pre-test answers wrong), learning with a pre-test was better than just studying twice.

Directly comparing both opportunities, post-testing and pre-testing relative to an extended reading condition, on a retention test 7 days later, Latimier et al., (2019) reported both posttesting (d = 0.74) and pre-testing effects (d = 0.35), with significantly better retention in the former condition.

Wissman, Rawson, and Pyc (2011) reported that retrieval practice of one set of material may facilitate learning of later material, which may be related or unrelated.

Similarly Karpicke (2009) reported the benefits of active retrieval improves students’ encoding when they restudy material.

"Pretrieval" practice - asking questions before teaching has a beneficial effect for learning. Most often attributed to learners being "primed" or "selectively attending" to pre-tested material - during the taught phase.

"Studies have shown that prequestions - asking students questions before they learn something - benefit memory retention," (Carpenter, Rahman and Perkins, 2018).

Pre-questions are things you can ask your students about material that they have not yet learned. Research shows that students who had been asked pre-questions were later able to recall almost 50% more information than their peers who had not. They were also able to remember other key information from the lesson too., Carpenter, et al., (2016).

In five experiments (combined n = 1,573), participants studied expository text passages, each paired with a pretest or a posttest, administered with or without correct answer feedback (Experiments 3–4). On a criterial test administered 5 min or 48 hr later, both test types enhanced memory relative to a no-test control, but pretesting yielded higher overall scores. That advantage held across test formats, in the presence or absence of feedback, at different retention intervals, and appeared to stem from enhanced processing of text passage content (Experiment 5). Thus, although the benefits of posttesting are more well-established in the literature, pretesting is highly competitive with posttesting and can yield similar, if not greater, pedagogical benefits. Pan and Faria (2021).

The diagnostic value of retrieval cues — the degree to which cues help people recover particular target knowledge to the exclusion of competing candidates — is the critical factor for all learning.

Retrieval may enhance learning because it improves the match between a cue and particular desired knowledge, or it may enhance learning by constraining the size of the search set— the set of potentially recoverable candidates that comes to mind in the context of a cue (Karpicke & Blunt, 2011b; Karpicke & Zaromb, 2010).

Carpenter (2009) items recalled from weak cues were retained better over time, such that this advantage was eliminated or reversed at the time of the final test. Of note, is that diminishing cues conditions consistently outperformed the study-only condition and the accumulating cue condition, Finley et al. (2011). RememberMore is currently exploring how best to apply this empirical research.

Simply, the design of RememberMore promotes efficient prompt design with retrievals tags enabling shortened retrieval prompts, reducing error rates. Furthermore, tags then act as a knowledge "filter," learners now free-recall from a redacted knowledge base, increasing retrieval rates.

Learning pre-sights provide the learner feed-forward about what they are about to review. Insights provide the learner with immediate feedback about the information they have just reviewed and deeper insights provide feedback and progress over time.

"Retrieval practice is often effective even without feedback (i.e. giving the correct answer), but feedback enhances the benefits of testing," Roediger and Butler, (2011).

Kornell and Son (2009) found that across experience with multiple study-test cycles, learners learned to self-test themselves more and did so at a faster rate when there was feedback on the tests at the end of each cycle.

Finally, rewards. Learners’ mnemonic performance is enhanced when incentives are present. RememberMore employs rewards, carefully scheduled and carefully constructed to complement each other and to deter ‘learner retirement.’ Also note that learners are more responsive to increased costs than increased benefits.

Social comparisons can also incentivize learners at recall with Fraundorf and Benjamin (2015) suggesting that learners are sensitive to how their experience with materials compares with others’.

Test anxiety is quite common. This nervousness can negatively affect student performance on those tests and can even impact their feelings about school and education. Retrieval practice had quite the positive effect on test anxiety experienced by middle and high school students. Data collected from 1408 students between the ages of 11-18 showed 72% of students reported retrieval practice made them less nervous for tests and exams, 81% of students reported they felt the same amount of test anxiety or less in the class that used retrieval practice relative to their other classes.

“This finding suggests that experiencing retrieval practice makes students less anxious regarding upcoming tests and exams for classes in which retrieval practice was implemented." Agarwal et al. (2014). Similarly Sullivan, (2017) found that percentage elevated to 90% for College students.

Furthermore, in another study, it was shown the retrieval practice was “had the most potential to create memories that were resilient to stress.,” Whereby retrieval practice appeared to create more distinct routes to the information to be retrieved, even if cortisol impairs one or more pathway, another may still be viable for retrieval.

Back to those re-reading strategies. Increased stress made re-reading strategies even more vulnerable. Studies showed that these learners experienced even great performance impairment. Up to 32% worse.

In perhaps the most extensive study of this effect, Agarwal et al. (2014) surveyed 1408 middle and high school students who had experienced classroom-based retrieval practice programs (e.g., like the one implemented by McDaniel et al., 2011). The key findings from their survey were that 92% of students viewed the classroom retrieval practice activities positively, believing that retrieval practice helped them learn, and 72% said that frequent retrieval practice helped them feel less nervous about classroom exams.

"Students perceived retrieval practice as helpful for retaining and retrieving information over time, for monitoring their comprehension, and for cueing important information for subsequent restudy," Cogliano, et al., (2018).

This concurs with what learners tell us about learning with Classroom and to a greater degree RememberMore.

If "Successive Relearning" aids retention, expectation of any kind of test, enhances the processing of studied material. is an important motivator driving students to commit more effort to prepare for subsequent tests (Agarwal & Roediger, 2011; Szpunar, McDermott, & Roediger, 2007; Yang, Chew, Sun, & Shanks, 2019). Simple, test performance is higher when students expect a final cumulative test compared to when they do not, even when the students are not provided with an opportunity to restudy the material, Szpunar, McDermott, and Roediger (2007). Why? Two theories are presented: By reducing learners’ mind-wandering during studying and second, by reducing interference from previously studied information Weinstein et al., (2014).

Other researchers have similarly found increased performance for learners expecting either a multiple-choice or essay test (compared to students with no test expectation). The research also suggests that telling students the type of test they should expect (e.g., multiple-choice or essay) and the target learning goal (e.g., memory, comprehension, or application) can benefit their learning.

Expecting an upcoming test can be helpful to learners, actually gaining experience with the test leads, to the greatest adoption of optimal learning strategies, Colak, (2015).

The more with teach with both classroom.remembermore.app and RememberMore, the more I am convinced that success drives motivation. That in turn, this motivation then drives success. The success-motivation-success cycle. What is more, I am convinced that the RememberMore approach protect learners (sometimes) fragile self-confidence. Here is how.

"If class was a poke game and you give each learner only one poker chip. Then each time you ask a question of the class you are asking learners to go all-in. Unsurprisingly, many learners choose to hold on to their chip, even those who are relatively confident, and look down. A few will risk, though only a few." -- My thanks to Chris Smith for this!

With low-stakes RememberMore, learners get both the prompt and the desired response, and they get to hold onto their chip. The knowledge become "sticky," they learn, and eventually remember, more. With a growing security, a deepening knowledge, their willingness to risk that chip become more and more likely.

It is not a question of whether or not prior knowledge is important for learning, it is a question of how important it is, and how important it is in facilitating the acquisition and consolidation of new learning. It is "all the knowledge one has before learning about a particular topic."

Here in the UK, setting aside data that describe the socio-economic circumstances of children, prior attainment at the end of primary school is found to be the most powerful available predictor of Secondary School attainment.

What about in classrooms? Prior knowledge has long been considered the most important factor influencing learning and learner achievement, with both "the amount and quality of prior knowledge positively influencing both knowledge acquisition and the capacity to apply higher-order cognitive problem-solving skills," Hailikari et al., (2008); Dochy et al., (1999); Chen, et al., (2018). With Dochy et al. (1999) go as estimating that between 30-60% of the variance in learning outcomes is explained by prior knowledge. Finally, given our focus on Successive Relearning, Van Kesteren, et al., (2018) demonstrated how prior knowledge structures are suggested to facilitate many stages of memory processing such as encoding, consolidation, and retrieval and reduce the interference of competing memories.

Cognitive research continues to uphold the importance of content knowledge in understanding academic material and the world around us. It is the foundation that enables us to make sense of what we learn. A seminal study by Recht and Leslie (1988) has shown that content knowledge is a better predictor of a learner's understanding of a text than reading ability; learners who are familiar with the relevant content of an article, understand it better than do their peers who are presumed better readers.

In the broader conversation of a "modern" education, we are often reminded that our learners future success, within an increasingly automated, artificially intelligent world, will demand more non-routine, creative thinking and refined reasoning skills than said declarative or substantive knowledge. However, this proscribed dichotomy is arguably misleading: with what do we make connections or challenge assumptions, if not with the thoughts, ideas and knowledge stored in our memory.

Value knowledge. Define what needs to be learnt, design the learning, and signpost exits routes from it. Without question, an essential factor in developing an integrated knowledge framework is aligning the educators expectations of learner prior knowledge, the learners actual knowledge base and what is to be learnt. Defining the parameters for substantive or declarative knowledge is important for both the educator and learner. Challenge your learners in your classes to acquire it, to use it and share it. And when they do, notice it and celebrate it. This is no easy task, though ignoring or undermining the importance of content knowledge in the hope that it will be assimilated is not a fair strategy either.

Vocabulary and simple definition acquisition is possibly one of the most obvious use cases for RememberMore (all the memorable when mixed with other content and knowledge). Why is vocabulary acquisition so important? Word knowledge is crucial to reading comprehension and determines how well students will be able to comprehend the texts they read. Now I accept that comprehension is far more than recognising words and remembering their meanings, however, if a student does not know the meanings of a sufficient proportion of the words in the text, comprehension is impossible.

Vocabulary experts agree that adequate reading comprehension depends on a person already knowing between 90 and 95 percent of the words in a text (Hirsch, 2003). Knowing at least 90 percent of the words enables the reader to get the main idea from the reading and guess correctly what many of the unfamiliar words mean, which will help them learn new words. Readers who do not recognise at least 90 percent of the words will not only have difficulty comprehending the text, but they will miss out on the opportunity to learn new words.

The importance of knowledge.

Knowledge (background knowledge) makes one a better reader in two ways. (There is a greater probability that you will have the knowledge to successfully make the necessary inferences to understand a text and second, a rich background knowledge means that you will rarely need to reread a text in an effort to consciously search for connections in the text.) Knowledge help you remember new information. A rich network of associations makes memory strong, with new material more likely to be remembered, as it is more likely to be related to what is already in memory. Converserly, remembering new information, on a brand new topic, is more difficult, it has nothing to connect with, get stuck to. (Remembering new information, on a familiar topic, is relatively easy if you practice developing associations between your existing network and the new material).

Knowledge for thinking

Knowledge enhances thinking in two ways in that . First, it frees up more working memory. Second, serves more known knowledge to work with. The area of education most fervently explores is problem solving, most often in mathematics and science, that is not to say knowledge for thinking is applicable to other subject areas - chess normally gets a mention here too. Burns (2004) positioned that the recognition process accounts for most of the differences among top blitz chess players, drawing from hundreds of thousands of stored sequences or moves.

We foresee RememberMore being adopted as a revision tool. How might RememberMore support effective revision practice in addition to adaptive, spaced retrieval? RememberMore:

is mobile, accessible, encourages learners to build retrieval / revision into their practice.

tags break down learning into manageable (and potentially, interleaved) chunks

enable revision as a moments notice, offer the opportunity for shorter, more frequent, revisions periods

Calibration has been defined as the degree of fit between a person’s judgement of performance and his or her actual performance (Keren, 1991). As such, calibration reflects a metacognitive monitoring process that provides information about the status of one’s knowledge or strategies at a cognitive level. Based on this information, control at a metacognitive level can be exerted to regulate one’s knowledge or strategies (though often this is under utlised by learners). Therefore, greater accuracy in a person’s judgments of performance, being well calibrated, creates greater potential for self-regulation.

Incentives to be accurate, significantly improved calibration accuracy only among lower-achieving students, Schraw et al. (1993).

Routines are at the heart of the integration of RememberMore and classroom.remembermore.app (CRM). We know first hand, the power of routines, securing overt written retrieval routines, moving to to covert routines that can present extensive Successive Relearning opportunities.

Inspired by Peps McCrea thread on routines, here is how RememberMore promotes a productive, learning environment.

RememberMore is exceeding fast to set up, less than 10s. There is minimal on screen distraction. Instruction can be independent, coral or draw on educator-learner interaction. High learner agency, self-assessment, presenting both retrieval prompt and response, high success rates, as well as the routines themselves, promotes confidence and safety. What Ben Windsor coined "The Deadpool Effect" of RememberMore, now on it's forth iteration:

The Deadpool Effect: Two parts psychological security (the correct answer is always provided) + one part student agency (self-marking and correction) + one part low stakes (retrieval not assessment).

Importantly, class learning is integrated with RememberMore, which also measures performance and tracks progress.

"One of my students is up to 3 hours. Now in class, where before I got shrugs, I know have a hand shooting up frequently and vigorously." @MrClassics

Having RememberMore routines almost instantly available, means educators have greater resource to allocate to teaching.

Importantly, learner 'attention' is shift from "how to," to retrieving or thinking about the activity presented. Learners working on the RememberMore app are focused the classroom is remarkable quiet. Successive Relearning via the app is personalised.

See our classroom routines page for a range of routines and we hope you will share you ideas with us.

One further area of professional interest, is instructing learners to generate questions based on the learning material which has been shown to yields medium to large effects on comprehension, recall, and problem solving (Song, 2016). Similar to retrieval, generating questions may stimulate a deeper processing and reflection of the learning material, as well as retrieval practice in comparison with restudying. However, in most of the reviewed studies, learners were trained on how to generate questions effectively and practiced this strategy in advance and under supervision. In addition, the learning material involved only short text passages, and only short-term effects were examined.

Of course, generating questions also give further opportunity for retrieval!

"Though cognitive-science research indicates that 95% of children can learn to read, rates of reading failure are typically around 30%," (Hempenstall, 2013) cited in Snow (2020). PISA reports 20% of 15 year olds read below "acceptable standards," (PISA 2015; Jerrim & Shure, 2016).

Oral language development is critically important for reading and writing which in turn underpins academic achievement. Oral language and reading as “inseparable friends,” who take turns to piggy-back on each other during the school years and beyond. We also know that reading and literacy are inextricable intertwined. That "Language is literacy is language," Snow (2016) and that influence is bidirectional.

Why should you be interested in vocabulary acquisition, accuracy, fluency and retention? Adding to the research of Lipowski et al., (2013). and Goosens et al. (2014a, 2014b, 2016) reading is how older learners acquire new knowledge, almost exclusively, new vocabulary.

Exposing learners to increasingly complex sentences and discourse, to background knowledge, is not enough in of itself to move a child, a learner, from being a ‘good talker’ to a ‘good reader.’ Moving a child, a learner, from being able to decode words, to understanding their meaning, requires teaching.

Learners rely on knowledge (that includes a rich and broad vocabulary) to attend to, and make sense of what they are reading. Vocabulary are the clues of "meaning making." We have to teach / instruct learners, to broaden their vocabulary needed for schooling and independent reading / learning. Vocabulary not present in oral communication.

Reading then, is required to access knowledge and it's importance increases as students get older.

Precisely how much vocabulary we need is unclear. A minimum estimate for minimally acceptable reading and listening comprehension is 95% recognition of words, which would need around 3,000 word families. Most researcher put the figure for unassisted reading and listening at 98% word recognition, which research has shown to require 6,000 word families for listening and 8,000 for reading, Hu and Nation (2000).

"We know that the explicit teaching of vocabulary matters. Disciplinary literacy matters. And it matters to every teacher and pupil. Indeed, success in each and every classroom depends upon it." Alex Quigley

Beck suggests we need to teach 7,000 Tier Two word families. Or 700 words per year. Beck et al., (1982) research calls for 400 words minimum, 400 words per year conforms to the rate at which improvements in word knowledge and in comprehension of text would be required "to be proficient." Introducing 400 is a clear maker. 700 is the aim. Nation (2006) shows that relationship quite clearly (see Table 14 and Figure 1). 95% coverage can be achieved by 5,000 word families. 98% coverage is achieved by an additional 2% provided by 6-9,000 words and an additional 1% of proper nouns.

We know from designing RememberMore class reader decks, most texts offer easily 80-120 words or word families to teach. That is our starting point, along with Dr Averil Coxhead's "Academic Word List" - 🚧 here 🚧.

16.07.21 Roots, Prefixes and Suffixes added. 99 cards.

20.07.21 We added "The Dirty Thirty" 30 Most Common Misspellings list to Classroom. We are working on a disciplinary list for Secondary Education.

How to present that vocabulary is currently being investigated:

• First iteration: vocab: definition - (notes - use or textual reference) and tagged.

• Second iteration: vocab or phonetic vocab: definition (notes - use or textual reference) and tagged.

• Current iteration: vocab (word class) or phonetic vocab: definition (notes - use or textual reference) and tagged.

Transfer of learning, or transfer, is the application of learned concepts or information to new or novel situations. Now, to remind you, Retrieval does more than measure memory, it modifies memory in a way that makes information retrieved on a test more likely to be remembered in the future than it would have been otherwise.

Butler et al., (2017) found that repeatedly retrieving and applying knowledge to different examples is a powerful method for acquiring knowledge that will transfer to a variety of new contexts.

There is evidence to suggest that retrieval practice can foster transfer across a variety of contexts and that it is most effective when coupled with feedback or elaboration. Of course, learners need to have been successful initially, need to recognise that their knowledge applies to this new or novel context.

The reporting of boundary effects for the testing effect have been highlighted in light a EEF f

Bertilsson et al., (2020) recently focused on had reported on how individual differences in personality traits and working memory capacity moderate the size of the retrieval-practice benefits. Learning was assessed at three time points: five minutes, one week, and four weeks after practice. The results revealed a significant testing effect at all three time points. That is to be expected. Further, the results showed no association between the testing effect and the personality traits, or between the testing effect and working memory, at any time point.

• The type of recall, free recall, cued recall, recognition recall (MQCs), confidence levels.

• The type of information, most regularly paired associates, by that is not the only knowledge investigated. Also declarative vs procedural.

• How easy the knowledge is to learn or perceived to be (ELER - easy to learn, easy to remember) vs desirable difficulties.

• Age of learner, curriculum area.

• Covert or Overt. Direct vs self-regulated.