With twenty plus years teaching experience, eight years as a Head teacher and Senior Leader, the past two years have also included the design, prototyping and deployment of RememberMore.
Needing to establish myself in a new school and accelerate learning, I turned to the cognitive psychology and eeducation research. As I read more widely, as put theory into practice, I looked to leverage the learner gains of spaced retrieval practice with feedback, interleaving, metacogition and personalisation.
September 2019 - Routine, purpose and 'meaningful' learning
For context, I was joining an oversubscribed, inclusive comprehensive Secondary school with "a student intake that reflects a wonderfully diverse and vibrant local community," in the third week of term. A school with a strong moral purpose, strong leadership and defined school values (Challenge, Creativity, Commitment, Co-operation and Courtesy.) Teaching English to predominantly Key Stage 3 (11-14 year-olds) mixed and low prior attaining classes.
Prior to meeting the classes, I built a basic class profile (names, photos, prior learning data and background) and reviewed the curriculum outline. The classes would be best described as low prior attaining, a number of learners had higher than average negative behaviour points. With the exception of one class, I was / we were nomadic.
I learnt, presented, and anchored everything to school values and left these on display. I learnt the Behaviour Policy and the terms used in School Reports. Building upon the curriculum outline I was provided and the learner course outlines that were "to be stuck in the front of class exercise books." Within two, at best three lessons, the challenge was clear and the curriculum could potentially have been described as out-of-reach. I swiftly built a basic of Knowledge Organiser that defined the declarative content. I had learners stick these in the back of their class exercise books. To this resource pairing, I introduced a low-stakes five retrieval practice lesson starter, presented each lesson via the board.
Routined lesson opening
a warm class welcome and greeting at the door, exercise books and class texts available for collection and distribution*
retrieval prompts displayed on the board as fast as possible - "Do it now," style supported by accessible curriculum outline and knowledge organiser in the exercise books
a set routine for responding to the prompts with the shared and a clear expectations of "All prompts. Every Practice." Later, I added a notice "Independent thinking time" and followed by "Underline the title and date" highlighted in florescent yellow as I sought to establish a purposeful learning climate. Finally, the "School values," underneath.
a separate visible timer
walk the room, check exercise book work and affirm standards, and frequent verbal 'positive' endorsement of learners "thinking-hard" or "on task" or revisiting past quizzes in their exercise books.
a short self-assessment marking routine (which, at the start was highly under-developed and undervalued)
A routine, that once initiated, was designed to set and manage the learning climate as learners arrived and to start lessons promptly, whilst allowing me the time take the register (and set up the forth coming lesson). To support the delivery of declarative or substantive knowledge required to access the lesson. To present a clear and clean and transition to the taught phase of the lesson settle.
🎯 The purpose was, and still is, simple. To define the declarative or substantive knowledge required to access and explore the lesson more broadly and deeply. It was about enabling and empowering learners to achieve better outcomes.
"As our knowledge becomes deeper and more comprehensive, our capacity for critical thinking, problem solving and creativity within that domain unfolds," Daniel Willingham, (2007).
Over the next two years, I would read veraciously, improve and refine the routine iteratively, slowly moving towards Successive Relearning. I would learn that a routined Successive Relearning approach offered so much more than I had initially conceived.
*Lesson entry was another set micro routine. Practiced and rehearsed. I was moving between classrooms and often arrived at the same time as learners. Exercise books and class texts were in labelled boxes at the door. Collected and distributed by the first two learners to arrive.
Routine, purposeful and meaningful... Building Knowledge Power
Routine, purposeful and meaningful. Three words that I would use to describe my teaching practice and empowerment of learners. Three words that I would also offer support my teaching and classroom management. To a large extent, routined retrieval practice married with that philosophy and it proved to be very effective, both contributing to a purposeful lesson starts and underpinning meaningful learning thereafter. Was it easy? No. The truth is, learners found the initial process effortful and with only nominal reward. I would later learn that I had missed a critical step.* We can come back to that later.
By half term, I had gained some important insights for retrieval practice lesson starters. I had the benefit of hindsight. The learners in my classes could now reference tangible learning gains. Building the routine, introducing and revisiting the declarative or substantive knowledge was proving to be an excellent investment in the learners and it was starting to pay dividends. Directly (what the learners knew and could remember) and indirectly (how they viewed themselves as learners, their confidence and contributions to lessons).
Together, 'we' were building a rapport. Normally, I survey class views at the end of the academic year, however, keen to build learner commitment, to show that they views matter, I surveyed the class that half term. Just three questions.
What did you like MOST about English with Mr Still?
What did you like LEAST about English with Mr Still?
Any comment or feedback you wish to add?* (optional)
One comment struck a cord.
"I have found it easier to feel comfortable in class and not feel pressured or under prepared when a question is asked." - Year 8 learner.
It was a comment that seeded my early interest in the relationship between learner motivation and achievement and encouraged further investigation and research. In reflection, that half-term, I worked as hard as I have ever worked. As well as getting to know a new school, reading and exploring new texts I was teaching, as I was teaching it, I was also building resources and writing retrieval prompts.
My own professional learning and insights with retrieval practice
At the time, I did not recognise that the routine that I had adopted was more "Successive Relearning," than simply retrieval. It was more than leveraging the benefits of the testing effect, spacing and feedback. It was often repeated retrieval and therefore, to some extent, spaced. Though minimal consideration for that spaced. It would best be described as routined, purposeful and meaningful teaching.
It was barely repeated, it was not criterion referenced, I had not truly considered the role of the learner, nor did I have an awareness of the various complexities of retrieval (direct and indirect benefits, overt and covert, metacognition, self-regulation), all that was follow over the coming months and few years.
At this point, I still had not read Dr Katherine Rawson's work, watched any of her presentations, nor did I have any understanding of criterion referenced learning. What I did learn was, that as the classroom routine became more secure, as we revisited the content, so knowledge retrieval accelerated and learning deepened. I learnt that I could expect more from the learners in the same 3-4 minutes allocated to the "Do now" routine than at the start of the term. It was a virtuous circle. As retrieval accelerated, we retrieved more and more prompts. As learners retrieved more, they more they remembered and the more they grew in confidence. The more confident they became, the more positive contributions, and more sophisticated contributions they made to class. What become known as the "success-motivation-success" cycle. Once it kicks in, learner confidence accelerates and the learning climate palpably changes.
Now with nineteen* (last updated 27.07.21) iterative experiences under my belt, I can almost define that pattern to the lesson (pending on the group variables: age, class climate and culture, dynamics.)
November 2019 - There must be a better solution?
After the encouraging half term summative assessment (November 2019) we moved to explore different digital solutions for retrieval practice. We tried a range of different platforms over a two-three months time period and these were very popular with one class in particular. So much so, that 80% of the class reported they would prefer 3 taught lessons and 2 retrieval sessions per week, 20% would like 1 session a week, rather than five taught lessons. On the flip side, the platform was not education friendly and booking the laptop trolley had its challenges. I will say, the learners time on task, was still, overall, excellent.
We continued with retrieval lesson starts. Kate Jones released "Retrieval Practice: Research & Resources for every classroom." I started reading more of the research journals. The arena of retrieval practice broadened and deepened. The boundaries of retrieval research were getting wider and more inter-connected, and they also deepened. Unsurprisingly, retrieval practice (optimising learning) has a deep history of research. It was Wozniak's research that first creaked open the door of spaced retrieval.
January 2020 - There must be a better solution?
After a four months of practical and theoretical investigation, some semi-successful classroom trials with various products, I had amassed an admirable list of dead-ends and setbacks. It was then I first approached Ty Goddard (Chair of EdtechukHQ) with an idea of a retrieval "product" for teaching. I reached out to one of two digital companies, but regrettably, all I had was an idea, a concept and I made no progress.
Another half-term passed and now had a confirmatory sets of data from three different classes, one Year 7 and two Year 9 classes, for the end of term assessment. With renewed confidence, I again approached Ty who provocatively replied with "What's stopping you from just doing it yourself?" Apart from teaching full time, applying for leadership rolls and settling my family?
He challenged me to "...just get on with it yourself!"
Successive Relearning - Dr Katherine Rawson
Like most educators, I first encountered retrieval practice before my investigations led to the work Dr Kathleen Rawson and Successive Relearning - specifically, Rawson, Dunlosky, & Sciartelli, (2013). Conducted in an authentic Introductory Psychology course, college students learned 64 course concepts using a computerised flashcard program under the following conditions.
Successive Relearning: Students engaged in retrieval practice spaced throughout the semester, and they had to retrieve information correctly three times.
Spaced Restudy: Students engaged in restudying, spaced throughout the semester (without retrieval) - arguably the most common approach
Baseline: Students engaged in "business as usual" without using the flashcard program
In two experiments, the successive relearning condition increased students' course exam performance by a letter grade (from a C to a B) compared to the spaced restudy and baseline conditions, consistent for both higher and lower performing students and more robust for three spaced retrieval sessions compared to only one retrieval session. I simply wanted that same benefit for my students.
I read most of Rawson's papers and watched as many presentation as I could find. Including her presentation at the Harvard Initiative for Learning and Teaching 2013 conference presentation. During this presentation, Dr Rawson clear directs educators to coach their students "how to approach their learning."
"... it's a strategy that students could use, in practice it's a strategy that they don't or don't use properly so students are going to benefit from some coaching from instructors so imagine just taking a few minutes class time near the beginning of the semester to talk generally to students about how to approach their learning outside of class and describe this strategy is one that could be particularly useful for many of their learning goals.
I recognised we had missed this crucial step in the introduction of our routine. We had not explicitly told learners that retrieval practice "is not the easy option.' That it would be "effortful," (to borrow from Blake Harvard blog title) and that the learning benefits are deferred down the line. Learners need assurance and it is down to us to provide it, to sign post it, and then notice those moments in lessons when knowledge surfaces.
Here is what I knew - so far anyway
Learners reported growing confidence and security. In class, improved knowledge retention and perceived learner engagement. Learner commitment was deepening and class behaviour had improved. Most interestingly, learner's written assessments had improved considerably (particularly the retrieval practice keywords). The research journals I was now reading regularly, largely supported the performance gains of "the testing effect," or retrieval practice. There was also a significant amount of interest in this topic from the casual literature also. I continued to read as much and as broadly as I could. Some of the research started to signal that AI was interested this area of learning research.
After numerous retrieval product experiment failures, I half-knew what the solution 'could' look like. I half-knew what did, and what might work in a classroom setting, and what might support the classroom teaching. I knew that the learners were keen and I had my learners wish-list. I certainly knew what did not work. According to my classes - more retrieval practice and less teaching! I was starting to recognise the importance and the challenge of spacing - without a database and algorithm. Of course, digital also meant collaborative. 🎯 It had always been about learners outcomes but a digital approach also presented an opportunity to curate and collaborate - and reduce educator workload.
Early March 2020 RememberMore
So I pitched my idea to Alex. I told him about what my classes has taught me. The potholes and pitfalls I had encountered. I pitched him the idea for RememberMore with the simple, but dual aims of improving my students outcomes and at the same time reducing teacher workload. It was not one or the other, it was both. A community of teachers, sharing their 'question decks' and making them available to their students, where ever they may be.
Days later the world changed. A global pandemic provided an unexpected opportunity to fully map out the idea of RememberMore. And still, without the unwavering support of Alex Warren (Senior Software Developer), RememberMore would never have seen the light of day.
From March to August, over a six month period, with support of more than thirty school leaders, researchers, educators, data scientists, UX (user experience) and UI (user-interface) designers and of course our the students in my classes, I worked every spare hour I had on RememberMore. Reading research, writing to researchers, speaking to teachers and discussing my teaching with my students.
In August we had access to a BETA RememberMore.
May 2020 - RememberMore blueprint
Persistently irked by the now infamous quotation Graham Nuttall’s "Hidden Lives of Learners."
"Our research has found that students already know, on average, about 50 percent of what a teacher intends his or her students to learn through a curriculum unit of topic. But that 50 percent is not evenly distributed. Different students will know different things, and all of them will know only about 15 percent of what the teacher want them to know,” (Nuthall, 2007 p35).
For all it's benefits, retrieval and the routines I was teaching with, were still being presented to the entire class. What I refer to here as "one-to-many" retrieval practice. We knew we wanted to design an adaptive, personalised learning approach. So we embarked on the task of seeking a more targeted, learner-focused retrieval, successive relearning model given Nuthall's emphasis that "Learning is highly individual."
(It was not until I re-read the book for a third time in May 2021, that further insights surfaced. Nuthall's emphasis on "the accumulation of at least three different sets of complete information about a concept makes the difference between a concept that is never quite learned and one that firmly connects to and integrates with previous knowledge, and hence is learned and remembered," the need to "connect and integrate" knowledge (tags) and Lastly the use of the term "successive," jumps off the page, having read so much of Katherine Rawson's research.
Add to that, the remarkable prediction rate for concepts learned (88%) and those not learned (85%) concepts, (Nuthall 2007, p127) - now garnered my attention.)
With it's unique content design, research informed delivery and architecture, underpinned and refined by hours classroom testing, RememberMore (as a single app) was starting to take shape.
August 2020 - Classroom hatched
Given the breadth of support from educators up and down the country, around the world even, across curriculum areas and sectors, and given that, first and foremost I was a teacher, we wanted RememberMore to be accessible at the rockface, in the classroom, without the need for a mobile device or the need to book the laptop trolley!
🐣 Classroom was hatched. A web-based Successive Relearning platform.
It would be a teaching companion for educators, giving them fast, flexible access to 10,000+ (27.08.21) categorised and tagged retrieval prompt pairs. Supported by 10+ instructional routines.
We now had Classroom, a web-based Successive Relearning platform supported by a handful of instruction routines and RememberMore, an optimised, adaptive and personalised Successive Relearning sharing the same content as displayed via Classroom.
🎯 Improve learner outcomes at the same time as reducing educator workload.
The 2020-21 academic year - Classroom and RememberMore
It took a solid term (60 lessons) to define, road test and refine the routine or "approach." A full term of lessons, iterated across three groups or six iterations (3x2). We also benefited from feedback from five partner schools.
A further two terms (3x4) that academic year and the insights were forming and consolidating. In each of the subsequent terms, with each class, the sequence and schedule of learning and classroom behaviour, across all groups was almost identical. (In fact, it would appear very consistent across our five partner schools and we look forward to sharing the pilot study results soon - last updated 28.08.21).
Effort with little reward - learner frustrations, complaints, groans even
Effort with some reward - a few supporters for the routine, a few quiet celebrations
New learning or knowledge starts to surface in lessons and oracy starts to deepen classroom discussion. Learners start to ask or request "Are we quizzing today?"
Knowledge transfers to written contributions (16+). In core subjects, on five lessons a week, that is typically week four.
We continued to look for ways to be more efficient with the routine. We explored different retrieval prompts or questions stems and styles, we explored quizzing at different points in the lesson. We explored scoring models, self-assessment models, we even explored learners writing the retrieval prompts - which was an excellent way for learners to see just how discerning they had become. Finally, we established the BIG Quiz. A celebratory, 'tough mudder' end to the term quiz, brutal and rewarding. And at some point, around this time, I encountered the work of Dr Katherine Rawson.
We continued to make small improvements to Classroom and are genuinely appreciate the feedback and feature requests we received.
Late Sept 2020
RememberMore was shared with a handful of learners for testing. Feedback was generally positive with a handful of feature requests. Personalisation top of the learners requests.
October 2020 - RememberMore development continues
With an integrated Classroom and RememberMore, our sights were set on developing the Dashboard. Conceived as part of the initial designs, we hadn't realise the power of RememberMore metrics and insights until we started to explore our metrics ourselves.
Learner, group and cohort / subject analytics and comparisons capabilities.
Performance, progress and gap analysis.
Insights from RememberMore that may enable educators to “intervene before specialist intervention is required."
We were well aware of the potential.
Harnessing the potential of Artificial Intelligence (AI) notifications based on internal analysis and "exam readiness."
November 2020 - RememberMore partnerships
In November we partnered with cognitive psychologists and researchers with the aim of initiating field trials in 2021. If applied research interests you, then do get in touch, we are planning more research projects in the coming academic year.
Nearly a year after those first conversations, our first pilot study returned some very interesting data and feedback indeed.
December 2020 - RememberMore Class trials
The results from our first small pilot study (n=52*) investigated of best use and integration of RememberMore in class learning in three, Year 8 mixed prior attainment classes (1 AP class). (4 students missed the assessment.)
All learners were introduced to Classroom retrieval practice and therefore had access to Classroom both in lessons and outside of class. Five learners accessed RememberMore (iOS) averaging 196 minutes of unsupervised practice in a six week period.
Those learners with access to RememberMore showed comparable and positive improvements in their final term written assessment class rank compared to their standardised score rank.
This is not a empirical study
February - 2021 more and more decks
Progress in class was hampered by a second lock down. However, access to Classroom and the RememberMore understandably benefited from the constraints of remote learning. Access to Classroom also made remote learning a lot easier.
The number of decks on Classroom accelerated. We now have 60 decks, 10,000s (21.07.21) of categorised and tagged retrieval prompts, across sectors and 10 curriculum areas. Five schools are using their own Classroom platform.
Developments are almost monthly now.
RememberMore (4.01) released as the companion to Classroom. Early feedback was promising. "It's pretty good tbf." - student Kelly B.
Feedback is very positive. "....you learn what you don't know so when your in class you're a lot more confident."
RememberMore Dashboard private release - available to partner schools.
300th journal paper on all things retrieval, spacing, feedback, memory, metacognition and more.
⤵️ Reorder before 🔄 Refresh. New Match mode added. Print button added.
A refined Successive Relearning protocol is adopted - Retrieve ➡️ Reflect... ➡️ Reveal ➡️Repeat.
2021 - Easter break - a Minimum Viable Product finishing line is draw in the sand
RememberMore (4.03) released. Promo video released. New visual statistics, new card counts and cross category card counts. Mpre insights, improved "Challenge level" calculation. New global settings in preparation for research. Research pilot study announced.
All development is now focused on the Dashboard.
We are excited to see further adoption and innovation with RememberMore. We continue to seek collaboration and partnership as we bring the benefits of Successive Learning to more educators and learners. Two schools globally join the pilot study.
2021 - Summer term - the vocabulary focus takes root
We have stepped into research. A small pilot study is underway with the researchers from Leeds University's School of Psychology and five schools in the UK. The pilot is reviewing both the educator and learner perspectives, with a focus on learner confidence of learners. Second, the pilot is looking at RememberMore usage data. When, how much and for how long.
An off-shoot of the progress with RememberMore has a growing interest in the acquisition and retention of vocabulary.
"Simply, exposure to increasingly complex sentences and discourse, to background knowledge, is not enough in of itself to move a child, a learner, from being a ‘good talker’ to a ‘good reader’. "
"Language is literacy is language," Snow (2016). And that "influence is bidirectional," Dr Ricketts.
We are investigating how RememberMore can more effectively employed to explicitly introduce vocabulary. How to present and integrate disciplinary vocabulary. We are very thankful for the support of PhD candidate Grace Pocock. We hope to be taking forward this professional relationship.
2021 April - July Forecasting performance and metacognition
A second mini pilot trial explored the metacognitive opportunities associated with RememberMore, specifically self-regulation and learners forecasting performance.
"The act of assessing one’s own judgement of learning is one of the most effective ways to deepen a memory trace," Sadler (2006).
Setting an assessment will often lead to an improvement in performance, even if that assessment does not take place. Where students benefit from assessment preparation. However, in this case study, Year 8 (n=52) mixed prior attainment English learners were asked to forecast their performance on a weekly 10 question quiz from a deck of 30 cards or 10/30 and on a final end of unit quiz with a shared expectation that learners be accurate in their assessment.
It was an opportunity to explore Barenberg and Dutke (2019) findings, who also examined the potential of retrieval practise during learning to improve the accuracy of confidence judgements in future retrieval. Results showed attainment and forecast accuracy improving and variance reducing, if always a little confident. The over confidence was expected, see Dunning Kruger Effect or the metacognition research of Dr Michael Serra eg Serra and DeMarree (2016).
What did I learn? To be accurate in their forecast, students needed to know their performance. To know their performance, students needed to practice. We created a virtuous circle. Student attainment improved and variance reduced, student forecast accuracy improved and variance reduced. On the back of this trial data and the conversations I had with students each Thursday, I will be looking to replicate the study. Should anyone else like to replicate the study, then let me know and we can make RememberMore available.
July 2021 - More reassurance from the students
As the summer term closes feedback from two classes in Year 7 (The Declaration) (one mainstream and in Alternative Provision) and two classes in Year 8 (The Hound of the Baskervilles) I can share optional* feedback on the RememberMore from a learners perspective. Of the 27 replies (27.07.21) 17 offer feedback on RememberMore.
"Very useful app to learn some specific vocab."
"I wish we could use the app next year too."
" Very good and a efficient way of learning."
"Easy to use very useful."
"Even better you learn what you don't know so when your in class you're a lot more confident."
"Very enjoyable and adaptive to how you learn."
"I like that it shows your stats and how much it showed that you remembered. I think it’s a good touch and it’s good to know how you do."
90% of learners either strongly agreed (42%) or agreed (48%) with the statement "I am a more confident learner now, than at the start of the year. " Why? I have now doubt that the model of Classroom and RememberMore works on many different levels, confidence in class one of the more potent.
“The connection of confidence and knowledge provides an acceleration of learning and improves student performance by creating a more confident and productive student,” (Adams and Ewen, 2009).
There are now 10,000s (21.07.21) of categorised and tagged retrieval prompts available for busy educators ⏰👍 and determined learners 🎯🥇. Fast and flexible, open and free.
ResearchEd National Conference proposal successful.
July-Aug 2021 - RememberMore Dashboard
Dashboard development is progressing swiftly now. Meanwhile we are making Classroom and RememberMore as widely accessible as possible, building credibility and supporting learners. The 'demoschool' showcase site is available for anyone interested in road testing RememberMore. If after using demoschool, you want to contribute to our development - we are offering a rolling RememberMore, Classroom and Dashboard offer in exchange for your support and feedback.
July 2021 - the new 'Deck Editor' is out for testing with our team of fantastic educators. Early feedback is very positive. The move to RememberMore Dashboard makes make Classroom and RememberMore super responsive, additions and updates are immediate. Of course the editor is design specifically for designing content for RememberMore - with responsive filtering, sorting, and the ability to show and hide columns. Individually, these are small gains, collectively, it is significant.
Decks can now be managed, cloned, created and updated in Dashboard as well as imported directly from Google Sheets.
Research update: late July, early August, a flurry of published research and conversations underlining and supporting the "success-motivation-success" and the interplay of confidence based assessment and metacognition supports our theoretical stand point and practice observations. Research supporting retrieval practice continues to be published widely.
After screening 2000 abstracts, 50 real world educational settings papers made the cut, 49 effect sizes and a total 5,374 students data. Agarwal et al., (2021) found "a wealth of evidence," that retrieval practice "improved learning for a variety of education levels."
Higham et al., (2021) reported that increased recall during relearning sessions was associated with improved metacognition, increased self-reported sense of mastery, increased attentional control, and reduced anxiety. Furthermore, that students found successive relearning to be enjoyable and valuable.
Sotola and Crede (2021) found that, across 52 independent classroom studies (N = 7864) students were 2.5 times more likely to pass a course if their instructor incorporated frequent low-stakes quizzing than if the instructor did not.
Dashboard now shows the total time invested, the past 7 day cycle, and the current 7 day cycle time below the tri-colour learning band. (unseen, seen and remembered).
A weekly, updating CSV export enables RememberMore to be used as a companion homework tool and remove the workload of homework marking.
ResearchEd National Conference followed by ResearchEd Surrey.
Final plans for the Dashboard architecture penned. Out first full term with RememberMore at the rockface. Years 10 and 11 English. 1239 hours invested.