NOTE: I first published this article on another blog several years ago, but found that given the current buzz about “The Science of Reading” and similar, it was time to pull it down, refresh it, and repost.
Robert Marzano first published The Art and Science of Teaching: A Comprehensive Framework for Effective Instruction in 2007—nearly 20 years before the writing of this article. As an educator in my third year of teaching, I remember the sea change that it brought: the idea that education was a science as much as an art; teaching was not something innate to the individual, but a process that could be studied, learned, and modified for greater effects.
Indeed, a year later John Hattie published Visible Learning: A Synthesis of Over 800 Meta-Studies Related to Achievement, which focused on the concept of “effect sizes,” and what constituted a “strong” effect size versus a “weak” one. It was, in many ways, considered the authoritative guide to teaching for student achievement outcomes, often resulting in broad education practices like direct instruction or homework being distilled down to a single “effect size” number that led educators to say whether we should implement that strategy (like direct instruction) or whether we should not (like homework).
The idea of the science of teaching has only grown stronger since. One need only look at the sweeping legislative and educational reforms brought by the Science of Reading movement to see how much educators and education-enthusiasts have come to rely on the terms “research-” and “evidence-based.”
The problem, of course, comes when we use “research” and “evidence” as a definitive reason for doing or not doing something in education—without truly understanding the nuances and limitations of the research itself.
When I work with educators on school or curriculum reform, I always remind them that while using educational research is an imperative first step in making decisions, it cannot be the only step. To that end, educators must ask themselves the following three questions about any study they use to justify a course of action.
1. How was effectiveness measured?
Despite test-makers’ attempts to use myriad item types, multiple choice still makes up the majority of items on standardized assessments—and it has, for many years, which means that studies of “what works” in teaching and learning are more often than not using this type of measure as their determination of student achievement.
Multiple choice items work best for measuring students’ abilities to remember, understand, and apply knowledge, and can occasionally even assess students’ abilities to analyze (Gareis & Grant, 2015). These types of items cannot assess students’ abilities to evaluate (which this requires supplying justification) or create, meaning that there are significant skills left out of the majority of student achievement scores.
Let’s look at an example using the inquiry instructional model (a model in which the teacher acts as a facilitator while students research, observe, and draw conclusions for themselves). In his book Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement, Hattie (2008) combined the results of multiple meta-analyses to determine each strategies’ average effect size on student learning. He found inquiry-based teaching has, on average, an effect size of d=0.4; direct instruction (a methodology in which the teacher presents the information), however, has an effect size of d=0.59. It is therefore tempting to conclude that direct instruction is a “better” teaching methodology.
However, most of the assessments used to determine effect sizes in a majority of the studies that comprise his meta-analysis consist mainly of multiple choice items, which do not assess students’ abilities to evaluate and create.
The point of inquiry learning, however, is to teach those higher-level critical thinking skills. Hattie (2008) described that inquiry-based teaching actually yields an effect size of d=1.02 on measures designed to assess students’ critical thinking skills; the d=0.4 score comes from measures that use only student achievement test scores.
In other words, it is not enough to simply use a table of effect sizes. We need to look at the measures that were used to determine effectiveness and choose strategies that work best for the skills we are trying to teach students. In the above example, if our goal is to teach students scientific facts, direct instruction is the better model. If our goal is to teach critical thinking skills, then inquiry-based teaching is the superior strategy.
Moreover, we need to consider a balance approached to education. Some knowledge and skills only need to be learned at a surface-level, and others require deeper or transfer learning (Hattie & Donoghue, 2016).
2. How was the strategy implemented?
Let’s consider the homework debate. While some educators deem homework an important staple of our educational system, many schools eschew homework these days, pointing to its low effect size (d=0.29) on student achievement (Hattie, 2008).
The case is not that cut and dry, however. Marzano and Pickering (2007) explained that homework in elementary school has been found to have an effect size of d=0.15, while middle school homework has an effect size of d=0.31, and the effect size of high school homework is d=0.64.
Marzano and Pickering (2007) further explained that other studies have shown that homework is strikingly more effective when teachers grade it and provide feedback, and that after a certain number of minutes of homework per night, there are diminishing returns on student achievement gains.
In other words, educators must be careful not to justify initiatives with statements such as “Research shows that homework has an effect size of d=0.29.” Unless we know the specifics of the strategy implementation, we should not make overall judgements about its effectiveness.
3. Will this strategy work for our students?
Hattie (2008) only used quantitative studies in Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Quantitative studies can tell us, broadly, which strategies tend to achieve certain outcomes across the majority of students in the majority of content areas and grade levels. What these studies cannot tell us, of course, is the impact that a strategy will have on any individual student.
Why is that? Most studies on educational strategies compare a group of students who were taught using the strategy being researched and one group who was not. Study participants are given a pre-test and a post-test. The average difference in score changes is then used to determine effect-size. Because these are averages, however, they tell us how students tend to score overall, not how any particular student will do when taught with the given strategy.
Because of this, a conclusion such as “Note-taking has an effect size of d=0.99” should really be, “On average, note-taking has been found by some researchers to have a d=0.99 effect size for many students much of the time.” We must remember that students are individuals, not averages. As tempting as it is to base all our instructional decisions on something concrete like effect sizes, educators must also personalize instruction based on individual student differences.
In Summary
Educational research is a strong place to start when considering new instructional initiatives. Research can provide important conclusions that help us make decisions, but should not make the decisions for us. For instance, retaining students tends to, on average, decrease student achievement (Hattie, 2008), so we should not use grade retention as a widespread strategy. However, because students are individuals, we also cannot assume that study results apply to all students; some students, in some circumstances, may benefit from grade retention. For any study, we must ask ourselves: “What measures of effectiveness were used?” “How was the strategy implemented?” “Does this strategy fit our particular students in our particular situation?” Reading the research is a great place to start conversations that lead to actions, but we must commit ourselves to probing deeper to make the best choices for the young people we serve.
References
Gareis, C. R., & Grant, L. W. (2015). Teacher-made assessments: How to connect curriculum, instruction, and student learning. Routledge: Eye on Education.
Hattie, J. A. C. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.
Hattie, J. A. C., & Donoghue, G. M. (2016). Learning strategies: A synthesis and conceptual model. NPJ Science of Learning, 1, 1-13. https://pmc.ncbi.nlm.nih.gov/articles/PMC6380372/pdf/npjscilearn201613.pdf
Marzano, R. J., & Pickering, D. J. (2007). The case for and against homework. Educational Leeadership, 64(6), 74-79.