recentpopularlog in

robertogreco : visiblelearning   9

The power of ‘evidence’: Reliable science or a set of blunt tools? - Wrigley - 2018 - British Educational Research Journal - Wiley Online Library
"In response to the increasing emphasis on ‘evidence‐based teaching’, this article examines the privileging of randomised controlled trials and their statistical synthesis (meta‐analysis). It also pays particular attention to two third‐level statistical syntheses: John Hattie's Visible learning project and the EEF's Teaching and learning toolkit. The article examines some of the technical shortcomings, philosophical implications and ideological effects of this approach to ‘evidence’, at all these three levels. At various points in the article, aspects of critical realism are referenced in order to highlight ontological and epistemological shortcomings of ‘evidence‐based teaching’ and its implicit empiricism. Given the invocation of the medical field in this debate, it points to critiques within that field, including the need to pay attention to professional experience and clinical diagnosis in specific situations. Finally, it briefly locates the appeal to ‘evidence’ within a neoliberal policy framework."
johnhattie  2018  teaching  howweteach  education  policy  evidence  visiblelearning  evidence-basedteaching  criticism  learning 
july 2018 by robertogreco
HOW TO ENGAGE IN PSEUDOSCIENCE WITH REAL DATA: A CRITICISM OF JOHN HATTIE’S ARGUMENTS IN VISIBLE LEARNING FROM THE PERSPECTIVE OF A STATISTICIAN | Bergeron | McGill Journal of Education / Revue des sciences de l'éducation de McGill
"The work of John Hattie on education contains, seemingly, the most comprehensive synthesis of existing research in the field. Many consider his book, Visible Learning (Hattie, 2008), to be a Bible or a Holy Grail: “When this work was published, certain commentators described it as the Holy Grail of education, which is without a doubt not too much of a hyperbole” (Baillargeon, 2014, para. 13).

For those who are unaccustomed to dissecting numbers, such a synthesis does seem to represent a colossal and meticulous task, which in turn gives the impression of scientific validity. For a statistician familiar with the scientific method, from the elaboration of research questions to the interpretation of analyses, appearances, however, are not sufficient. According to the legend, the Holy Grail is kept in the elusive castle of the Fisher King. When taking the necessary in-depth look at Visible Learning with the eye of an expert, we find not a mighty castle but a fragile house of cards that quickly falls apart. This article offers a critical analysis of the methodology used by Hattie from the point of view of a statistician. We can spin stories from real data in an effort to communicate results to a wider audience, but these stories should not fall into the realm of fiction. We must therefore absolutely qualify Hattie’s methodology as pseudoscience. The researcher from New Zealand obviously has laudable intentions, which we describe first and foremost. Good intentions, nevertheless, do not prevent major errors in Visible Learning — errors which we will discuss afterwards. The analysis process then leads to a list of questions researchers should ask themselves when examining studies and enquiries based on data analyses, including meta-analyses. Afterwards, in an effort to better understand, we give concrete examples that demonstrate how Cohen’s d (Hattie’s measure of effect size) simply cannot be used as a universal measure of impact. Finally, to ensure that our quest does not remain unfinished, we offer pathways of solutions with the objective of demystifying and encouraging the correct usage of statistics in the field of education."
johnhattie  research  visiblelearning  2017  education  pedagogy  teaching  schools  learning 
may 2018 by robertogreco
Invisible Learnings: A commentary on John Hattie's book visible learning: A synthesis of over 800 metaanalyses relating to achievement.
[via: https://twitter.com/alfiekohn/status/996382167772160000 ]

"This book by Professor John Hattie of Auckland University is the result of decades of careful research. He has synthesised some 800 meta-analyses comprising more than 50,000 studies and involving some 146,000 ‘effect sizes’. The announcement of the book has already led to a good deal of discussion both in New Zealand and overseas and seems to have captured the attention of policy makers. It is, therefore, important that members of the educational research community pay John Hattie the courtesy of subjecting his conclusions to critical scrutiny in a spirit of mutual truth seeking to ensure that: (1) discussions are based on a careful reading of the book, rather than on half-baked ‘reactions’ in the popular media; (2) the caveats which Hattie himself sets out are carefully noted so that decisions are not made in opposition to the message of this book and (3) the findings are not ‘appropriated’ by political and ideological interests and used in ways which the data do not substantiate."



"In conclusion, we want to repeat our belief that John Hattie’s book makes a significant contribution to understanding the variables surrounding successful teaching and think that it is a very useful resource for teacher education. We are concerned, however, that:

(i) despite his own frequent warnings, politicians may use his work to justify policies which he does not endorse and his research does not sanction;

(ii) teachers and teacher educators might try to use the findings in a simplistic way and not, as Hattie wants, as a source for “hypotheses for intelligent problem solving”;

(iii) the quantitative research on ‘school effects’ might be presented in isolation from the historical, cultural and social contexts, and their interaction with home and community backgrounds; and

(iv) there may be insufficient discussion about the aims of education and the purposes of schooling without which the studies have little point.

It is important that students preparing for teaching learn about the research process and how easily it leads to error rather than truth. They need to respect research but be acutely aware of its limitations. The research that they need to know about goes beyond what happens in schools and classrooms. As this review has shown, what students bring from their social class, family, culture, home background and prior experiences is more important than what happens in the school, even though what happens in the school (particularly what teachers are and do) is very important. The secret of school improvement lies in the recognition of these factors and their integration into a social, economic and educational programme."
johnhattie  visiblelearning  education  2009  learning  pedagogy  teaching  schools 
may 2018 by robertogreco
School leadership and the cult of the guru: the neo-Taylorism of Hattie: School Leadership & Management: Vol 37, No 4
"As one of the central institutions of society, schooling is subject to significant public interest and scrutiny. Fads and fashion successfully developed elsewhere are often rebadged and relaunched in education for the purpose of improvement. Such interventions are often quickly identified as intruders and frequently fade into obscurity, but what of internal interventions, the education research that becomes widely accepted and promoted? In this paper I argue that contemporary thought and analysis in Australian school leadership has submitted to the cult of the guru. Specifically, I contend that dialogue (much less debate) has settled on the work of John Hattie’s meta-meta-analysis giving rise to the Cult of Hattie. This paper is not an attack on Hattie as a person, or even his work, rather an argument about the conditions which have facilitated the rise of a guru. I argue that the uncritical acceptance and proliferation of this cult is a tragedy for Australian school leadership."
johnhattie  neo-taylorism  taylorism  2017  meta-analysis  education  pedagogy  teaching  learning  schools  visiblelearning 
may 2018 by robertogreco
Update: Opportunity Knocks Again, And Again, And Again … / Reflections of a Reluctant Writer
"What interests me is the work that has been done to shine a spotlight on the short-comings of using meta-analysis and effect sizes to validate all manner of commercial and educational activity and supposed policy legitimacy. For example, back in 2011 Snook et al wrote a critique of Visible Learning. Of particular note were their concluding concerns. After picking apart the methodological inconsistencies, the authors noted that “politicians may use his work to justify policies which he (Hattie) does not endorse and his research does not sanction”. They go on to state that “the quantitative research on ‘school effects’ might be presented in isolation from their historical, cultural and social contexts, and their interaction with home and community backgrounds”.

Beyond a schools choice to adopt strategies which anchor themselves in meta-analysis, there is the bigger question of how far up the system chain does the acceptance of intervention effectiveness go and how wide does the sphere of influence extend? Simpson (2017) has noted that our preoccupation with “‘what works’ in education has led to an emphasis on developing policy from evidence based on comparing and combining a particular statistical summary of intervention studies: the standardised effect size.” The paper suggests that research areas which lead to the array of effective interventions are susceptible to research design manipulation – they stand out because of methodological choices. It also asserts that policy has fallen victim to metricophilia: “the unjustified faith in numerical quantities as having particularly special status as ‘evidence’ (Smith 2011)”. Dr Gary Jones does a great job of highlighting this and other worries in his blog post about how this paper puts another ‘nail in the coffin’ of Hattie’s Visible Learning. Similarly, Ollie Orange ably dismantles the statistical concerns of Hattie’s meta-analysis.

The seductive rhetoric of Hattie’s work can be found almost everywhere and certainly seems compelling. With questions being asked of the methodological credibility upon which all else gushes forth, shouldn’t we be questioning how much we buy in to it? Surely we cannot ignore the noise, not necessarily because of its message, but because the noise is becoming a cacophony. As Eacott (2017) concludes,
“Hattie’s work is everywhere in contemporary Australian school leadership. This is not to say that educators have no opportunity for resistance, but the presence and influence of brand Hattie cannot be ignored. The multiple partnerships and roles held by Hattie the man and the uptake of his work by systems and professional associations have canonised the work in contemporary dialogue and debate to the extent that it is now put forth as the solution to many of the woes of education.”
"
jonandrews  2017  johnhattie  meta-analysis  policy  education  schools  research  manipulation  garyjones  ollieorange  neo-taylorism  scotteacott  edugurus  cultofpersonality  skepticism  visiblelearning  measurement  certainty  uncertainty  silverbullets  ivansnook  johno'neill  johnclark  anne-marieo'neill  rogeropenshaw 
may 2017 by robertogreco
John Hattie admits that half of the Statistics in Visible Learning are wrong | ollieorange2
"At the researchED conference in September 2013, Professor Robert Coe, Professor of Education at Durham University, said that John Hattie’s book, ‘Visible Learning’,  is “riddled with errors”. But what are some of those errors?

The biggest mistake Hattie makes is with the CLE statistic that he uses throughout the book. In ‘Visible Learning, Hattie only uses two statistics, the ‘Effect Size’ and the CLE (neither of which Mathematicians use).

The CLE is meant to be a probability, yet Hattie has it at values between -49% and 219%. Now a probability can’t be negative or more than 100% as any Year 7 will tell you.

This was first spotted and pointed out to him by Arne Kare Topphol, an Associate Professor at the University of Volga and his class who sent Hattie an email.

In his first reply –  here , Hattie completely misses the point about probability being negative and claims he actually used a different version of the CLE than the one he actually referenced (by McGraw and Wong). This makes his academic referencing, hmm, the word I’m going to use here is ‘interesting’.

In his second reply –  here , Hattie reluctantly acknowledges that the CLE has in fact been calculated incorrectly throughout the book but brushes it off as no big deal that out of two statistics in the book he has calculated one incorrectly.

There are several worrying aspects to this -

Firstly, it took 3 years for the mistake to be noticed, and it’s not as though it’s a subtle statistical error that only a Mathematician would spot, he has probability as negative for goodness sake. Presumably, the entire Educational Research community read the book when it came out and they all completely missed it. So, the question must be asked, who is checking John Hattie’s work? As a Bachelor of Arts is he capable of spotting Mathematical errors himself?

In Mathematics, new or unproven work is handed over to unbiased judges who go through it with a fine toothcomb before it is considered to have the stamp of approval of the Mathematical community. Who is performing this function for the Educational community?

Secondly, despite the fact that John Hattie has presumably known about this error since last year there has been no publicity telling people that part of the book is wrong and should not be used. Surely he could have found time between flying round the world to his many Visible Learning conferences to squeeze in a quick announcement.

As one of the letter writer’s stepfather, a Professor of Statistics said

“People who don’t know that Probability can’t be negative, shouldn’t write books on Statistics”"

[See also: http://literacyinleafstrewn.blogspot.co.uk/2012/12/can-we-trust-educational-research_20.html ]

[Part 2: http://ollieorange2.wordpress.com/2014/09/24/half-of-the-statistics-in-visible-learning-are-wrong-part-2/

"In an earlier post we discovered that John Hattie had admitted (quietly) that half of the Statistics in Visible Learning were incorrect. John Hattie uses two statistics in the book, the ‘Effect Size’ and the ‘CLE’. All of the CLEs are wrong through-out the book.

Now, I didn’t really know why they were wrong, I thought, maybe he was using a computer program to calculate them and it had been set up incorrectly. I didn’t know. Until I received this comment from Per-Daniel Liljegren. He was giving a seminar on Visible Learning for some teachers in Sweden and didn’t understand some of what he’d found, so, he wrote to Debra Masters, Director of Visible Learning Plus, asking for help.

“Now, when preparing the very first seminar, I was very puzzled over the CLEs in Hattie’s Visible Learning. It seems to me that most of the CLEs are simply the Effect Size, d, divided by the square root of 2.

Should not the CLE be some integral from -infinity to d divided by the square root of 2?”

And if you grab your copy of Visible Learning and check, he’s right! The CLEs are just the Effect Size divided by the square root of 2.

He never received a reply to his letter.

If we look at this article that tells us how to calculate the CLE – How to calculate the Common Language Effect Size Statistic - we see that dividing by the square root of 2 actually finds the z value. This should then have been converted into the probability using standard Normal distribution tables, a very basic statistical technique that we teach to Year 12s in S1 A Level.

Throughout the book, Visible Learning, John Hattie has calculated the z values and used them as his CLEs when he should have converted them into probabilities.

Three very worrying things about all this -

1. John Hattie doesn’t know the difference between z values and the probabilities you get from z values. Really, really, basic stuff.

2. John Hattie knows about this mistake but has chosen not to publicise it. This could mean that many teachers are still relying on it to instruct their teaching.

2. No-one picked up on it for years, despite the fact the CLE is meant to be a probability. So, throughout the book he is saying that probability can be negative or more than 100%. So, who is checking John Hattie’s work? Because the academic educational establishment doesn’t appear to be.

Again we are left with two options to choose from

1. John Hattie is a genius who is doing things that even Mathematicians don’t understand.

2. John Hattie is a well meaning man with a Social Sciences degree who has made a mistake in using statistical techniques he didn’t realise were unknown to Mathematicians and incorrect.

The choice is yours."]
johnhattie  visiblelearning  education  teaching  statistsics  research  math  mathematics  2014 
november 2014 by robertogreco
Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement (Paperback) - Routledge
"This unique and ground-breaking book is the result of 15 years research and synthesises over 800 meta-analyses on the influences on achievement in school-aged students. It builds a story about the power of teachers, feedback, and a model of learning and understanding. The research involves many millions of students and represents the largest ever evidence based research into what actually works in schools to improve learning. Areas covered include the influence of the student, home, school, curricula, teacher, and teaching strategies. A model of teaching and learning is developed based on the notion of visible teaching and visible learning.

A major message is that what works best for students is similar to what works best for teachers – an attention to setting challenging learning intentions, being clear about what success means, and an attention to learning strategies for developing conceptual understanding about what teachers and students know and understand…"
johnhattie  education  learning  teaching  schools  practice  meaning  challenge  success  attention  strategy  curriculum  visiblelearning  via:cervus  books  routledgeinternational  toread 
february 2011 by robertogreco

Copy this bookmark:





to read