Recently the annual rankings for Ontario elementary and secondary schools were published by the Fraser Institute. The key message of this post is any published ranking of schools is too good to be true.

First of all, the Fraser Institute does not employ the psychometric expertise or educational measurement professionals that are capable of formulating some educated, informative, and most of all accurate rating system that takes into account the myriad of variables that would play into student achievement – one of the key outcome measures the Fraser Institute utilizes in their rankings.

Second, any credible psychometrician or educational measurement professional would not put their names on any system that attempts to rank schools based on heavy reliance on parent income and single-sex environment variables. These variables, fairly easy to measure, are difficult to account for in statistical models, without the proper use of multilevel statistical models, using hierarchical linear modelling (HLM) software or the like.

Even with the expertise, and the software, there is much well-founded debate in the global educational measurement community as to what variables should be included in models that attempt to account for student achievement. One of the larger contributors to student achievement, is the classroom. In much academic research on school effectiveness, classroom effects on student achievement are greater than school effects. The Fraser Institute tries to account for school effect with the parent income and single-sex environment variables, but how do they measure classroom effect? The answer is simple… they don’t. They don’t because they can’t! There are no instruments or tools currently in use throughout the province, or even in a suitable sample of schools, that measure accurately what goes on inside the classroom in terms of pedagogy, time on task, opportunity to learn and other key variables. Also, we are unsure that even the parent income and single-sex environment variables used by the Fraser Institute are used in a hierarchical or multilevel statistical model to properly account for school effect.

Without even looking at the current rankings, as I wouldn’t waste my time, you can bet that private schools (for which parent income tends to be higher than average) which offer single-sex environments (especially those just for girls) do well on the rankings. Well, no kidding!! Almost 50 years of credible educational research tells us that these variables as well as parents’ education, and other socio-economic factors contribute to higher student achievement. This is old news. Newer approaches to looking at student achievement use the aforementioned multilevel statistical models in a value-added approach. Value-added looks at socio-economic factors, and basically asks, “Given the socio-economic status (SES) of the school, what other factors contribute to student achievement?”

But the Fraser Institute does not take a value-added approach to their rankings. Look at schools that don’t have high parent incomes, have largely transient populations and have a great many learners for which English is not their first language. These schools would score low on the Fraser Institute’s rankings, but some superior teaching is going on in some of these schools. The schools and their teachers add value to the learning environment. They are truly the strong schools. The aforementioned rankings do not account for that.

Let’s give credit where credit is due now. The Ontario Ministry of Education has identified “Schools on the Move,” schools with difficult environments, but have higher than typical student achievement. The Ministry has recently put out an RFP to study what is going on these schools, who have the decked stacked against them, but are still producing high student achievement. This research will likely shed light on important classroom factors that contribute to student success. I would suggest that the Fraser Institute read the findings of this study to further inform their currently misguided ranking system.