Friday, December 9, 2016

Who's telling the truth about M'sia's Pisa 2015 scores?

Published     Updated

Yesterday, DAP's Serdang MP Ong Kian Ming claimed Malaysia was not featured in the rankings for Mathematics, Reading and Science in the Programme for International Student Assessment 2015 (Pisa 2015) report.

Then, the Education Ministry issued a statement, insisting that Malaysia did indeed appear in the report.

So, who's telling the truth and who's not?

In simple words, both are correct: Malaysia did appear in the Pisa 2015 report, but it was not ranked with the other countries due to bungled data submitted.

Here's a breakdown of what happened.

How did it start?

It began with Education Ministry deputy director-general Amin Senin proudly announcing that Malaysia had improved its performance in Pisa 2015 on Wednesday.

He pointed out Malaysia's score for Mathematics had increased from 404 to 421 while the score for Reading improved from 398 to 414 and Science from 420 to 422 as compared to Pisa 2012.

What did DAP MP Ong Kian Ming say?

Ong poured cold water on the boast, claiming that Malaysia was not ranked as Malaysia's data only had a 51 percent response rate from schools, therefore it did not meet the minimum 85 percent requirement by Pisa.

He also speculated that the 51 percent response rate may be skewed to high performing schools, therefore giving an impression that Malaysia's performance had improved even though the data is not representative of the whole country.

How did the Education Ministry respond?

The Education Ministry pointed out that the Pisa 2015 results, published by Organisation for Economic Cooperation and Development (OECD), appeared in the Pisa 2015 Initial Report Volume 1 (Excellence and Equity in Education) and Volume 2 (Policies and Practices in Excellent Schools).
"Malaysia's results were reported in both the Pisa 2015 reports.

"Malaysia's results can be found in the figures of the report as well as in the tables in the annexes," it said in a statement.

A check by Malaysiakini found that Malaysia was absent from page 44 of the Pisa 2015 Results Volume 1 (Excellence and Equity in Education), which displayed the mean score for Science, Reading and Mathematics.

Malaysia did receive a mean score but when it appeared in figures from page 320 to page 442, it was isolated in a separate category because the data could not be used for comparison.

The report explained in Annex A4 on page 304, which concerned quality assurance of the data.
It stated: "In Malaysia, the Pisa assessment was conducted in accordance with the operational standards and guidelines of the OECD.

"However, the weighted response rate among the initially sampled Malaysian schools (51 percent) falls well short of the standard Pisa response rate of 85 percent.

"Therefore, the results may not be comparable to those of other countries or to results for Malaysia from previous years."

You can find the report here.

So, the Education Ministry was also right that Malaysia did appear in the report.
But Malaysia's score, as clearly stated in the report, may not be comparable to previous results so the Education Ministry's boast of improving performance is not accurate.
The Education Ministry in its statement did acknowledge that its data to Pisa only achieved the required school response rate.

But it stated that the data for students was accepted as 8,661 of students out of a sample of 9,660 students responded and it was therefore considered representative.

It added that OECD had identified weaknesses in the computer-based reporting system and improvements would be introduced for the Pisa 2018 report.

No comments: