5 May 2005

New Zealand Scholarship - SSC: Key findings

Overall: The overall finding is that the difficulties experienced with the 2004 Scholarship resulted from inadequacies in both policy advice and implementation. Para 16

Organisational culture: NZQA was aware there would be variation, but was not aware that the nature of this variation would be a surprise to nearly everyone else. The 2004 approach can be contrasted with NZQA's previous experience with unscaled School Certificate, where a decision was made whether the aggregate results were within 'professional public tolerances'. Para 87 There was a climate of polarisation between assessment approaches. Scholarship requires a strong internal culture, particularly at NZQA, which welcomes different opinions about assessment so that robust policies and practices can be implemented. Para 100

Advice to Government: The Government received inadequate advice on policy risks associated with the 2004 Scholarship. A stocktake and analysis might well have led to a further refinement in the policy settings for the 2004 Scholarship, along the lines of those recently approved by the government. Para 17, 37

The government was not provided with adequate advice on the strategic risks associated with implementing both the high stakes Level 3 NCEA and Scholarship in the same year, 2004. Para 19

Risk analysis: Officials saw the variability of results between subjects as a consequence of the approach to Scholarship, rather than as a significant risk that could undermine the credibility of the examinations. Para 18

Officials were focused on operational risks and lost sight of the higher level implementation risks which impacted on outcomes. Para 19

The risks were not adequately taken into account in the approach to the 2004 Scholarship. The consequence was that teachers and some students were not well prepared for the 2004 Scholarship and the outcome was unfair to those students. Para 20

Teacher support and student guidance: Scholarship was still being implemented part way through the normal examination cycle; the registered standards were not available until December 2003, delaying the writing of exemplars; there was only one exemplar produced for each subject; there was no dedicated professional development of teachers on Scholarship. Para 20

Teachers felt not well prepared because of a combination of deficiencies in professional development. There appears to have been a general unawareness in the profession of what the Scholarship exams would require. Para 93

Planning and implementation: There was no comprehensive implementation plan for Scholarship developed by NZQA, with an analysis of implementation risks and strategies to deal with them, a communications plan and a plan to monitor progress. Para 28

Agency relationships: There appears to have been a degree of role confusion. NZQA saw some issues as operational in nature and did not involve the Ministry in their resolution. Some were in fact substantive policy issues and both agencies should have been involved. Para 31

Communication with sector: It was clear that in 2004 the variability of results would be more transparent than ever before, because the results of standards-based assessment are not scaled to achieve comparability across subjects or years. The Ministry and NZQA should have given far greater attention to preparing the education community and the public to understand it. Para 39

SSC: Key recommendations

A lead agency: That the Ministry of Education take the lead in ensuring all activities associated with the 2005 Scholarship are properly co-ordinated and deliver a consistent approach.

Risk identification: Operational policy matters and risks relating to the 2005 Scholarship and beyond be clearly identified by NZQA in consultation with the Ministry and conveyed to the Associate Minister of Education.

Better planning: NZQA develops a comprehensive plan for the 2005 Scholarship and beyond to be approved by the Associate Minister of Education.

Capacity building: The NZQA and the Ministry of Education build capacity over time within the secondary sector to widen the already stretch pool of examiners and markers.

An implementation team: NZQA sets up as a matter of urgency a dedicated implementation team for the 2005 Scholarship, with representation from the Ministry of Education, which has strong feedback loops with the sector and which is responsive to concerns raised.

Guidance for markers: NZQA, with the input of the Ministry of Education, establishes conventions that provide consistent guidance to marking panels of different subjects on how to award Scholarships.

Teacher resources: NZQA ensures the development of multiple exemplar resources for teachers to use in the 2005 school year including some with marked students' scripts.

Professional development: The Ministry of Education conducts two professional development days in 2005 and two in 2006, tailored specifically to the needs of Scholarship.

Timing changes: NZQA provides urgent advice to the Associate Minister of Education about the potential for an earlier start to the examinations and a later release of results than occurred for the 2004 Scholarship.

Informing the public: A concerted information campaign be conducted in 2005 to set realistic professional and public expectations for the release of results and to provide reassurance that the issues that emerged in 2004 have been addressed.

Information hot line: An examination "hot line" between agencies and the office of the Associate Minister of Education be maintained over the Christmas holiday period.

Summary of Scholarship and NCEA Reviews:

- In early February the Government identified that there was an unacceptable level of variability in the 2004 scholarship results.

- The Government believes that, while improvements can be made, on the whole the implementation of NCEA Levels 1-3 has gone well. Level 1 was implemented in 2002, level 2 in 2003, and level 3 in 2004. 2004 was the first year of full implementation of NCEA.

- The Government has sought assurances by way of the State Services Commission Review that observed variability in some achievement standards are not the result of any deficiencies in NZQA systems.

Note: While the National Certificate in Educational Achievement (NCEA) levels 1-3 is part of the National Qualifications Framework (NQF), New Zealand Scholarship is a separate qualification and is neither part of the NQF nor part of NCEA. Although New Zealand Scholarship examines the Year 13 curriculum, students were able to enter Scholarship without having entered NCEA level 3.

A summary of Government actions on these matters is as follows:
- Distinction Awards
-State Services Commission Review of Scholarship and NZQA
- NZQA Internal Review of Scholarship
- Scholarship Reference Group
- Secondary Leaders’ Forum

Distinction Awards
- On 8 February the Government announced the establishment of Distinction Awards to ensure that high performing students in subjects with unacceptably low scholarship pass rates were not disadvantaged.

- The Government also allowed extra time for students to submit their exams for review or reconsideration.

State Services Commission Review of Scholarship and NZQA
- On 15 February, the Associate Minister of Education David Benson-Pope wrote to the State Services Commissioner, asking him to use his powers pursuant to sections 6(b) and 11(4) of the State Sector Act 1988 to:
"review the effectiveness of the New Zealand Qualifications Authority in carrying out its functions relating to the school qualification system and in particular review its performance in managing the New Zealand Scholarship examination in 2004".

- The terms of reference of the SSC Review can be found at http://www.ssc.govt.nz/nzqa-tor-review

- The Review is entirely independent of the Minister.

- The SSC Review is in two parts. The first part concerns the 2004 Scholarship examinations.

- The second part of the review is a wider review of NZQA systems and processes. This part will be reported back to the State Services Commissioner by 31 July 2005.

- On 25 February, the Associate Minister formally wrote to the State Services Commissioner asking that:
"As part of the review, [he] wished to know whether or not the apparent variation in NCEA results both between subjects in a year, and when the 2004 results for a particular subject are compared to the results for that subject in earlier years (where there are such results), are a result of deficiencies in NZQA systems."

NZQA Internal Review of Scholarship
- The Board of NZQA is also conducting its own review of the 2004 Scholarship examinations.

Scholarship Reference Group
- On 15 February the Government announced the setting up of a group of education practitioners and experts to recommend on the shape and structure of New Zealand Scholarship for 2005 and beyond.

- On 30 March 2005 the government announced that it had endorsed a series of recommendations made by the Scholarship Reference Group.

Secondary Leaders’ Forum
- The Leaders’ Forum is a forum of secondary sector representatives. It provides the government with feedback, advice and direction from the education sector on the NCEA and its ongoing implementation.

- The Leaders' Forum has representatives from PPTA, secondary principal associations, School Trustees Association, Independent Schools, Teacher Educators, the NZ Vice Chancellors Committee, Wharekura and Deputy and Associate principals. It meets 3 to 4 times a year and is jointly chaired by the Chief Executives of the NZQA and Ministry of Education.

- The Leaders' Forum met on 10 and 11 March. The Minister asked the Forum to continue with the stocktake of the implementation of the NCEA initiated by Mr Mallard. The Minister confirmed that the Leaders' Forum has a mandate to guide and direct this process.

- The Minister identified three areas of priority, namely:
- the need to consider issues of variation and to generate greater confidence in moderation processes;
- support for, and communication with, schools, teachers, students and their parents; and
- professional development.

- The Minister also wanted the Forum to address a number of other issues, namely:
- removing the grade point average from students’ Interim Result Notices;
- non-reporting of non-achievement because of uncertainty over whether students seriously attempted a standard;
- creating a student-orientated easily accessible web-based source of information for all aspects of NCEA. Students, with the help of their parents, armed with correct information will be able to make well-informed choices for themselves;
- Addressing the issue of student motivation and school guidance over subject and credit selection.

- A sub-group of the Leader's Forum has been established to progress these issues in detail.

Recent Relevant Media Releases:

Minister introduces new distinction awards – 8 February 2005
Associate Education Minister David Benson-Pope has acted to ensure lower than expected pass rates in some scholarship subjects will not disadvantage students.

A Ministry of Education and NZQA report commissioned by Mr Benson-Pope has identified subjects where the number of scholarship passes is less than would have been expected based on the ability of the students involved. While a certain level of variability was expected under the new standards-based exam system, he says in some subjects that variability has gone beyond what the Government or the general public would consider acceptable.

"There is no one reason for this outcome," says Mr Benson-Pope. "But I take it very seriously and my first priority is making sure no students will be disadvantaged because of it – neither their future opportunities nor financial circumstances.

"For this reason, in those subjects where students have done exceptionally well at NCEA level 3, and the subject is under-represented at scholarship, we will be introducing a 'distinction certificate' in recognition of their achievement. At the same time there will be a new 'distinction award', meaning students who have a combination of scholarship passes and distinction certificates in any three scholarship subjects, will be eligible for an award of $1500 per year for three years.

"This new award, as well as a package of new awards which was announced last week, will mean around 215 more students will be eligible for financial honours. This largely mitigates the financial impact of the variance in the 2004 results."

Mr Benson-Pope says the New Zealand Qualifications Authority (NZQA) has already held discussions with universities around interpretation of the 2004 results.

"No tertiary courses exclude students who have not achieved scholarship results. But, scholarship students of any subject who feel they have been marked incorrectly or graded wrongly can resubmit their exam papers for reconsideration in the normal way."

Mr Benson-Pope says the Ministry of Education and NZQA report he received confirms that the level expected of our top scholars under this new exam system is extremely high.

"The New Zealand Scholarship system has not been designed to give a predetermined number of passes but rather to test students on a level considered to be comparable to first-year university study," said Mr Benson-Pope.

"Those standards were set by tertiary and secondary educators who developed the New Zealand Scholarship scheme to extend our very best students. It appears that in subjects like physics, chemistry and biology, those standards may be demanding more of students than is currently realistic.

"There are many factors that have come into play. Certainly it appears that one of the most significant factors influencing these results has been moving from a system that used scaling to produce the desired result, to one that does not. It is difficult to compare the old system with the new because the former measured relative performance, the new system measures absolute performance.

"The architects of the New Zealand Scholarship decided for many good reasons to move away from scaling. Our expectations may have to change under a system without scaling because we are not trying to reward students for a ranking but for achieving certain standards.

"Having said that, I believe the system can, and should, deliver more consistency between subjects."

Other reasons identified for the variability in results include:
- Some schools entering a relatively high proportion of their students in an exam designed for an elite group
- Variation in the specification of scholarship standards between subjects. For example, subjects like media studies and art history do not have an official national curriculum statement
- Some subjects, especially languages, have small numbers entered and are influenced by first language speakers or those who have been immersion students, which explains high performance rates
- Significant variability in the preparation of students for scholarship exams
- Students being unable to explain complex scientific ideas using appropriate language

Mr Benson-Pope says more work will be done to minimise the level of variance between subjects for those sitting scholarship exams in 2005. This will include NZQA and the Ministry of Education improving the information and support given to teachers to improve their preparation of students.

The Ministry of Education has already begun a review of the curriculum. In addition, a planned review of scholarship standards following this first year of implementation will go ahead as planned, although there is no intention to make these exams easier.

NZQA has also been told to enhance its early alert and analysis processes for future New Zealand Scholarship examinations.

Students get more time to submit Scholarship papers for review – 14 February 2005
Scholarship students have been given extra time to decide if they want to submit their exam papers for review or reconsideration, Associate Education Minister David Benson-Pope has announced.

Mr Benson-Pope says students initially had until tomorrow to send back papers. This has now been extended by an extra week, meaning the New Zealand Qualifications Authority will accept review or reconsideration applications up until 22 February.

Mr Benson-Pope says the existing refund mechanism will be used for students who decide to submit their paper for review or reconsideration.

"I have looked at this issue and am sure the appropriate way forward is for students who are concerned about the marking of their papers to resubmit them in the usual way," said Mr Benson-Pope. "I am particularly influenced by the fact that under the existing policy if mistakes are found in the marking of exam scripts then the application fee is refunded automatically.

"Regrettably, the call by some to have the fee waived entirely, may have given some students the impression that their papers might be remarked under different criteria or that grades would be adjusted as a result. This is not what the review and reconsideration process is for.

"The normal reconsideration process stands and is appropriate. However, if uncertainty had been caused by these calls for change students now have an extra week to decide if they would like to apply for reconsideration under the existing appeals policy," said Mr Benson-Pope.

The Minister says he has been advised that the review and reconsideration process normally takes between three and four weeks to complete. As with other years, people receiving awards will be formally notified by April.

Minister acts to secure future of scholarships process – 15 February 2005
The shape of this year's and future New Zealand Scholarship exams will be determined by a group of education practitioners and experts being assembled by Associate Education Minister David Benson-Pope.

The group will include school principals, teachers and a university representative. In addition, Mr Benson-Pope announced that he had invited the State Services Commission to review the performance of the New Zealand Qualifications Authority (NZQA).

Mr Benson-Pope today outlined the process and timeline of a review of the scholarship scheme to ensure there is no repeat of the significant variability that became apparent in the 2004 results. As a result of that variability the Government was forced to act in the interests of fairness, introducing a new "distinction award" to mitigate the effects of the inconsistency in the 2004 results.

Mr Benson-Pope says the focus must now shift to students sitting the New Zealand Scholarship this year and beyond.

The Minister says he will be inviting nominations of one representative from each of the following organisations to sit on the review group: the Secondary Schools Principals' Association, the New Zealand Vice Chancellors' Committee; the Post Primary Teachers Association (PPTA); the PPTA Principals' Council; the School Trustees' Association; and Independent Schools New Zealand. It will also include two additional principals nominated by the Minister and a person from the Wharekura (Maori secondary schools) area. They will be assisted by senior representatives from the Ministry of Education and NZQA.

Mr Benson-Pope says the expert group will have two weeks to prepare a report, including recommended options that he as Minister will take back to Cabinet: "It is not a lot of time but we have teachers and students in classrooms who need to know the shape of the exam system that will be used at the end of this year."

Mr Benson-Pope has also written to the State Services Commissioner asking him to review the performance of the New Zealand Qualifications Authority.

Mr Benson-Pope says he has written to the Commissioner, Dr Mark Prebble, asking him to use his powers pursuant to sections 6(b) and 11(4) of the State Sector Act 1988.

Minister names taskforce to secure future of New Zealand Scholarship – 18 February 2005
Associate Education Minister David Benson-Pope today named the education leaders who will determine the future shape of the New Zealand Scholarship exams.

The group is made up principals, teachers and educationalists representing the major organisations with an active interest in senior secondary education.

"I believe these are the people with the necessary skills, integrity and commitment to identify what changes are needed to ensure the results of the exam in 2005 and beyond will be less variable," says Mr Benson-Pope. "We will be taking action to ensure the New Zealand Scholarship delivers outcomes in line with public expectations."

The following people form the panel:
- Graham Young - Secondary Principals' Association of New Zealand (Tauranga Boys High School)
- Kate Gainsford - Post Primary Teachers' Association (Porirua College)
- Don McLeod - Post Primary Teachers' Association Principals' Council (Mt Hutt College)
- Tim Oughton - Independent Schools New Zealand (Scots College)
- Ray Newport - New Zealand School Trustees' Association
- Professor Luanna Meyer - New Zealand Vice Chancellors' Committee
- Wharekura representative (to be confirmed).
The Minister has also nominated three principals to join the panel. They are:
- John Morris - Auckland Grammar School
- Margaret McLeod - Wellington Girls' High School
- Roger Moses - Wellington College
In addition, Mr Benson-Pope has added two educationalists to the group for their expertise in statistics and examinations systems. They are:
- Professor Terry Crook – Co-director Education Assessment Research Unit, Otago University
- Dr John Langley – Dean of Education, Auckland University

Mr Benson-Pope confirmed that the group would meet Monday in Wellington and as earlier indicated, would have two weeks to report back.

Government responds to Scholarship Review - 30 March 2005
The government has endorsed a series of recommendations made by an expert panel looking at the New Zealand Scholarship exams.

Associate Education Minister David Benson-Pope says changes recommended by the Scholarship Reference Group will deliver the certainty the government, parents, and students require from New Zealand Scholarship.

"The government intervened earlier this year because scholarship exams are an important mechanism for recognising top academic performance, and as a means for distributing significant financial awards," said Mr Benson-Pope. "The unacceptable variability in the 2004 results created an obvious unfairness that had to be addressed."

Mr Benson-Pope says the key recommendation of the review group is that scholarship should be awarded to a set percentage of students in every subject. The review group recommends this percentage be set at a figure between two to three per cent of the total number of students studying a subject at NCEA Level 3.

A National Scholarship Monitoring Panel will be established to advise NZQA on the implementation of these recommendations. Proposed membership and terms of reference for the panel will be presented to Cabinet at the end of May. Final decisions on the target percentage will be made in conjunction with the panel.

Mr Benson-Pope says some departure from this target may be necessary for individual subjects to preserve the integrity of the exam. This might happen in subjects where there are small numbers of students taking a subject or where there are low scholarship entry levels.

"The recommended changes will require students to be ranked, and we have been assured that this is possible within a standards-based system providing the examination is structured appropriately," said Mr Benson-Pope.

Mr Benson-Pope says Cabinet has endorsed all but one of the reference group's 26 recommendations. It has deferred a decision on whether all students gaining a scholarship should get a financial award until the Ministry of Education look at the issue. They will report back by 30 May on this issue.

The minister paid tribute to the reference group: Hohepa Campbell, Terry Crooks, Kate Gainsford, John Langley, Margaret McLeod, Don McLeod, Luanna Meyer, John Morris, Roger Moses, Ray Newport, Tim Oughton, and Graham Young.

ENDS

© Scoop Media