5 May 2005
New Zealand Scholarship - SSC: Key findings
Overall: The overall finding is that the difficulties experienced with the 2004 Scholarship resulted from inadequacies
in both policy advice and implementation. Para 16
Organisational culture: NZQA was aware there would be variation, but was not aware that the nature of this variation
would be a surprise to nearly everyone else. The 2004 approach can be contrasted with NZQA's previous experience with
unscaled School Certificate, where a decision was made whether the aggregate results were within 'professional public
tolerances'. Para 87 There was a climate of polarisation between assessment approaches. Scholarship requires a strong
internal culture, particularly at NZQA, which welcomes different opinions about assessment so that robust policies and
practices can be implemented. Para 100
Advice to Government: The Government received inadequate advice on policy risks associated with the 2004 Scholarship. A
stocktake and analysis might well have led to a further refinement in the policy settings for the 2004 Scholarship,
along the lines of those recently approved by the government. Para 17, 37
The government was not provided with adequate advice on the strategic risks associated with implementing both the high
stakes Level 3 NCEA and Scholarship in the same year, 2004. Para 19
Risk analysis: Officials saw the variability of results between subjects as a consequence of the approach to
Scholarship, rather than as a significant risk that could undermine the credibility of the examinations. Para 18
Officials were focused on operational risks and lost sight of the higher level implementation risks which impacted on
outcomes. Para 19
The risks were not adequately taken into account in the approach to the 2004 Scholarship. The consequence was that
teachers and some students were not well prepared for the 2004 Scholarship and the outcome was unfair to those students.
Para 20
Teacher support and student guidance: Scholarship was still being implemented part way through the normal examination
cycle; the registered standards were not available until December 2003, delaying the writing of exemplars; there was
only one exemplar produced for each subject; there was no dedicated professional development of teachers on Scholarship.
Para 20
Teachers felt not well prepared because of a combination of deficiencies in professional development. There appears to
have been a general unawareness in the profession of what the Scholarship exams would require. Para 93
Planning and implementation: There was no comprehensive implementation plan for Scholarship developed by NZQA, with an
analysis of implementation risks and strategies to deal with them, a communications plan and a plan to monitor progress.
Para 28
Agency relationships: There appears to have been a degree of role confusion. NZQA saw some issues as operational in
nature and did not involve the Ministry in their resolution. Some were in fact substantive policy issues and both
agencies should have been involved. Para 31
Communication with sector: It was clear that in 2004 the variability of results would be more transparent than ever
before, because the results of standards-based assessment are not scaled to achieve comparability across subjects or
years. The Ministry and NZQA should have given far greater attention to preparing the education community and the public
to understand it. Para 39
SSC: Key recommendations
A lead agency: That the Ministry of Education take the lead in ensuring all activities associated with the 2005
Scholarship are properly co-ordinated and deliver a consistent approach.
Risk identification: Operational policy matters and risks relating to the 2005 Scholarship and beyond be clearly
identified by NZQA in consultation with the Ministry and conveyed to the Associate Minister of Education.
Better planning: NZQA develops a comprehensive plan for the 2005 Scholarship and beyond to be approved by the Associate
Minister of Education.
Capacity building: The NZQA and the Ministry of Education build capacity over time within the secondary sector to widen
the already stretch pool of examiners and markers.
An implementation team: NZQA sets up as a matter of urgency a dedicated implementation team for the 2005 Scholarship,
with representation from the Ministry of Education, which has strong feedback loops with the sector and which is
responsive to concerns raised.
Guidance for markers: NZQA, with the input of the Ministry of Education, establishes conventions that provide consistent
guidance to marking panels of different subjects on how to award Scholarships.
Teacher resources: NZQA ensures the development of multiple exemplar resources for teachers to use in the 2005 school
year including some with marked students' scripts.
Professional development: The Ministry of Education conducts two professional development days in 2005 and two in 2006,
tailored specifically to the needs of Scholarship.
Timing changes: NZQA provides urgent advice to the Associate Minister of Education about the potential for an earlier
start to the examinations and a later release of results than occurred for the 2004 Scholarship.
Informing the public: A concerted information campaign be conducted in 2005 to set realistic professional and public
expectations for the release of results and to provide reassurance that the issues that emerged in 2004 have been
addressed.
Information hot line: An examination "hot line" between agencies and the office of the Associate Minister of Education
be maintained over the Christmas holiday period.
Summary of Scholarship and NCEA Reviews:
- In early February the Government identified that there was an unacceptable level of variability in the 2004
scholarship results.
- The Government believes that, while improvements can be made, on the whole the implementation of NCEA Levels 1-3 has
gone well. Level 1 was implemented in 2002, level 2 in 2003, and level 3 in 2004. 2004 was the first year of full
implementation of NCEA.
- The Government has sought assurances by way of the State Services Commission Review that observed variability in some
achievement standards are not the result of any deficiencies in NZQA systems.
Note: While the National Certificate in Educational Achievement (NCEA) levels 1-3 is part of the National Qualifications
Framework (NQF), New Zealand Scholarship is a separate qualification and is neither part of the NQF nor part of NCEA.
Although New Zealand Scholarship examines the Year 13 curriculum, students were able to enter Scholarship without having
entered NCEA level 3.
A summary of Government actions on these matters is as follows:
- Distinction Awards
-State Services Commission Review of Scholarship and NZQA
- NZQA Internal Review of Scholarship
- Scholarship Reference Group
- Secondary Leaders’ Forum
Distinction Awards
- On 8 February the Government announced the establishment of Distinction Awards to ensure that high performing students
in subjects with unacceptably low scholarship pass rates were not disadvantaged.
- The Government also allowed extra time for students to submit their exams for review or reconsideration.
State Services Commission Review of Scholarship and NZQA
- On 15 February, the Associate Minister of Education David Benson-Pope wrote to the State Services Commissioner, asking
him to use his powers pursuant to sections 6(b) and 11(4) of the State Sector Act 1988 to:
"review the effectiveness of the New Zealand Qualifications Authority in carrying out its functions relating to the
school qualification system and in particular review its performance in managing the New Zealand Scholarship examination
in 2004".
- The terms of reference of the SSC Review can be found at http://www.ssc.govt.nz/nzqa-tor-review
- The Review is entirely independent of the Minister.
- The SSC Review is in two parts. The first part concerns the 2004 Scholarship examinations.
- The second part of the review is a wider review of NZQA systems and processes. This part will be reported back to the
State Services Commissioner by 31 July 2005.
- On 25 February, the Associate Minister formally wrote to the State Services Commissioner asking that:
"As part of the review, [he] wished to know whether or not the apparent variation in NCEA results both between subjects
in a year, and when the 2004 results for a particular subject are compared to the results for that subject in earlier
years (where there are such results), are a result of deficiencies in NZQA systems."
NZQA Internal Review of Scholarship
- The Board of NZQA is also conducting its own review of the 2004 Scholarship examinations.
Scholarship Reference Group
- On 15 February the Government announced the setting up of a group of education practitioners and experts to recommend
on the shape and structure of New Zealand Scholarship for 2005 and beyond.
- On 30 March 2005 the government announced that it had endorsed a series of recommendations made by the Scholarship
Reference Group.
Secondary Leaders’ Forum
- The Leaders’ Forum is a forum of secondary sector representatives. It provides the government with feedback, advice
and direction from the education sector on the NCEA and its ongoing implementation.
- The Leaders' Forum has representatives from PPTA, secondary principal associations, School Trustees Association,
Independent Schools, Teacher Educators, the NZ Vice Chancellors Committee, Wharekura and Deputy and Associate
principals. It meets 3 to 4 times a year and is jointly chaired by the Chief Executives of the NZQA and Ministry of
Education.
- The Leaders' Forum met on 10 and 11 March. The Minister asked the Forum to continue with the stocktake of the
implementation of the NCEA initiated by Mr Mallard. The Minister confirmed that the Leaders' Forum has a mandate to
guide and direct this process.
- The Minister identified three areas of priority, namely:
- the need to consider issues of variation and to generate greater confidence in moderation processes;
- support for, and communication with, schools, teachers, students and their parents; and
- professional development.
- The Minister also wanted the Forum to address a number of other issues, namely:
- removing the grade point average from students’ Interim Result Notices;
- non-reporting of non-achievement because of uncertainty over whether students seriously attempted a standard;
- creating a student-orientated easily accessible web-based source of information for all aspects of NCEA. Students,
with the help of their parents, armed with correct information will be able to make well-informed choices for
themselves;
- Addressing the issue of student motivation and school guidance over subject and credit selection.
- A sub-group of the Leader's Forum has been established to progress these issues in detail.
Recent Relevant Media Releases:
Minister introduces new distinction awards – 8 February 2005
Associate Education Minister David Benson-Pope has acted to ensure lower than expected pass rates in some scholarship
subjects will not disadvantage students.
A Ministry of Education and NZQA report commissioned by Mr Benson-Pope has identified subjects where the number of
scholarship passes is less than would have been expected based on the ability of the students involved. While a certain
level of variability was expected under the new standards-based exam system, he says in some subjects that variability
has gone beyond what the Government or the general public would consider acceptable.
"There is no one reason for this outcome," says Mr Benson-Pope. "But I take it very seriously and my first priority is
making sure no students will be disadvantaged because of it – neither their future opportunities nor financial
circumstances.
"For this reason, in those subjects where students have done exceptionally well at NCEA level 3, and the subject is
under-represented at scholarship, we will be introducing a 'distinction certificate' in recognition of their
achievement. At the same time there will be a new 'distinction award', meaning students who have a combination of
scholarship passes and distinction certificates in any three scholarship subjects, will be eligible for an award of
$1500 per year for three years.
"This new award, as well as a package of new awards which was announced last week, will mean around 215 more students
will be eligible for financial honours. This largely mitigates the financial impact of the variance in the 2004
results."
Mr Benson-Pope says the New Zealand Qualifications Authority (NZQA) has already held discussions with universities
around interpretation of the 2004 results.
"No tertiary courses exclude students who have not achieved scholarship results. But, scholarship students of any
subject who feel they have been marked incorrectly or graded wrongly can resubmit their exam papers for reconsideration
in the normal way."
Mr Benson-Pope says the Ministry of Education and NZQA report he received confirms that the level expected of our top
scholars under this new exam system is extremely high.
"The New Zealand Scholarship system has not been designed to give a predetermined number of passes but rather to test
students on a level considered to be comparable to first-year university study," said Mr Benson-Pope.
"Those standards were set by tertiary and secondary educators who developed the New Zealand Scholarship scheme to extend
our very best students. It appears that in subjects like physics, chemistry and biology, those standards may be
demanding more of students than is currently realistic.
"There are many factors that have come into play. Certainly it appears that one of the most significant factors
influencing these results has been moving from a system that used scaling to produce the desired result, to one that
does not. It is difficult to compare the old system with the new because the former measured relative performance, the
new system measures absolute performance.
"The architects of the New Zealand Scholarship decided for many good reasons to move away from scaling. Our expectations
may have to change under a system without scaling because we are not trying to reward students for a ranking but for
achieving certain standards.
"Having said that, I believe the system can, and should, deliver more consistency between subjects."
Other reasons identified for the variability in results include:
- Some schools entering a relatively high proportion of their students in an exam designed for an elite group
- Variation in the specification of scholarship standards between subjects. For example, subjects like media studies and
art history do not have an official national curriculum statement
- Some subjects, especially languages, have small numbers entered and are influenced by first language speakers or those
who have been immersion students, which explains high performance rates
- Significant variability in the preparation of students for scholarship exams
- Students being unable to explain complex scientific ideas using appropriate language
Mr Benson-Pope says more work will be done to minimise the level of variance between subjects for those sitting
scholarship exams in 2005. This will include NZQA and the Ministry of Education improving the information and support
given to teachers to improve their preparation of students.
The Ministry of Education has already begun a review of the curriculum. In addition, a planned review of scholarship
standards following this first year of implementation will go ahead as planned, although there is no intention to make
these exams easier.
NZQA has also been told to enhance its early alert and analysis processes for future New Zealand Scholarship
examinations.
Students get more time to submit Scholarship papers for review – 14 February 2005
Scholarship students have been given extra time to decide if they want to submit their exam papers for review or
reconsideration, Associate Education Minister David Benson-Pope has announced.
Mr Benson-Pope says students initially had until tomorrow to send back papers. This has now been extended by an extra
week, meaning the New Zealand Qualifications Authority will accept review or reconsideration applications up until 22
February.
Mr Benson-Pope says the existing refund mechanism will be used for students who decide to submit their paper for review
or reconsideration.
"I have looked at this issue and am sure the appropriate way forward is for students who are concerned about the marking
of their papers to resubmit them in the usual way," said Mr Benson-Pope. "I am particularly influenced by the fact that
under the existing policy if mistakes are found in the marking of exam scripts then the application fee is refunded
automatically.
"Regrettably, the call by some to have the fee waived entirely, may have given some students the impression that their
papers might be remarked under different criteria or that grades would be adjusted as a result. This is not what the
review and reconsideration process is for.
"The normal reconsideration process stands and is appropriate. However, if uncertainty had been caused by these calls
for change students now have an extra week to decide if they would like to apply for reconsideration under the existing
appeals policy," said Mr Benson-Pope.
The Minister says he has been advised that the review and reconsideration process normally takes between three and four
weeks to complete. As with other years, people receiving awards will be formally notified by April.
Minister acts to secure future of scholarships process – 15 February 2005
The shape of this year's and future New Zealand Scholarship exams will be determined by a group of education
practitioners and experts being assembled by Associate Education Minister David Benson-Pope.
The group will include school principals, teachers and a university representative. In addition, Mr Benson-Pope
announced that he had invited the State Services Commission to review the performance of the New Zealand Qualifications
Authority (NZQA).
Mr Benson-Pope today outlined the process and timeline of a review of the scholarship scheme to ensure there is no
repeat of the significant variability that became apparent in the 2004 results. As a result of that variability the
Government was forced to act in the interests of fairness, introducing a new "distinction award" to mitigate the effects
of the inconsistency in the 2004 results.
Mr Benson-Pope says the focus must now shift to students sitting the New Zealand Scholarship this year and beyond.
The Minister says he will be inviting nominations of one representative from each of the following organisations to sit
on the review group: the Secondary Schools Principals' Association, the New Zealand Vice Chancellors' Committee; the
Post Primary Teachers Association (PPTA); the PPTA Principals' Council; the School Trustees' Association; and
Independent Schools New Zealand. It will also include two additional principals nominated by the Minister and a person
from the Wharekura (Maori secondary schools) area. They will be assisted by senior representatives from the Ministry of
Education and NZQA.
Mr Benson-Pope says the expert group will have two weeks to prepare a report, including recommended options that he as
Minister will take back to Cabinet: "It is not a lot of time but we have teachers and students in classrooms who need to
know the shape of the exam system that will be used at the end of this year."
Mr Benson-Pope has also written to the State Services Commissioner asking him to review the performance of the New
Zealand Qualifications Authority.
Mr Benson-Pope says he has written to the Commissioner, Dr Mark Prebble, asking him to use his powers pursuant to
sections 6(b) and 11(4) of the State Sector Act 1988.
Minister names taskforce to secure future of New Zealand Scholarship – 18 February 2005
Associate Education Minister David Benson-Pope today named the education leaders who will determine the future shape of
the New Zealand Scholarship exams.
The group is made up principals, teachers and educationalists representing the major organisations with an active
interest in senior secondary education.
"I believe these are the people with the necessary skills, integrity and commitment to identify what changes are needed
to ensure the results of the exam in 2005 and beyond will be less variable," says Mr Benson-Pope. "We will be taking
action to ensure the New Zealand Scholarship delivers outcomes in line with public expectations."
The following people form the panel:
- Graham Young - Secondary Principals' Association of New Zealand (Tauranga Boys High School)
- Kate Gainsford - Post Primary Teachers' Association (Porirua College)
- Don McLeod - Post Primary Teachers' Association Principals' Council (Mt Hutt College)
- Tim Oughton - Independent Schools New Zealand (Scots College)
- Ray Newport - New Zealand School Trustees' Association
- Professor Luanna Meyer - New Zealand Vice Chancellors' Committee
- Wharekura representative (to be confirmed).
The Minister has also nominated three principals to join the panel. They are:
- John Morris - Auckland Grammar School
- Margaret McLeod - Wellington Girls' High School
- Roger Moses - Wellington College
In addition, Mr Benson-Pope has added two educationalists to the group for their expertise in statistics and
examinations systems. They are:
- Professor Terry Crook – Co-director Education Assessment Research Unit, Otago University
- Dr John Langley – Dean of Education, Auckland University
Mr Benson-Pope confirmed that the group would meet Monday in Wellington and as earlier indicated, would have two weeks
to report back.
Government responds to Scholarship Review - 30 March 2005
The government has endorsed a series of recommendations made by an expert panel looking at the New Zealand Scholarship
exams.
Associate Education Minister David Benson-Pope says changes recommended by the Scholarship Reference Group will deliver
the certainty the government, parents, and students require from New Zealand Scholarship.
"The government intervened earlier this year because scholarship exams are an important mechanism for recognising top
academic performance, and as a means for distributing significant financial awards," said Mr Benson-Pope. "The
unacceptable variability in the 2004 results created an obvious unfairness that had to be addressed."
Mr Benson-Pope says the key recommendation of the review group is that scholarship should be awarded to a set percentage
of students in every subject. The review group recommends this percentage be set at a figure between two to three per
cent of the total number of students studying a subject at NCEA Level 3.
A National Scholarship Monitoring Panel will be established to advise NZQA on the implementation of these
recommendations. Proposed membership and terms of reference for the panel will be presented to Cabinet at the end of
May. Final decisions on the target percentage will be made in conjunction with the panel.
Mr Benson-Pope says some departure from this target may be necessary for individual subjects to preserve the integrity
of the exam. This might happen in subjects where there are small numbers of students taking a subject or where there are
low scholarship entry levels.
"The recommended changes will require students to be ranked, and we have been assured that this is possible within a
standards-based system providing the examination is structured appropriately," said Mr Benson-Pope.
Mr Benson-Pope says Cabinet has endorsed all but one of the reference group's 26 recommendations. It has deferred a
decision on whether all students gaining a scholarship should get a financial award until the Ministry of Education look
at the issue. They will report back by 30 May on this issue.
The minister paid tribute to the reference group: Hohepa Campbell, Terry Crooks, Kate Gainsford, John Langley, Margaret
McLeod, Don McLeod, Luanna Meyer, John Morris, Roger Moses, Ray Newport, Tim Oughton, and Graham Young.
ENDS