National Standards: School Sample Monitoring and Evaluation
Report:
NationalStandardsSchoolSampleMonitoringandEvaluationProject201012Part1.pdf
NationalStandardsSchoolSampleMonitoringandEvaluationProject201012Part2.pdf
National Standards: School Sample Monitoring and Evaluation Project 2010-2012
Executive Summary
The National Standards School Sample Monitoring and Evaluation Project describes and evaluates the implementation of National Standards in New Zealand schools. It started in 2010 when the standards were first introduced. This report describes information collected in 2012 and outlines trends observed over the three years of implementation to date.
In 2012 information was collected from a stratified sample of 96 English medium state sector schools with years 1 to 8 students, representative of the population of schools in terms of school type, school decile, and geographic location. Five types of data were collected at two time points. Copies of schools’ 2012 student achievement targets and 2011 analysis of variance reports were collected in the middle of the year. At the end of the year Overall Teacher Judgments (OTJs) in reading, writing, and mathematics were collected for all students, and copies of end-of-year reports to parents, families and whānau were obtained. Online surveys of principals, Boards of Trustees Chairpersons, and teachers were also conducted at the end of the year. The teacher survey contained assessment scenarios that collected information about teachers’ judgments in relation to the National Standards.
Overall Teacher Judgments
The OTJ, as a judgment of each student’s achievement in relation to the National Standards, is central to the implementation of the standards initiative overall. The information OTJs provide is reported to parents, families and whānau and to Boards of Trustees. It is also used by schools to tailor teaching programmes and target students for intervention. For these programmes and interventions to successfully raise student achievement, OTJs need to be dependable.
Evidence suggests that increasing proportions of schools made and moderated OTJs from 2010 to 2012, and that in general, the quality of schools’ processes for making and moderating OTJs improved in this time. The efficiency with which schools made OTJs increased (for example, 39% of teachers made writing OTJs efficiently in 2010 and this increased to 52% in 2012), as did the proportion of teachers that can be considered to be using current assessment evidence to inform OTJs (for example, 37% of teachers used current evidence to make reading OTJs in 2010 and this rose to 72% in 2012). The proportion of schools using formal processes to moderate OTJs also increased over the first three years of the standards implementation (for example, 56% of schools used formal processes to moderate reading OTJs in 2010 and this increased to 62% in 2012).
Three sources of information were examined to investigate the dependability of OTJs: the consistency of students’ OTJs over time, a comparison of the OTJs of Year 7 and 8 students in full primary and intermediate schools, and results from assessment scenarios that collected information about teachers’ judgments in relation to the National Standards. Considered together, this evidence suggests that OTJs lack dependability, which is problematic as students’ OTJs are the basis on which schools tailor teaching support with the ultimate aim of improving achievement. It does need to be noted that these concerns do not mean that all OTJs are inaccurate. While general trends can be identified in the data collected there is no way to ascertain the accuracy of any individual OTJ or to estimate the proportion of accurate OTJs. It is also likely that the inconsistency in teachers’ ratings is a result of the relatively broad nature of the National Standards scale and the current lack of tools available to support National Standards judgments.
Reporting to parents, families, and whānau
Clear reporting to parents, families and whānau is an important part of the National Standards initiative. The intention is that families are well informed about their children’s learning and, therefore, more able to support this in the home.
Findings indicate that schools increasingly reported National Standards information to parents, families, and whānau from 2010 to 2012. The proportion of end-of-year reports that refer directly to the National Standards increased over time (79% in 2010 and 91% in 2012), as did the proportion of reports that sufficiently describe student achievement in relation to the National Standards (60% in 2010 and 73% in 2012). Results suggest that the clarity of reports may be of concern, with less than half (43%) of National Standards reports rated as clear in 2012.
Student achievement targets
OTJs are reported to Boards of Trustees and used to inform annual student achievement targets, which guide decisions about the teaching support individual students receive.
Evidence from the project suggests that increasing proportions of schools included targets in their schools’ charters that addressed student achievement in relation to the National Standards (75% in 2011 and 93% in 2012). The targets were increasingly differentiated to accelerate progress for specific groups of students (57% of National Standards targets were differentiated in 2011 and an average of 64% of targets were differentiated across the three areas in 2012). The level of challenge inherent in schools’ targets may be a cause for concern however, with less than half of schools’ National Standards targets in reading (47%), writing (43%), and mathematics (48%) rated as challenging in 2012.
Schools use of National Standards data
It is intended that schools will use National Standards data to provide both tailored professional development support to teachers and targeted teaching interventions to students, with the ultimate aim of improving student achievement.
Results indicate that increasing proportions of schools collated National Standards achievement data (for example, 76% in mathematics in 2011, increasing to 93% in 2012) and used this to tailor professional development support for teachers (45% in 2010 and 54% in 2012). In turn, teachers indicated that the standards have had an impact on their professional knowledge and practice. In particular, increasing proportions of teachers reported becoming more systematic in their collection of evidence about students’ progress and achievement as a result of the National Standards (63% in 2012) while about half of the teachers surveyed noted they have a better understanding of what students need to be achieving at the level(s) they teach (57% in 2012).
Principals also reported increasingly using National Standards data to inform the provision of tailored teaching interventions for students (an average of 61% across the three areas in 2011 and 89% in 2012). Teaching support was provided in a variety of ways; approximately half of the principals reported that regular classroom teaching programmes were increasingly differentiated to meet students’ learning needs (43% in reading, 48% in writing, 54% in mathematics), with principals noting that support external to the classroom programme was provided both by qualified teachers (72% in reading, 57% in writing, 35% in mathematics) and teacher aides (33% in reading, 20% in writing and mathematics). The quality of these teaching interventions, or the extent to which they were matched with students’ learning needs was unable to be investigated.
National Standards achievement data, 2010 to 2012
There have been small increases in the proportions of students rated ‘at’ or ‘above’ the Reading, Writing and Mathematics Standards over the three years of implementation to date. For example, 72% of students were rated ‘at’ or ‘above’ the Reading Standards in 2010, and this increased to 74% in 2011, and 76% in 2012. Substantial increases in the proportions of students rated ‘at’ or ‘above’ were observed for some demographic sub-groups: Pasifika students and Year 7 students in reading, writing, and mathematics, and students at low decile schools in reading and mathematics. These increases must be interpreted with caution; they represent changes in teachers’ judgments of student achievement over time. Because other evidence raises concerns over the dependability of OTJs, this data cannot necessarily be taken as evidence that student achievement is improving over time.