As powerful as an inquiry process might be, it is only good if practiced regularly.

Recently, we met with teams of teachers in Florida who are learning the TERC Using Data process of school-based collaborative inquiry. Between our two scheduled face-to-face sessions, these data teams returned to their schools to apply the process they had learned and dig deeper into their own data analysis with colleagues. One returned with an epiphany. “I thought we were learning a quick way to ‘fix’ things. I now realize that there is no quick way to do this. You just have to take the time to engage in the process, understand what to do to get results, and do it!”

Meaningful data analysis, pinpointing student learning problems by triangulating multiple data sources, deconstructing student work samples, finding root causes for emerging problems, and launching a plan to tackle these problems takes time.

Anyone who has ever integrated inquiry into classroom instruction knows how time-consuming it is…and how valuable. The same holds true for a data analysis process based on collaborative inquiry.

Given the frantic pace of the school day and the school year, finding time in schools, even for the most worthwhile activities, can seem almost impossible. However, there is no way to have an inquiry-friendly culture without allocating time for teams of teachers to use data, plan, monitor, and reflect. For most schools, that requires rethinking how time is allocated. Here are some ideas and strategies that successful Using Data schools are using*:

Freed-up time

This strategy entails freeing teachers from regular instructional time to participate in data-focused professional development or data analysis activities. It is achieved by hiring substitute teachers or by recruiting administrators, parents, or other volunteers to serve as subs. Volunteers can also cover teachers’ recess and lunch duties.

Restructured or rescheduled time

This solution requires formally altering overall instructional time—the school day, the school year, or teaching schedules. Examples of this time-creating strategy are a move to a team-teaching approach, a year-round school schedule, or a revised schedule that allows for regularly-allocated early student release days.

Common time

Many schools encourage common teacher preparation and planning time, rather than individual prep time. This enables teachers to meet as grade-level or subject-area teams. Common meeting time, when coupled with a lunch break, for example, can result in approximately 90 minutes of uninterrupted time.

Better-used time

Most schools already have time built-in for teachers to meet, such as staff, department, and grade-level meetings. Encouraging a culture of electronic communication can greatly reduce the amount of time spent at these meetings on administrative-related activities. And more time to engage in inquiry. Additionally, comprehensive reassessment of existing professional development plans, and using more of that time for data analysis and collaborative problem-solving, can often lead to greater gains overall.

Purchased time

Some schools and districts are able to reallocate existing funds and occasionally provide stipends for teachers to engage in improvement planning activities outside the school day.

Wise education leaders, as well as classroom teachers, who recognize the power of inquiry and collaborative data analysis to improve learning, work hard to find creative solutions to the time crunch. They concede that they do spend more time planning up front, but the time is well spent. Time isn’t wasted solving the wrong problems or taking actions that won’t get results. And taking time to set up a good data management system adds to the efficiency of the process. So think of this time spent as an investment—one that pays off when results show student achievement on the rise!

*Excerpted from Using Data/Getting Results, TERC, Cambridge, MA.

I very much enjoyed Part I of Jill Thompson’s blog series about “Using Data to Drive Instruction in the Classroom.” According to her bio, Jill is an elementary math and science facilitator.

I applaud her for sharing her insights and passions about this subject. As a former classroom teacher, and currently as a facilitator for TERC’s Using Data process, I find myself in step with her thinking. Regularly integrating formal and informal assessments into the instructional planning process is a must. It’s not adding more to the plate — it IS the plate…understanding the impact of the teaching process on student learning and using that information to plan the necessary next steps—not only what to teach, but how to engage kids in the learning.

These days there is so much negative emphasis on testing, and I understand the rub when I see test scores being used to punish teachers and categorize kids. But let’s be clear that using data and testing are not the same thing. Data comes in many shapes and forms, well beyond test results and grades (these are just one data point). Teachers have the opportunity to use data as a valuable resource to guide a teaching and learning approach that can ignite learning for all students. As Jill notes–it just takes time and know-how (and an understanding that it’s a non-negotiable).

I plan to follow Jill’s blog series on this topic, and I recommend it to you. Thank you, Jill, for sharing your experiences and helping those who might be uncertain about how to put their data to work as an instructional tool. Your ideas illuminate understanding of a process for using data that can profoundly impact student engagement and achievement.

“Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.” Albert Einstein

Once a school or grade-level data team has analyzed several data sources to pinpoint a student learning problem, they often feel ready to leap into action and solve it. To ensure that the solution pursued produces the hope-for results, it’s essential to engage in a collaborative process of causal analysis to identify the “root” cause of the problem.

There are many tools that support root cause analysis, one of them is referred to as Why-Why-Why—a question-asking technique used to explore cause and effect relationships. Why-Why-Why helps a group look beyond symptoms to underlying causes by taking the identified problem and asking why it exists at least three times—each time probing more deeply.

For example, your team learns that math scores on the state test noticeably improved, except for students in the bottom quartile. On the first round of “why” team members respond that many of these students are special education or Title I. On the second round they speculate that the new math curriculum, which is closely aligned with the state test, is too hard for some students. On the third round they consider that often the special needs students are pulled out of class for individual instruction and may not be getting access to the new curriculum. This could be a root cause!

Are you ready to give Why-Why-Why a try?

Action Steps

1. Pinpoint a student learning problem. Be sure to analyze at least three data sources for confidence that the problem is valid. Clearly state the student learning problem in writing on chart paper (or use the Why-Why-Why Form at http://usingdata.terc.edu under the heading Are You Using Your Data Well?).
2. Engage in collaborative dialogue with your data team. Ask “why” do we have this problem, and record one response beginning with “because…”
3. IMPORTANT: Then discuss if this “because” needs confirmation. What other data needs to be consulted to be sure?
4. Continue this same process three or more times. In business, five is recommended.
5. Discuss the data-confirmed causes. Which one seems to be the “root” cause—the one that will get results if changed? This is your “solutions” starting point. But be careful, the Why-Why-Why process has some limitations.

Limitations

The “Whys” process is not scientific. It can’t be repeated, and different groups might uncover different potential root causes based on the limitations of their current knowledge or experiences. That’s why Step #4 is important. Think of Why-Why-Why as an easy and effective starting point for launching the dialog that will move you from problem to targeted solution as you collaboratively engage with data-driven decision-making.

To help data teams think about causes beyond what might be immediately identified, TERC Using Data has developed a set of Causal Analysis Cards that suggest causes grounded in research. You might consider using these as your next step toward discovering, confirming, and discussing solutions that will improve student achievement.

Website: TERC Using Data
For education and data news, follow us on Twitter.
And we encourage you to subscribe to this blog for weekly email updates.

There are often revelatory moments in the data inquiry process, where your analysis will lead to great insight and discovery in a way that challenges your assumptions and changes the way you think about teaching and learning in your school. There are other times when the data shows exactly what you were expecting, confirming your predictions and giving you valuable evidence in making your case to others. Many times, however, the data doesn’t show anything at all.

This can be somewhat dispiriting to an enthusiastic data team, but it doesn’t need to be. Sometimes the data may show nothing, but that’s still valuable information that puts you ahead of where you were before you looked. We don’t complain when our dentist finds no cavities, when the mechanic finds nothing wrong with our car, or when a medical test comes back negative. Similarly, in data inquiry, even a finding of nothing can really be something, if you know how to interpret what it means.

What does it mean when you find no achievement gap?
Counterintuitively, teams can be excited to find a significant difference in performance between different sub-groups within their schools. They’re not happy that a gap exists; they’re just glad that they’ve found it. But when you do the analysis and find no gap, you should be just as pleased, if not more so. Keep disaggregating the data using different criteria, but if you don’t find any achievement gaps in the end, enjoy the discovery that your school seems to be providing relatively equitable educational opportunities for all of your students.

What does it mean when testing data shows even performance across strands?
It’s a clear path to drill down into a content strand where students are clearly showing deficiency. But when strand data doesn’t provide easy answers, it’s time to start looking for pervasive problems that reach across strands. Could students be having trouble with multi-step problems in Algebra, Geometry, and Measurement alike? Are students having vocabulary issues, regardless of whether the question is asking for a literary response or critical evaluation? A deeper look within the individual strands can usually reveal more fundamental problems.

What does it mean when students leave answers blank?
While a telling distractor answer might give you insight into why students got the wrong answer, a significant number of blank answers can signal that students didn’t even understand the question. It also might indicate a problem with stamina, if students are leaving a lot of questions blank towards the end of the exam. A student guessing blindly has a one in four chance of making a lucky choice, but blank answers are often deafening in their silence.

What does it mean when surveys show ambivalence from respondents?
Surveys often measure participant attitudes using a Likert scale, a series of statements that the survey-taker can agree or disagree with to different degrees. Many researchers prefer to offer an even number of responses, so the respondent is forced to choose one side or the other. But if you neglect to include a middle option—the “No Opinion” choice—you are missing out on valuable data. The idea of a survey is not to force anyone to generate results that will be more interesting to you; the survey should be designed to collect the most accurate data possible, and that includes allowing respondents to express their ambivalence if that’s how they feel.

A data analysis that shows nothing is not the same thing as a lack of data or an inconclusive result. So the next time your hard-earned data analysis shows no results, go ahead and make a big deal out of nothing!

*Teaching Matters is a non-profit organization that partners with educators to ensure that all students can succeed in the digital age. They are an official TERC Using Data partner organization, conducting the Using Data for Meaningful Change institute for New York City schools.

“Make data observations. Then generate possible explanations that inform next-steps to finding the best teaching and learning solutions.”
(from: Love, Nancy et al. The Data Coach’s Guide to Improving Learning for All Students, 2008.)

Data analysis is more effective, and more on-target for getting student achievement results, if a team of stakeholders first observe and list as many details as possible about what the data reveal, followed by making inferences about these observations, and then asking “why is this happening?” “what else do we need to know to be sure?”.

Infer/Question is the fourth stage in a team-based, 4-phase dialogue process* that guides deep discussion toward deriving accurate meaning from performance data. (See more information about Step 1: Predict, Step 2: Go Visual, and Step 3: Make Observations.)

These action steps will help you and your data team share inferences about the story the data reveal—inferences that will inform important next-steps toward identifying a valid student learning problem and its true causes.

Action Steps

• Infer/Question: After capturing a complete set of observations drawn from analysis of aggregate, disaggregate, strand, or item data, begin to generate possible explanations for what you observe. It’s important to understand what is happening, and why, before moving to solutions. Think about these questions:

–What inferences and explanations can we draw about our observations?
–What questions do we need to consider?
–What tentative conclusions might we draw?
–What additional data might we explore to verify our explanations?

Begin your inferences with phrases like, “I wonder if…, Could this situation exists because…, I would like to know if…, We really should explore…, A question I have is…”

Inference statements link back to your data-informed observations and might look like this:

“We really should explore whether district scores improved more than our school scores because some schools are on a year-round schedule.”

“I wonder if mathematical reasoning is not emphasized enough in our curriculum.”

“I’m surprised that our regular education and special education students had the same difficulty with the vocabulary used in this open response science question.“

“Our observations of disaggregate data indicate a high mobility rate. A question I have is…do we have programs for kids who come to our school in the middle of the year to help them catch up?”

• Now, set out to find the answers to your questions or confirm your inferences by identifying additional data and indicators you can collect. If you began by looking at aggregate data, start to drill down and look at disaggregate, strand, and item data. Or consider a look at common grade-level assessments, student work, or even survey data. Are you observing new things? Does the new data inform your inferences? Does it change your thinking?

After your data team successfully completes the making inferences phase of the data-driven dialog process, there is one important last step before adjourning the meeting. Ask yourselves,

What are the implications of what we just learned?

What action do we need to take next?

Who needs to know?

Once you’ve collected additional data to clarify and answer your inferences, subsequent meetings will focus on using it to pinpoint very specific student learning problems and their causes—bringing you one step closer to finding solutions that can effectively impact student achievement.

Making inferences and generating questions that will be verified and answered before finding solutions is a classic example of the “go slow to go fast” strategy. It gets you on track for making sure the problem you are solving is one you actually have!

*The four-phase dialog process is adapted from Wellman, B., & Lipton, L., 2004. Data-Driven Dialogue: A Facilitator’s Guide to Collaborative Inquiry. Sherman, CT: MiraVia LLC. Used with permission.

Summer has arrived, and the last of our current data institutes, like the school year itself, has come to an end. But as the participating data teams leave, carrying not a diploma but an action plan, they realize that their work is only just beginning. It is not a graduation; it is a commencement. And the first step in the journey ahead is to introduce the action plan they developed to the principal, administrators, teachers, parents, and other stakeholders in their school communities. This requires another planning document—a strategic communications plan, inviting others to invest in a shared vision for bringing the action plan to reality.

So…what are the elements of a good communications plan that will get others behind the action plan?

The anatomy of a good communications plan starts inside the head. Before presenting an action plan to colleagues, make sure you have a clear sense of the logic behind it. What data convinced you of the student learning problem the team uncovered? What in your investigations revealed the causes of that problem? What does research say is the most effective way to battle these causes? The more logical and evidence-based your chain of cause and effect is, the more confident you will be in your action plan, and the more persuasive you will be communicating it to your audiences.

However, it is just as important to appeal to their hearts if you want to have a lasting impact. Connect your plan to their hopes and aspirations. What is important to each of your audiences? How would they like to see themselves?  How does your plan align with the school culture? If their participation is critical to the success of the plan, they should be made to feel like they are included as part of the solution, not as obstacles to be overcome. Another way to the hearts of your colleagues is through their stomachs. Providing refreshments at meetings not only keeps morale high, but it also sends an important message: we value you as fellow professionals, and we appreciate your time and participation.

In that spirit, it’s important to remember not to point the finger. The idea is to get buy-in for your plan, and if you start playing the blame game, you risk alienating the very people from whom you will need the most effort. Don’t ignore critical causes of your student learning problem, of course, but exercise diplomacy in how you phrase them. Instead of saying that “some teachers aren’t doing” a critical task, note instead that we have “a lack of consistency in the way we do” that task, or that there is “not enough emphasis on doing” that task school-wide. The purpose is to reverse harmful trends, not to cast blame for them.

Once the meetings are over, it’s time to put legs to your intentions. Don’t let your hard-earned action plan become merely a document gathering dust on a shelf. Collect data that confirms your plan is being implemented faithfully and is having the desired effects. Plan for regularly scheduled time to analyze this new data, and be prepared to adjust your plan if necessary.

The success or failure of an action plan can often depend on how it is initially rolled out. Spending some time over the summer considering a strategic communications plan for the fall can help you start the new year on the right foot!

*Teaching Matters is a non-profit organization that partners with educators to ensure that all students can succeed in the digital age. They are an official TERC Using Data partner organization, conducting the Using Data for Meaningful Change institute for New York City schools.

Since the story broke, the media and bloggers have actively covered the details about a recent test-score-fixing fiasco—and the news continues (links provided below). There have been commentaries about who really suffers from a scam of this magnitude (students robbed of remedial opportunities) and who is to blame—their motivations…and their motivators (“…targets were implemented…in such a way that teachers and administrators believed that they had to choose between cheating to meet targets or failing to meet targets and losing their jobs.” Volume 3: Conclusions: Why cheating occurred and cover-up allegations, p. 4). I’m particularly taken with “the data-made-me do-it” explanation. There’s even talk of legal prosecution. This whole mess feels like a major attraction in a very tawdry sideshow of the school reform circus. And sadly, although the media is focused on one district at the moment, test score and data manipulation is not new news.

Yes, there’s explaining to do, and some one (or many) will need to be held accountable, but wouldn’t it be great if the lion’s share of the energy fueling our collective indignation, disbelief, and need for retribution could be channeled to establishing more positive, long-term improvements to a testing and assessment system that has surely gone awry. I’m not yet so tainted that I can’t believe we (saints and cheaters alike) all really want the same thing: exemplary schools, highly qualified teachers, and well-educated students who are life-long learners ready to succeed in their adult lives.

I weigh in with Diane Ravitch on this one, “I don’t want to get rid of testing. But tests should be used for information and diagnostics to improve teaching and learning, not to hand out bonuses, fire teachers, and close schools. When high stakes are attached to tests, people often act in ways that compromise educational values. High-stakes testing incentivizes narrowing of the curriculum, gaming the system, teaching to bad tests, and cheating. “ (Letter to the editor, NY Times, July 5, 2011)

Consider the up side. The data from a standardized test can offer a starting point for collegial discussion about why scores are low, who is struggling most, and the efficacy of existing pedagogy. It can help us: see if curriculum is well aligned with standards, examine the amount of time spent teaching certain topics or skills, and determine if what we teach is aligned with what we assess. It begs us to triangulate state data with results from district benchmark tests, student work samples, and informal student assessments, including observation. It can help us set learning goals and focus instruction to serve student needs. It can also offer clues to the types of teacher professional development needed and encourage peer-to-peer support. (See Assessment as a Tool for Learning for more.)

The recent Atlanta investigation report points out, “APS is indeed a ‘data driven system,’ and whether or not a school meets targets is the most important data of all. What has become clear through our investigation is that ultimately, the data, and meeting “targets” by whatever means necessary, became more important than true academic progress.” (Volume 3: Conclusions: Why cheating occurred and cover-up allegations, p. 9)

If all education stakeholders can use this expose as an opportunity to refocus and take a higher road that leads toward developing and supporting a system of data use that informs improvement, then we have gained some ground.

To help us move on, I offer a call to action. Leaders who pressured teachers to organize answer-changing “bubble parties,” teachers who did the answer-changing, and all of us who have even for a moment valued test scores over seeking pathways to better student learning, read this blog post about The Leadership Walk. We need to recognize that, in a way, we all are potential education leaders, so reflect on who might be watching your walk and what it has to say about you. Then take that higher road. Let your stakeholders—teachers, students, parents, community members, pundits, the media, readers—see you walk the walk of using test scores more intelligently. Use them to guide and focus improvement to the system, the curriculum, the leadership, the teachers, and the students. That’s what we all really want. Right?

And to return to Diane Ravitch for a moment, check out how some other countries organize testing and apply test data. Some of the ideas might help us leave the path along the school reform sideshow and hit that higher road with gusto.

***************************************************************
Testing Scandal in the Recent Media

If you haven’t kept up with the commentary, here is a sampling of chronologically organized links to explore, beginning June 30, 2011 with the release of the investigation report:

Special Investigation into CRCT Cheating at APS

Volume 1: Overview, interviews, school summaries
Volume 2: School summaries, cont.
Volume 3: Why cheating occurred and cover-up allegations

Mayor: A Complete Failure of Leadership in APS in Testing Scandal
Atlanta Journal Constitution (includes video), July 5, 2011

State Investigation Reveals Widespread Cheating in Atlanta Schools
EdWeek, July 5, 2011

Investigation Into APS Cheating Finds Unethical Behavior Across Every Level
Atlanta Journal Constitution, July 6, 2011

Atlanta and New Orleans Schools Show the Many Ways Administrators Cut Corners
The Washington Independent, July 6, 2011

High-stakes Tests and Cheating: An inevitable combination?
Hechinger Report, July 6, 2011

Cheatin’ on Peach Tree Street
Eduflack Blog. July 7, 2011

Atlanta Could Have Averted Its Cheating Scandal If It Had Listened To Its Local Teachers Union
ThinkProgress.org, July 7, 2011

The Most Sickening Part of Atlanta’s Cheating Scandal
Washington Post, Valerie Strauss, July 11, 2011

Cheating: Is NAEP Next?
Scholastic Blog, July 12, 2011

What do we do with the cheaters?
Learning Matters Blog, Taking Note, John Merrow, July 12, 2011

Atlanta Schools Could Owe Up to $260K
CBS Atlanta, July 13, 2011

Cheating Teachers Play the Blame Game: Atlanta’s dishonest teachers say the data made them do it.
reason.com Blog, July 13, 2011

Teachers’ Attorney: Atlanta test cheating report inaccurate, falsely accuses the innocent
11Alive TV (includes video), July 14, 2011

Why Organizational Misconduct Happens: A look at the Atlanta cheating scandal
Sociological Eye on Education Blog, Hechinger Report, July 14, 2011

And…Larry Ferlazzo’s picks pblished on July 17, 2011 for:
The Best Posts & Articles About The Atlanta Testing Scandal

Depending on where you sit, and which frame of reference shapes your work, you either celebrate charter school efforts or think charters reflect a “right” wing or “left” wing  (take your pick) conspiracy to undermine the role of public education in the United States.

A blog post is too short a space to weigh into the considerable arguments both pro and con that can be made regarding the place for charter schools in America. To my thinking, charters are a natural consequence of Americans seeking a solution to a social problem. We may not agree on the substance of the problem or the direction of the solution, but in a society that values and applauds entrepreneurial efforts, charters are here to stay. That is, they have a place until we learn more about the experience (hopefully by examining the data) or, have a collective epiphany about the impact of poverty on kids’ success in learning and activate the collective will to change the way we fund and support local education.

Charter School Vision Equally Blurred

Based on my experience working in schools across the country, the reality is that teachers in charter schools bring the same passion and desire to help children learn as teachers in any other public or private setting. They face the same staggering challenges and then some. And they bring the same blind spots to the table when examining their student learning data.

It’s interesting that as teachers, we tend to share similar assumptions about the reasons behind students’ poor performance in the classroom. Karen Seashore-Louis, Rodney Wallace Professor of Educational Policy and Administration at the University of Minnesota, calls this the “tacit knowledge” we carry in our hearts about who can learn what, and when they can’t learn, why (see Our Reality is Our Reality). This tacit knowledge greatly influences how we interpret the student achievement data we examine, until we become conscious of our “blind spots.” Sometimes the blind spots relate to beliefs about student abilities, the demographics of our student population, or unmentionables about teacher quality and content knowledge. Once these blind spots are revealed, we see the data, and potential solutions, through a new and more effective lens.

As experienced drivers, we become aware of our car’s potential blind spots, and we take additional steps to make sure the lane is open before moving over. Likewise, experienced data teams learn to take extra caution (examining multiple evaluation measures, frequently making equity and cultural checks) before taking action.

A recent experience when I was facilitating data teams from charter schools in New York State revealed how teachers were beginning to become aware of their blind spots. Having completed a drill down through state and local data, we had just completed a round of examining student work to shed additional light on student thinking. That’s when data team members made a comment that indicated a turning point: “Analyzing data is more than just looking at the numbers. There is so much more we can learn.” This sort of epiphany comes as a result of learning about and engaging in a structured, facilitated process of data drill-down and honest collaborative inquiry.

A Tool to Improve Data Vision

Administrators, take note. It makes no difference if you lead a public school, a charter school, or a private school. Furnishing your grade-level teams with elaborate binders of student data and giving occasional scheduled time to analyze the data, doesn’t necessarily translate into deeper understanding of the nature of student learning problems and a collective responsibility to address them. Too often we don’t go beyond looking at that data through our blind spots and selecting solutions that can be off the mark.

I offer a gauge that can help determine the extent to which your schools’ analysis of data is accurate and will translate into decisions that can successfully address the learning challenges indicated in that data, leading to increased student achievement. The Low- and High-capacity Data Use Continuum (scroll down to the fourth bullet) developed by TERC’s Using Data staff can be used as a self-assessment, as a talking tool during professional conversations with teachers, or as a collaborative assessment of how the data team perceives the quality of data use in the school. It can serve as an indicator about whether your teachers might benefit from professional development that could support them in becoming high-capacity data users able to take a leadership role in making data-informed decisions that improve student outcomes in your school. We find it helpful in our work, and we hope you will, too.

Data-savvy investigators never make important decisions based on a single source. When teams following the Using Data process believe they may have found a student learning problem, based on their analysis of standardized testing results, they know to confirm the problem through an examination of student work and other common formative assessments. When they do this, it’s important for them to have a norming process in place to ensure that the data being generated is reliable and useful.

Norming is the process of calibrating the use of a single set of scoring criteria among multiple scorers. If norming is successful, a particular piece of work should receive the same score regardless of who is scoring it. With the advent of the Common Core State Standards Initiative, we may anticipate that curriculum-embedded performance tasks will begin to gain prominence over traditional multiple-choice tests, and it will be even more important for teachers to be aware of how to make the best use of these assessments. Whether or not they are rigorous about norming can make a very big difference.

Many years ago, I was an open-ended response scorer for the New Jersey State High School Proficiency Exam, a test students had to pass in order to graduate. My fellow scorers and I were trained on, and given a qualifying exam for, each question we scored. The exam consisted of twenty sample responses to that question.  If we gave nineteen of them the correct score, we were cleared to work on that question. Once on the job, responses would show up on a computer screen (with no names, so it would be blind to gender and ethnicity), and we would type the numerical score on our keypads. Each response was graded by two scorers independently. If the two disagreed, it would get bumped up to a supervisor. We were evaluated by volume, and by how few times we were overturned. It was an incredibly efficient and reliable system.

Compare this process to the way the writing sections are currently scored on the New York State English Language Arts (ELA) Exam. Different sections of the state have different norming procedures, which means the state as a whole has none. I’ve talked with many New York City teachers who have scored the exam, and they report that there was very little effort to norm. Different scorers had wildly different standards for interpreting the rubric, and even the same scorer could become more lenient as the days went on. The final scores, then, were as much of a function of geography, timing, and luck as they were of student performance. How can we possibly make use of this data to reliably identify student learning problems, let alone make high-stakes decisions about school, teacher, or student performance?

Teacher teams have the opportunity to be smarter than this in the way they score their local assessments. Before any rubric-based scoring begins, the teachers involved should meet. They should each score the same piece of student work using a common rubric. They may then compare their scores, and use the comparison to guide a conversation about how the rubric will be used. Three such rounds can be fit comfortably within a common planning period. The goal is for the teachers to align their scoring practices with one another, so that scoring will be consistent and fair.

Norming can often be dismissed as extra work for an already-busy department. But without it, performance-based assessments will not yield reliable data. It’s good form to norm!

*Teaching Matters is a non-profit organization that partners with educators to ensure that all students can succeed in the digital age. They are an official TERC Using Data partner organization, conducting the Using Data for Meaningful Change institute for New York City schools.

A beautiful morning walk in Falmouth, Massachusetts

A recent weekend wedding celebration on Cape Cod brought together a stirring mix of sapphire shimmering ocean, stunning bride (who I had the joy of watching transform over the years from little girl to accomplished young woman), and satisfying conversation with long-time friends—an enjoyable mix of hilarity, nostalgia, and sometimes serious discourse.

During several early-morning walks with friends connected through our work as educators, the more serious discourse returned time and again to an impassioned discussion about teacher evaluation. This conversation was prompted in particular by a June 4 Washington Post article, Maryland Teacher Evaluation Redesign Bogs Down. We were somewhat stunned to learn two pieces of data: 1) Almost a year ago, Maryland won a $250 million federal grant to build a “transparent and fair” teacher and principal evaluation model that would tie their success to student test scores and learning, and 2) The state is seeking a year’s extension to fully execute the evaluation system it has yet to develop.

“Two-hundred fifty million,” my friend mused. “They could hire 250 people and pay them a million dollars each. With that kind of brain trust you’d think something could be developed in a year.” We laughed, but think about it.

WE did think about it. And talked about it. And then talked some more. Over the duration of several walks and shared meals, the “business” conversation on our end included talk like this:

• We speculated that all that money must have been awarded in the hopes of creating a model system 1) capable of broad adoption nationwide, and 2) ensuring an accurately assembled collection of algorithms that could reliably provide an automated handicapping routine able to take into account: the influence of previous teachers on any given student; the fact that my class this year might have a larger population of English language learners or special needs kids than yours (and what are those exact needs?); the fluid, just-in-time collaboration with my grade-level colleagues as we team-teach student skills development; and an endless list of other variables—yet still be able to pinpoint the relationship of MY individual influence on the specific standardized test scores of an individual student.

Can you fairly translate student data to teacher evaluation?

• We were not convinced that any amount of money can build an automated system that alone can provide “transparent and fair” educator evaluation. This thinking seems ironically echoed in a June 5 New York Times article also focused on teacher evaluation models in Maryland, Helping Teachers Help Themselves, in which it was reported that the Montgomery County Superintendent would not take federal Race to the Top money because the grant required districts to include students’ state test results as a measure of teacher quality.

• We admitted to one another that we are not against considering student test scores in the same sentence with teacher evaluation measures, but it’s only one data point, not THE data point. And not even the most important or reliable data point.

There was lots more talk about the need for trained leaders to facilitate effective teacher evaluation, the role of the unions, evaluation horror stories and successes we had experienced during our teaching careers, the use of teacher-focused Individual Education Plans (IEPs) linked to annual performance reviews, and ways to blow up, not just tinker with, a very old system.

I’m an English language arts major, which perhaps accounts for how overwhelmed I feel about a mathematically based system that can accurately tie standardized test data to teacher evaluation. When I hear the word “system,” I hope it refers to the concept in a more inclusive way—one that integrates human elements that can be described, and systematized, but really can’t be automated. My friends and I are interested in your contribution to this conversation. What does a “transparent and fair” system look like? What are the essential components? How can they work reliably? Maybe we can share some thoughts with the Maryland team.

More Information If You’re Interested

There have been a plethora of articles, reports, and opinionated discourse similar to what my friends and I had over the wedding weekend about teacher evaluation tied to standardized tests, and from the sound of it all, there will need to be a whole lot more before anything sensible (and let’s add “fair and transparent”) is created, piloted, evaluated, and proven (and we are hopeful that piloted, evaluated, and proven are part of the equation before the firing commences!). Here are a few references if you want to stimulate some talking points:

Can Teachers Be Evaluated By Their Students’ Test Scores? Should they be? The Use of Value-Added Measures of  Teacher Effectiveness in Policy and Practice
Annenberg Institute for School Reform, Brown University
NYU professor Sean Corcoran uses data analysis to argue that value-added models are not precise enough to be useful for high-stakes decision-making or professional development. Corcoran cautions to be fully aware of the limitations and shortcomings of these models and consider whether their minimal benefits outweigh the cost.

Experiments in Evaluating Teachers
This year, 17 California schools voted overwhelmingly to begin using an evaluation system that combines individual and school test scores and multiple classroom evaluations by teachers and administrators, along with regular discussions about teacher effectiveness.

Helping Teachers Help Themselves
Montgomery County, Maryland’s 11-year-old Peer Assistance and Review Program (PAR) for evaluating teachers relies a collaborative peer review process.

Making Data Work
UK educator Mike Bostock proposes that sorting out our approach to the use of performance data holds the key to several troublesome areas of current education practice, including having greater influence over school inspection, knowing where to target school improvement activities, improving professionalism, and valuing the good work that teachers do.

The System For Teacher and Student Advancement (TAP)
According to their website, TAP helps teachers become the best they can be by giving them opportunities to learn better teaching strategies and holding them accountable for their performance, see More Than Measurement: Lessons from TAP Teacher Evaluation.

Best Resources For Learning About The “Value-Added” Approach Towards Teacher Evaluation
Blogger educator Larry Ferlazzo offers many “best of” lists.  Here’s his compilation on this topic.

And one last note…
During the wedding reception, the DJ played Etta James’ dreamy hit At Last, which reminded me that one of our previous blog entries about teacher evaluation was inspired by that very piece of music, At Last: The Saner Side of Teacher Evaluation Data.