Practical Theory - Comments

Most products are judged by satisfaction over time. I am sure that the measurements Toyota was using as they pushed products out the door said that their vehicles met standards. Look at how the products are faring today. An in process measurement is simply feedback by which we make adjustments. The true measure of what we do is how our students fare over time. Can we possibly manage to measure our impact when our students are out there doing what we tried to prepare them for? It is the only way we will know how our schools have really performed.

#2 Dave Meister (Link) on 2010-02-13 10:54

The psychometricians that analyze these tests can show you all kinds of measures of internal and external validity, reliability, etc. They also can show you high levels of correlation with other assessment measures. The statistical analysis on these is extensive. There's lots of number-crunching that verifies that 'these tests' measure 'what they're supposed to.'

So... given all that, what would you like to see?

#3 Scott McLeod (Link) on 2010-02-13 11:33

I suppose I'd like to see humility, Scott. I've read the data studies on various tests, and I won't deny that there is often correlation. But I've known too many teachers -- and had too many experiences myself -- where SAT or PSSA or Regents results left an educator scratching his/her head, saying "These tests don't reflect what I saw with my own eyes.

The other issue, of course, is that these tests might be good at measuring what they measure. But they aren't very good at measuring what they don't measure. And our need for the perception of clarity is having a powerful effect on what we value. And the psychometricians don't want to take any responsibility for that.

So that's what I want, Scott. I want humility. I want us to admit we might not know everything yet. I want us to admit that trying to measure what a child learns from us / with us might be really hard. And that our best ideas are just that - ideas.

Because if we do that, a) we might get the debate we need about education we need in this country and b) we might accidentally stumble across a pretty good value to model in our classrooms, too.

#3.1 Chris Lehmann (Link) on 2010-02-13 11:49

Okay, I'll admit it. I occasionally hang with statisticians. =)

I was a fairly active member of the Minnesota Assessment Group when I was a prof at U. Minnesota. I got my own stats training from the Iowa Testing folks at U. Iowa. I'm giving a talk next month for the ACT annual meeting. So I know this group a little bit.

The statisticians and test-makers that I know are very careful to delineate what their tests measure and what they don't. I have seen them make comments many times to try and rein in others' overly-broad interpretations and/or applications of test results. In other words, I have seen the test-making community be pretty 'humble' about what they're doing and what their constructs are good for...

Now, I'm not saying that the psychometrician community is blameless or without fault - goodness knows they are eager to have their work and influence spread. But maybe the real issue is not the test-makers' desire to be important so much as it is the rest of us? We educators, politicians, policymakers, bloggers, etc. may overgeneralize and overextend and overvalue their work where we shouldn't?

#3.1.1 Scott McLeod (Link) on 2010-02-13 11:58

Regardless of what any test measures, the resulting metric is useful only in establishing that the stuff tested is measured.

A teacher working with an individual child can use a test and triangulate whether that test accurately reflects the learning profile for that child.

In other words a child gifted in something other than what's tested may get a terrible score on a test yet be well-developed to pursue their bliss in life - a dancer may never be good or interested in math.

Now the test is real good at determining that the dancer can't compute but what ethical right does the State have to insist that child conform to learning math? Is it not the purpose of school to provide the opportunity to learn math, to learn science, art and so on. And isn't it the God given right of the child to follow their bliss?

And that brings us to teacher evaluation based on these scores. The government will now insist that the dancer must be broken to conform and that the teacher must be broken to resist nurturing the dancer and instead shape an automaton.

And any teacher who loves the child as a child of God will become unwelcome in the schools. Whose idea of education is this? Hitler?

#3.1.1.1 Frank Krasicki on 2010-02-13 14:56

I think there's plenty of blame to go around... And yes, I hold the policy-makers responsible for much of the thinness of the debate these days.

And I have met some thoughtful psychometricians (and the ITBS folks are among them) who are definitely doing good work that matters, but I've also met some folks who have a certainty and faith in their work that I do not believe is warranted. Daniel Koretz's work "Measuring Up" is the best thing I've read about the way we mis-use standardized tests these days, and he's hardly a crunchy-granola, project-based learning guy.

What I want is debate, not shouting. What I want is humility, not certainty. And what I want is a recognition that we may not know all there is to know about education yet.

#3.1.1.2 Chris Lehmann (Link) on 2010-02-13 15:08

Scott,

I think the we should consider the following as we assess teacher effectiveness:

1) increased teacher attendance
2) increased teacher retention
3) increased evidence of teacher collaboration
4) increased evidence of differentiated instruction (with possible attention to Gardner's theory of multiple intelligences)
5) increased student attendance
6) increased variety of home/school/community communications
7) increased parental and community involvement

increased opportunities for extra-curricular activities
9) increased participation in extra-curricular activities
10) increased evidence of student skills with projects that emphasize a variety of reading, writing, speaking and listening (and visualizing in NJ) strategies
11) increased evidence of student skills with projects that emphasize a variety of adding, subtracting, multiplying, dividing and graphic organizing strategies
12) increased evidence of student inquiry, collaboration and reflection

Some of these extend beyond the school environment and into the larger community, and I'm OK with that.

The question for me then becomes "How can the local, state and national infrastructure be designed to support children so that when they arrive at school, they are ready to learn?"

#3.2 Gamal Sherif (Link) on 2010-02-14 07:22

I completely agree! I am currently combing through data for my students after our last benchmark and it is very frustrating. Some of my best students performed lower than proficient. Should this indicate their true potential? Or serve as a qualifier for my ability to teach dynamic lessons that, sometimes, include stepping outside the educational spectrum for a risk or two.

Data is cheap, easy, and convenient, but does not truly measure our students' ability to learn, process, and apply skills. The importance on data needs to change and unfortunately, this is our culture we live in and have to deal with. I want my students to leave my classroom with skills and the ability to think, question, and seek out information in a world covered with information.

I really found it refreshing to spend time at Educon and see how SLA functions. You have a great system in place and you can tell the students are truly engaged.

#4 Andrew Marcinek (Link) on 2010-02-13 11:34

Scott and Chris - Really?

"The statisticians and test-makers that I know are very careful to delineate what their tests measure and what they don't."

And they make a point to go out and scream about how misused their testing is!???? They take every chance they get to make sure EVERYONE knows that?!!! Is this before the company they work for makes a few more hundred million???

Because my school (And thousands of schools nationwide) live that every day. And what we hear are statistics about how students that aren't reading at grade level by 3rd grade make up 85% of the prison population (like we need the motivation) and our test scores aren't showing that we are keeping many students out of prison is implied way too often.

Please talk to your testing friends for the students at my school and others like it that are now spending 75% of their time prepping for those test designed by your well meaning buds.

You might pick up that testing time is "Now 21 days away, what did you do today to make your students successful?"

Learning is messy!!!!

#5 Brian Crosby (Link) on 2010-02-22 22:48

Name
Email
Homepage
In reply to
Comment	Enclosing asterisks marks text as bold (word), underscore are made via _word_. Standard emoticons like :-) and ;-) are converted to images. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above:
	Remember Information? Subscribe to this entry