Sometimes you really wonder.
Results. Four students got 100%, another four got an A, and 5 an A-. There was 1 B+, 7 B, 12 C+, 4 C, 18 D and 23 Fails with 27 No Shows. Overall average 71%. So some exceptional performances but overall not so hot.
What gives? Clearly there is an end of term exhaustion issue, and among the no-shows were a fair number of students who already had high marks for the class tests so did not need to do this one. Their absence dragged the average down, for sure. And alot of the bad marks came from students who attend erratically. This test was brutal on those who skip class. And most of those with a bad score on this test have never been to any revision session. In fact, I think I have seen less than 1/4 of the class in revision sessions. Odd.
Still, this course has many ways to get a good grade, and doesn’t penalize the odd bad test. Even for the tests, 83 of the 100 students have a B or better for overall test score.
Was the test too hard….?
I don’t think so. If it was, it would have generated more teachable moments. But the class as a whole came unstuck on only five questions. Two of those involved last week’s guest instructor on energy futures. Only half the class were even present, and a bunch of those that were there played on their phones and computers. I am pleased I set questions that rewarded those that were present and listening.
Three other questions show I have to go back over randomized control trials (RCT) and why they can be so powerful when done well (they dispense with reverse causation and third variable problems).
No, the odd thing was that those five questions aside, the class overall did well on each question. So it shows that most students are having trouble doing consistently well across the board. Maybe that comes back to: go to every class; revise tests; ask Andrew about past questions you did not understand.
I did take immense heart from the way the students handled the risk questions. Serious progress there.
A question on the scientific method was upsetting: 48% of respondents chose an option consistent with “Data can be made to fit any hypothesis”. If that was true, most of us would be dead by now. Worryingly, this question was a slightly re-worded version of a question given in Class Test 1. Way back then, <10% of the class thought “data could be made to fit any hypothesis”. It could be that I am unteaching the students. But a student told me that in my tests ‘all of the above’ is always the right answer, so for this test I liberally sprinkled ‘all of the above’ about the place…
There was of course, Q#22. I screwed up the grammar which confused things, so I dropped the question from the test mark (a third of the students chose one of each of the three options, never a good sign). Oddly, only four students raised the mistake with me, even though there is extra credit in being first. Brittany Musaffi won that prize at 5.47pm, three quarters of the way through the exam period.
This evening I did two revision sessions on the test. About half the fewer than 20 students who turned up had in fact done really well on the test. They want to know how they missed 100%…. I so wish some of the serious under-performers would turn up so I could figure out what is going on.
I found no new errors in the test, and it was not hard to teach, explain and justify my answers. I am left thinking the test was easier than any of the earlier ones. I can only guess that the many students who did poorly have not been coming to class or revision sessions – or they did not like that I came at the standard SC200 issues from slightly different directions. But that’s how to test (and develop) real understanding. And that’s real life.
I started thinking about question #23 again. The “Before the study began” being the key phrase to denote the accepted probability of the 5% standard.
However, let’s say they had factored in a different confidence interval (say 7%), how could we extrapolate that information? Would we just have to look at the actual published study rather than a news article on it?
It is true that some studies start out with a different Type 1 error rate. But that is unusual – or it is an adjustment aiming at keep the %5 error rate across the whole study when there are many individual tests involved.
The key take home message I am trying to get across is that ALL studies testing hypotheses have an appreciable change of falsely concluding that something is going on when it fact it is not.