Monthly Archives: November 2010

Stress testing the test

I gave today’s Pop Quiz to my 8th grade son, who calls himself the biggest G in ghetto (ghetto?). Without having participated in the course, he got a D.  Is he a nerd, brilliant, the product of a sad upbringing — or are my tests too easy?  My son’s theory is that the class struggles with my tests because they can’t understand my accint. [What he calls the true meaning of yiss]

Tests that test.

Ironically, the most friction I have had with the students has been over the part of the course I am most pleased with.  They hate my tests. I think my tests do good.


After some reflection (1, 2, 3), I believe this tension is because we have diametrically opposed views on what tests are actually for.  The students think that, like their drivers licence tests, any semiconscious person should be able to pass, and that a flicker of neural activity should generate an A (class comments).  I think tests should test.  The right tests (mine!) are a way of forcing students to think.  I had complaints about this.  Thinking is not popular. Especially not thinking hard. But that’s what university should be about.  I also think tests should tell me what I am not getting across to the class as a whole, and tell individual students what elements of the course they are not understanding.  Well designed tests (mine!) should also generate teachable moments.  Ideally, lots of them.  Stretch the students, watch them snap, and then teach those moments.
But of course the students want A‘s. Not stretching.  And so then they threaten to give the course bad ratings.  Bad outcome all around.

So I wrestled with this, and probably beat myself up about it more than was useful. How to make the tests test, without a student revolt? Here’s what I hit on. I was setting 25 multiple choice questions. This means only a few wrong and even the best students head B-wards.  So why not ask 28 questions, and mark out of 25?  This allows three wrong answers before marks start to drop.  The students interpret this as a gift, and I interpret it as space to stretch the students.  The merit of this scaling system is that it rewards the better students, without generating freebies to those who get a lot of things wrong.
Student reaction after I scaled Class Test 3 this way:-
Then I got an A!! Thank you Andrew!
this is so much more helpful and easier on us. thank you Andrew!
I love u Andrew
total bro move thanks dr. Read


More generally, I think this is the 21st Century challenge to university teaching. They come here wanting a great transcript.  We (and broader society) want them to become better educated.  How to reconcile those two missions?


We roll into Thanksgiving week, and then there are just two weeks left. Hard not to ruminate on course failings. Did we do poorly on transmitting the Passion, Beauty and Joy? Damn. Might be the most important.

"I don't understand"

I discovered a curious thing in class Tuesday.  There are students who don’t understand something, yet don’t ask in class, or anonymously text the comment wall, or drop by after class, or send a follow-up email, or ask at a revision session.   

I berated the class about this. They tell me the problem is even worse in other courses. I told them early in my course that NO QUESTION IS TOO STUPID, and that there are many reasons why they might not understand something, the least likely being their stupidity. Much more likely, Faculty explained things badly.
To my berating, two responses on the comment wall:
I don’t ask because I feel like any answer I get still won’t make any sense. I’ll end up going “*nod* Okay…” and thinking “Still have no clue.”   Groan.  I know no Faculty who would be happy with that.  Ask, ask, ASK.

You can’t just say I don’t get it you have to ask specific questions as you have told us. Without specific questions, no one can help.  Try this: identify the point where the wheels fell off.  We can go from there. If you really understand nothing at all – really, zero, zip, nada, zilch? – then that’s important to identify and we can start over.  But without communication, nothing can happen.   True of so much of life, from teaching to marriage.
I spend my working life saying “I don’t understand”.  All the cool stuff is in that zone. Why aren’t students happy to admit when they are there too?  Is it laziness on their part?  Or something wrong with the culture we create?  Surely we want to encourage our students to say when they don’t understand. That question drives science. The world would be a better place if it was the fundamental question in all walks of life. I don’t understand.  

How lousy is non-science?

Perhaps the most important implication of the gross inefficiencies of medical science actually lies beyond science.  If just 2% of medical ideas generate positive benefits, and as many generate harm as help, think what the hit rate must be in fields of human inquiry where experimentation and hard inference are impossible.

Think economics.  Whole chunks of politics.  And law.  Policy makers must be really groping in the dark.

More humility.

Another of Dean Larson’s pearls came after he had told the students that when they were born, we thought the stuff in the universe was star dust.  Today, we think that less than 5% of the the universe is star dust; what constitutes the remaining 95+% is mysterious (the physicists, who can get away with such things, call it ‘Dark’).


The lesson the Dean drew:  “Humility in the face of persistent great unknowns is the true philosophy that modern physics has to offer”  — Joseph Silk, Cosmologist.
So while biologists were decoding genomes and building new life forms, the astronomers and cosmologists were discovering ignorance on a vast scale. Physicists have been ahead of the curve so far, so there is probably a lesson in this for those of us with more earthly concerns.

The Dean on the Cosmos

Dean Larson gave the class a crash course in astronomical and cosmological thought last week.  Two hours on the whole lot.  Two startling moments:


1.  His conclusion to the question, How big is the universe?:  “Darned big”.

2.  In front of a huge Hubble telescope shot of a distant galaxy, and having explained it likely contained 100 billion stars, the Dean wanders up an aisle, stops mid sentence, and says “I think that is just awesome”.
Memo to self.  Inject more wonder into the course.  And make Darned Big an option in a multiple choice test.


After I talked about the inefficiencies of science at my lab meeting, one of my PhD students drew my attention to an extremely thought provoking article in The Atlantic.  I think for SiOW next year I will do more in this area.  I didn’t even get into the lessons of bad or absence science in medicine for the life decisions of individual SiOW graduates.  There are so many hooks here.  Not least, the way I can use this to teach the human frailty of science, the problems of public perception, and why faith healers flourish.
I also love the quotes at the end of the article.  Especially:  
“The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing”.
“Science is a noble endeavor, but its also a low-yield endeavor.”  Amen.

How lousy is science?

Who’d have thought that medical science can be used to examine how good scientists are?
Large double-blind randomized control trial (RCTs) are about as good as it gets for figuring out whether some new medical treatment actually works.  In RCTs, patients are randomly allocated to either placebo treatments or the experimental treatment, and if it is all done well (blinded, large scale, properly randomized), an answer emerges which is a great deal more reliable than can be had any other way.  But at the heart of RCTs is a paradox.  Only treatments which scientists think will work are tested in these expensive trials.  So what patient in their right mind wants to be in the control arm?  Everyone should want to get the new treatment – the scientific bet is on the new treatment. Indeed, if medical science is any good, the very trials needed to test large-scale efficacy and safety are unethical!

cancer-drug-companies.jpgFortunately, it turns out that medical science is actually not that good.  Analysis of 50 years of U.S. cancer RCTs have shown that that most of the time, the new experimental treatments do nothing at all, but that when they do, they are as likely to harm as help. Consequently, the trials are – magically – ethical (whew).  On average, you are as well off getting the placebo as you are getting the fancy experimental treatment.
That’s good news for medical science in one sense, but appalling news in another.  It means that despite all our knowledge of biochemistry, cell and molecular biology, all the test-tube experiments, all the animal studies, all the small-scale patient trials, all the theory and experience in the world, when it comes to identifying treatments that will work, we’re no better than a coin toss.  Importantly, only around 20% of the trails are breakthroughs. There is 80% waste (or harm) in the process, and that is excluding all the ideas that never survived to get to RCT stage — say ten times as many again? 
Weeks after I taught this to the class, I still find it staggering.  I think my science has a better hit rate than  2% — but then, there is no way to quantify success and failure in my business.  In perhaps the one area of science where success rates can be quantitified, it turns out we are staggeringly poor at discovery.
Of course, all that waste has nonetheless powered enormous improvements in cancer outcomes.   And it has certainly been way more successful at cancer treatment than centuries of exploration by other approaches to knowing (think religion, witch doctors, homeopathy, philosophy, and non-evidence based medicine).  But still. It just shows how much we really are groping in the dark.

One of the best things about teaching…

Oddly, I found myself presenting one of my SiOW classes at lab meeting this week. Odder still, none among the assembled PhD students, Post-docs, Senior Research Fellows and Faculty complained about being taught a class I designed for undergraduate non-scientists.  


It was on what is NOT science.  It is easier to recognize and understand science if you explore other ways of knowing.  For thousands of years, people calling themselves physicians and doctors have embodied the very essence of NOTscience, working instead on the basis of personal experience, half-baked theory, anecdotes, and wisdom handed down from eminent people.  Most physicians still do. Great stuff to teach to because it is easy to show that millions of people have been killed by their doctors’ scientific failings, even this century, in the US.  Powerful stuff, easy to get across.  
What I was not expecting is that I myself would learn something about science by talking about medical non-science.  And that I would in turn talk to so many people about what I found out, including the lab group – and now the blogosphere.
Teaching can be important and very challenging.  For researchers, it can also be a great way to shine a light on the world beyond our daily focus.  Certainly, since Dean Larson taught the class astronomy last week, the view from my hot tub got a whole lot more interesting.