Monthly Archives: December 2013

Fall 2013: Let it go Andrew

OK, I’m done. It’s the end of the year, and time to get back to the future.

walk away.jpg

But before signing out, the lessons from 2013?
  • No major changes in course structure or assessment rubrics are needed, except perhaps concerning attendance (see below). And maybe drop the extra credit for filling in class surveys.
  • Re-double efforts to manage student expectations, tell them why things are being done the way they are. Keep telling them throughout, not just at the beginning when they don’t take it in.
  • Re-double efforts to get students looking for an easy A off the course. Many, many deserving students can’t get a place.
  • Make the course open to freshmen only. Freshmen have more energy, are more open to new challenges, and are less cynical. Plus, the thinking skills involved in this course sets anyone up for a better College experience.
  • Do a better job of explaining how scientific consensus is built; of how individual studies can reach erroneous conclusions, but many studies, done right, can lead to consensus and progress. This would deal with the confusions I had this year over meta-analyses, errors and the file drawer problem.
  • Make available tests and exams from previous years so students can practice.
  • Tell them from the get-go they will be initially slammed in tests and slammed in blog grading. Then make it clear what resources are available to help them improve. Emphasize as much as possible that this course rewards improvement, but you have to be engaged to improve. That many students do very well. But it is not a given. Keep saying that throughout.
  • Attendance. Forget the extra credit for having been at all pop-quizzes; that penalizes the unlucky. Instead, how about doing ten pop quizzes at random times throughout the year, and giving 10% of grade to those who are present for 8 of 10 of them? That would be brutal. It would mean people could only get an A by coming to class. Would it work? Is it fair? [that rubric would have cut 2 A’s and 3 A-‘s this year].
  • Wake the students up with the first exam. From the get go, ask penetrating questions that are only answerable for if they have been in class. Manage expectations so that they don’t get crushed. For many of them, this is the first time in their lives they have got a C.
  • Figure a way to get better attendance at exam revision tutorials, particularly from the students NOT doing well.
  • Offer even more guidance on what makes a good blog. In particular, show/emphasize that it needs to be science-related, and that it needs to extend things beyond a simple topic. Relentlessly give examples of good practice early on. In the nicest possible way, stomp on nonsense and mediocrity on the blog fast. Students get influenced by the majority of the blog (i.e average performance), not best practice (which is rare to start with).

And finally, tackle some new topics. Big topics. Like the meaning of life. And evil.

Customer Satisfaction 3. Three things learned.

Much to my surprise, it turns out that by far the most useful question to ask on the end-of-semester Student Rating of Teaching Effectiveness surveys is not one of the questions the university or department requires you ask. It is instead What are the three most important things you learned? (I thank Dean Williams for suggesting last year that I add this question). A smattering of responses from the class of 2013:
  • hand_3Fingers.jpg(1) problem solving skills, (2) appreciation for science, (3) random conversation starters!
  • (1) time management, (2) science is always very uncertain, (3) critical thinking skills.
  • (1) We can’t rely on the news (2) how to read a study and weigh significance myself, (3) just because you teach an amazing class with the utmost clarity, does not mean your students will appreciate it/take advantage of the information you are supplying to the them.
  • I learned how to look at statistics and studies more critically. The most important thing I learned was how not to be ignorant and take all information at face value. I have already used the skills I learned in this class in other classes.
  • (1) Question everything, (2) learn from your mistakes, (3) science can be for everyone
  • Aliens aren’t real. Animals have gay sex. Pruney fingers resemble river flows.
  • (1) You don’t have to be a scientist to think like one. (2) Critical thinking is one of the most important things in our lives. (3) There is ALWAYS room for chance.
  • (1) I should get vaccinated for HPV, (2) we have crappy intuition, (3) science’s answer is only good until someone else proves them wrong.
  • (1) How to be a more critical thinker, (2) how to challenge everything in life and not be so content accepting facts, and (3) not to slack off at the beginning of semester.
  • I learned not to hate science
  • How to write a blog, time management, and how to do research.
  • (1) how to think through problems, (2) how to analyze situations, (3) how to reach my own conclusions.
  • (1) we have more than five senses, (2) albatrosses can be gay, (3) how to build a blog portfolio for my major.
  • (1) Critical thinking can come from any class even if you don’t expect it. (2) I can’t rely on my photographic memory to teach me everything. (3) How to answer questions more intuitively.
  • How to interact with peers. You can find science in almost everything. Don’t procrastinate.
  • Science is important in medicine, people aren’t that good at making new ideas that work, and I liked the Nobel Prize guy.
  • (1) Parasites control rat brains, (2) animals can be homosexual, (3) the size of the universe
  • I learned how to analyze study reports. I learned the difference between correlation and causation (VERY IMPORTANT! :)). I also learned that science isn’t just chemistry and biology, it is everything around us.
  • Science is about making mistakes until you find the answer which relates to life in a lot of ways
  • 1. Nothing. 2. Don’t believe this is a non science course. 3. Science still sucks.
  • NOTHING
  • 1. How to blog. 2. How to speak in an accent. 3. How to take on line tests.
  • Something I actually once loved can be made very boring
  • Science is fun! Science is cool! Science is unlimited!
  • I have not learned one interesting thing in this class.
  • 1. To think critically. 2. Not to procrastinate. 3. New Zealand accents make class more interesting.
When you are teaching large classes, it is easy to focus on the problem students and the negativity. But (most of) these comments make me want to keep teaching lots of non-scientists who don’t want to be there. Teaching science to scientists, or teaching small groups of elite students…that’s important of course. But some how it feels like I can make more of a difference to more students with this course. And it still challenges me more than any other teaching assignment I can imagine.

Customer Satisfaction 2. The verbals

The end of semester student surveys generate numbers, which are of limited value in my opinion. The open-ended questions generate the gems.

green-tick.jpg

My favorite positives:
  • I found this course very interesting, and because of it, I am now a much more critical thinker!… All the resources I needed to succeed were given to me.
  • Andrew was great
  • Andrew’s explanations and the way he manages the class are phenomenal
  • Andrew’s incredible enthusiasm
  • Andrew’s personal interest in the material and his character while teaching.
  • Your open mind
  • Entertaining and off beat lessons helped bring the true ‘lessons behind the lessons’ to life
  • the relevant topics kept me interested such as cell phones causing breast cancer
  • Professor Reed (sic) is incredibly engaging. The topics made me want to go to class but also made me want to BRAG about getting to go.
  • The way that Andrew explained how to think through a problem has helped me in many other courses and even in everyday life.
  • We mostly talked about the same basic scientific concepts and idea, but using different discussions about varying facts and topics made it easy to see how the same idea fitted into different things.
  • What helped me [learn] most I think was the topics being interesting they all made you think.
  • Even as a kid who sits in the back and isn’t always focused I would always be thinking about the topics in the back of my mind.
  • The professor. Andrew was really cool.
  • The lectures are super interesting which helped me understand the concepts better.
  • The professor had a good system for asking questions – you could raise your hand or text into the comment wall. I liked that and the way he was willing to stop class and answer any questions thoroughly or until the person asking the question understood.
  • The most helpful thing was the actual lectures. The notes were a base but what Andrew actually described was what needed to be grasped.
  • I am not a science major but this class was very interesting to me. It forced me to look things up I was interested in but never had time to sit down and research them.
  • Professor Read did an impeccable job of preparing the course in such a way that I as a student genuinely looked forward to class…there was never a class from which I left where I did not fully understand what was discussed.
  • This course forced me how to think critically, which will overall help in everyday life.
  • Professor Read was one of my favorite professors that I’ve had at Penn State. He constantly engaged the class with interesting, informative lectures. Yet the class was also entertaining as Professor Read kept the mood light and friendly.
  red-cross-mark-on-isolated-background.jpg

Among the most notable negatives

  • I don’t think I really learned anything in this class
  • This course basically taught me that no electives are easy. I took this as an easy credit and it ended up being the most annoying class I’ve ever taken in terms of the tests and the homework.
  • I learned nothing. The class isn’t important because it does not prepare any student for the tests
  • Honestly, this course made me cry and at times i felt ‘not so smart’ and tried and tried but never met the goal I set for myself
  • Nothing [helped me learn]. While the materials had the potential to be interesting, it wasn’t of interest to me.
  • There needs to be more clarity on the topics and better notes. The test are way too hard.
  • This was such a hard course
  • Blog grading unfair… exam questions biased to teacher.
  • What is the point of notes if they are not going to help us take the tests? I understand they are supposed to help us think critically, but its like we have to take the test choosing the right answers the professor wants us to pick. It is extremely difficult.
  • The exams are extremely tough. I found myself studying for days and still receiving poor grades.
  • Exams are unrealistically hard.
  • I hated the structure of the course. Doing class blogs was tedious and did not teach me anything.
  • [To help me learn, I need] more important topics, the class is filled with fluff. I did not like that.
  • I went to class almost every week and still found the exams to be the hardest I’ve taken in my three years here.
  • The tests in this course were designed for people to fail…. I would NEVER recommend this course to anyone.
  • I understand he is trying to get us to think but I think they can be bias toward people of various ways of thinking
  • The grading system is disorganized and unfair.
  • The tests are ambiguous and just silly
  • My biggest concern is the grading…for future classes, it needs to me more achievable to receive a good grade.
  • The exams are impossible
  • The exams are absolutely absurd
  • The exams taught me nothing. I don’t know how people do well on them. They are way too difficult for the course. I kind of gave on this course towards the end because it was just ridiculous.
  • More lenient grading [would help me learn].
  • NOTHING IMPORTANT HORRIBLE CLASS AND HORRIBLE GRADING
  • …Finally, not tapeworms. Gross!

Capturing life's complexities.. in multiple choice

The class test and the final exam are multiple choice questions (here’s an example). What else can they be, given the class size?

quarter of the class thought I graded too hard. A subset of those also thought their learning would be improved if I made the multiple choice options less ambiguous.

quiz-multiple-choice-choosing-best-answer-test-words-several-answers-person-selection-as-his-32875758.jpg

I really need to get better at managing student expectations here. I tell them from the get-go that my tests are not like their driver’s license ‘test’. I am not assessing them. I want them to practice thinking; I want them to examine things, to apply the critical thinking skills I am trying to get across.
One of the challenges for me is to shoehorn the real world – which is all shades of gray – into a multiple choice format. I actually like the discipline. Some students object to questions with many options (up to 8). Well, oftentimes life has many options. And sometimes it does not. Sometimes life does boil down to the dreaded (a) yes, (b) no, (c) not enough information to decide.
And real life is about making mistakes, and getting back up to try again. I think my tests reflect that; some of the students hate them for precisely that reason. They get two goes at the tests and they want to know which question they got wrong, so they can pick from the other answers. They have learned there should be an obvious and easy right answer. I wish reality were so. Out there, it is about more than chasing an A. I have to explain that better.
My favorite answer set of the semester: (a) probably, (b) probably not, (c) impossible to tell. Several of the students objected to that. Welcome to real-world decision making.

How hard (again)

Students are asked in the end of course surveys What changes would improve your learning? Of the 127 SC200 students who answered this question this year, the most common suggestion (made by over a quarter of them) was that I grade easier.

I can not see how making it easier to get an A would improve learning. Nor can I see that students or society are well served by dumbing things down. In fact, the more I look at the world our students are entering, the more I think we have stretch them further than we currently do.
The only way to get students to stretch is to grade hard. I wish that were not so, but it is. Students have so many competing demands on their time (academic and non-academic) that grades become the arbiter of effort.
Indeed, am I grading hard enough? Deciding how high to set the bar is one of the most difficult problems in Higher Education. All else flows from it, but so far as I can tell, no one has any idea what we are shooting for. I don’t know if I have the bar set right in my course. About half the class got an A or an A-; a quarter think I grade too hard. Is that about right?
Let’s say it is. What that means is that I need to figure out ways to help those who are feeling really stretched. Somehow I have to figure a way to encourage the unhappy 25% to make use of the resources I have available. And to figure out what else would help.
climbing2.jpg
For everyone’s sake, we need to figure ways to help them achieve – without lowering the ambition.

Customer Satisfaction 1. The numbers

The Student Rating of Teaching Effectiveness results are just in. Students are asked to rate a variety of things on a scale of 1-7. It is hard not to fixate on these numbers, despite their shortcomings. And their shortcomings are many. The scores almost certainly reflect something, but is it teaching effectiveness? The scores can’t be used to compare courses, not least because audiences differ (most of my students aren’t there by choice), or to compare teachers (popular teachers are not necessarily effective teachers). But the scores are good for identifying under-performing teachers (usually scores <4), and for comparing the same teacher across years. 

Evidently I am getting worse.

SC200 scores2.jpgThe shape of the curves fit with my general experience of university teaching. The second year of a new course is the best. You’ve ironed out the kinks, but the material is still fresh and exciting to teach. It is hard not to lose the edge once things become routine.
But looking more closely at these numbers, I can’t help but wonder if something else is going on. One of the questions asked is ‘Rate the clarity of the syllabus in stating course objectives, course outline and criteria for grades’. The syllabus is the document which should lay out everything about the course. I have not changed mine over the four years. Moreover, I have never had a single complaint about it. Yet the students rate it as less and less clear through time (black dotted line). Is it possible students are getting more ornery?

The bottom line

Final Course Grades2.jpg
Class average: 88%. Three people failed, one who dropped out but did not withdraw, and two who did effectively no blogging. Among the 32 A’s are seven students who got more than 100% with the extra credit. In the absence of extra credit, the highest score would have been 97%.

Final exam 2013

I am always amazed how the final exam goes better than any of the class tests, even though all five tests are identical in format and difficulty. This year was no exception.

The breakdown: A, 42; A-, 30; B+, 19; B, 17; B-, 16; C+, 11; C, 6; D, 13, Fails: 7 + 3 no-shows.

Excluding the three no-shows, the class average was 86%, well up from the average of 70% in Class Test 1 and 4, and still higher than Class Tests 2 or 3. Nobody got all questions right, but eight students got more than the 25 of 28 questions needed for 100%. Best of all though, over 40% of the class got an A or an A-. In those earlier tests, there was a serious shortage of A’s – never more than 30% of the class.
I had a lot on in the few days after Class Test 4, and I struggled to find time to write the Final Exam. In the end, I ran out of time to get someone in my research group to test drive the thing, so I was a bit nervous there would be an error. In the end there was not. All the questions worked well. I hope the test made the students think. And realize how much most of them came on.

Impact

I do this course to make a difference. How do I know if I am? At the start of semester, I ran a survey to gauge the students’ attitudes to science. I ran the same survey last week so I could measure the impact of my course.

I made no impact. None. Zip. Denada. ZERO.

There must be something wrong with the survey.
For the record, after the course, the students agreed more forcefully with one of the 20 statements offered to them. Astute SC200 graduates will recognize 1/20 as the false positive rate expected by chance if in fact I made no difference.
[However, that particular statement concerned perhaps the key learning objective of the course, Feynman’s statement that ‘Scientific knowledge is a body of statements of varying degrees of certainty – some most unsure, some nearly sure, none absolutely certain’. I take a crumb of comfort from the possibility that students understood that better after a semester of my labors. And if that shift in attitude was in fact due to chance, how are we to know without repeating the experiment?]
The remaining questions:-
  • Scientists are curious about the world
  • I think about science in everyday life
  • Knowledge in science consists of many disconnected topics
  • To understand science, I sometimes think about my personal experiences
  • I take science credits because I want to make a contribution to society
  • It is important for the government to approve new scientific ideas
  • Learning about science changes my ideas about how the natural world works
  • Science has little relation to what I experience in the real world
  • There is usually only one correct approach to solving a science problem
  • Learning about science that is not relevant to human health is not worth my time
  • Mathematical skills are important for understanding the world
  • I enjoy explaining scientific idea that I learn about to my friends
  • The general public misunderstands many scientific ideas
  • For me, science is about learning known facts as opposed to investigating the unknown
  • Science is a civilizing enterprise that generates wonder and awe
  • I need experts to tell me what to think about a scientific claim
  • In my lifetime, science will change the way people will think about the human condition and humanity’s place in the universe
  • Well educated citizens need to know something of how science works
Students were asked if they strongly agreed, agreed, were neutral, disagreed or strongly disagreed with the statement. With perhaps the exception of the first question, where 82% of the students strongly agreed, less than 30% of students strongly agreed or strongly disagreed with any statement, and for the most part <10% strong agreed or strongly disagreed – in other words, if student attitudes had got stronger for or against any of those statements, there was plenty of room for the survey to have detected that. No such shifts occurred.
Return rates:- 84% at start of semester, 87% at end

Dear Professor

So far this week, I have spent over eight hours responding to students complaining about some aspect of their grade. Over 50 students have emailed since December 1 (30% of the class), some about several different issues. I judged three of the complaints to have merit.

K-RatherBeComplain_400x400_2_jpg_400x400_upscale_q85-300x300.jpg

The sheer volume of the deluge began to fascinate me, so I started collecting data. On average, it takes me 15 minutes to investigate, make a decision and write back. Often times, students respond to the response faster than I can process the original complaints, so the net volume of correspondence actually grows as I process it. Of the responses to the responses, about half are graceful; the other half disagree and require I respond to the response to the response. I respond to about a quarter of the responses to the responses to the response. Which I guess shows I am not being firm enough. Or that I have gone mad.
It felt like there were a lot more complaints this year, but as I tell my grad students, always get the data. Checking, I learned that this year is in fact only about 20% ahead of last year, but it is 50% ahead of the year before. Extrapolating, there won’t be enough hours in a week to do the responses by 2020. If we scale the class up, I will need to employ someone to handle all this. A shrink maybe.
The saddest correspondents are the ones that talk about having put in many hours on x, y or z, and not got the returns. I do feel sorry for those folk. Partly because they have put the hours in for disappointing returns, but mostly because those students seem to think that hours in equate to rewards out. That only works when you are flipping burgers or waiting tables. Maybe not even then. 
Perhaps none of this is the fault of the entitlement generation (the standard faculty explanation). Maybe I did something wrong? I thought I did my best job ever of defining expectations, that the graders did a better job of feedback on the blogs than ever before, and that Kira and Ethan, this year’s TAs, did an outstanding job and were available more or less 24/7. I ran as many revisions sessions as I ever have before; fewer students than ever took advantage. Attendance in class hit a record career low for me (<50% at the end). Not for the first time this year (test grades, blogs), I wonder what happened.