Many of us incorporate some level of teamwork in our courses. I typically teach in the College of Information Sciences and Technology, where nearly every course has some level of teaming. For my course in particular, IST 446, over 1/2 of the points in the course are based on team assignments. With this much emphasis on teaming, it is often difficult to fairly assess team work. I tried something a bit different this semester, and want to share what I felt were the pros and cons of my method.
First, I used the Comprehensive Assessment for Team-Member Effectiveness (CATME) system that originated from Purdue University. This is a web-based tool that allows faculty (regardless of university) to upload a course list with team data, and create assessments that allow students to rate one another based on 13 different categories related to teaming (all from teaming research and literature). This year, I used the following categories:
- Contribution to work
- Interacting with teammates
- Keeping Team on track
- Expecting quality
- Team satisfaction
The system will generate emails to all students that need to complete team evaluations. This is actually a 40-point assignment in my course, requiring each student to login to the system and evaluate their teammates. I tell them that if they complete the surveys for each team member, they will always start out with a 40/40. Then, based on their evaluations by their peers, I make adjustments to the grade of the assignment. Here are the steps I used:
- I created a ‘team average’ based on the scores of each team member. For instance, if we had 4 people per team, I simply added up their total scores and divided by 4 for the average.
- Next, I compared each individual’s average to the team average. Individuals that were below a whole point (on a 5-point scale) received a letter grade decrease for the assignment. Assuming the average was a 4.2, and someone received an average of a 3.1, that student just went from an “A” to an “A-“
- I then went to .50 intervals. So if the student had, for example, a 2.5, I would then drop the student to a “B+”.
Overall, this was good in concept but I implemented it poorly. Those students that were rated poorly by their peers brought the team average down very low, in essence making it very difficult to be less than a whole point below the team average. If I did this again, I would probably use .25 intervals to adjust grades.
Also, I did not find a way to incorporate the qualitative comments the system collects. On one team, for example, 3 of the 4 members all commented about the lack of participation of a single group member. My quantitative method did drop the student from an “A” to an “A-” for the assignment, but it was clear to me that I should have dropped the student further. But I did not have any method to somehow standardize on written feedback to impact the grade.
Lastly, the CATME system does a lot of interesting analysis for you, and highlights specific students that meet certain criteria. For instance, one student was flagged as “Overconfident – The team’s average rating for this student is less than 3, and the student has rated themselves more than one point higher on average than this rating”. Another student was flagged “Personality Conflict – This student has rated one or more team members a 2 or less, but the median rating of the student(s) by the other team members gives a score of at least 3. Perhaps this student just didn’t get along with the student(s) that got poor ratings?”
Finally, several students were marked as “High Performer – This condition indicates that the average rating for this student by the other members of the team is more than half a point higher than the overall average rating of the team. The students average rating must be higher than a 3.5 to qualify”.
My method this year didn’t do a good enough job penalizing poor performers, or rewarding high performers. I think moving to .25 increments from the team mean will help me better penalize poor performers, but I’m still not sure how to best reward high performers based on the current structure of the assignment. Ideas?