Dunning-Kruger: Teachers of Pre-Med Students

Continuation of this discussion on teaching pre-med students.

So I find myself in something of a conundrum. I recognize the reasons that my students are unable to obtain the scores they want on the MCAT. Broadly, they either have poor study skills or they have poor problem solving skills. But I'm left banging my head against the Dunning-Kruger effect:

"When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it."

Basically, in this situation what it means is that those students who are capable of realizing they are doing things poorly and improving have already realized their flaws and tried to fix them. That is the difference between the 30-to-36 student and the 24-to-28 student.

As I've come to understand these things, I've come to understand that I too am flawed. I stand at the front of the classroom and lecture on things (the facts of chemistry and biology important to know for the MCAT) that ultimately don't address the fundamental needs of these students. While it might be flattering to think that because I understand what the true needs of my students are, I'm better than the other teachers, despite my understanding I'm not really sure how to help them. I, too, am a victim of Dunning-Kruger. As a teacher, I have an obligation to help these students learn, but I am unable to figure out what to do. There are teaching techniques, ways of reaching and influencing students that I should know but which I don't know.

So I come back to these different types of questions Morris proposes in his article on anosognosia. I've asked a question that I don't know how to answer, a question I believe no one actually knows how to answer: How do you help students become better test-takers? Take problem solving skills as an example, I've done some research on the issue. It turns out that problems are solved in two steps. Before a test, one must internalize the steps necessary to solve a large number of types of problems. Then, when taking a test, one must properly categorize each particular problem as one of those types and then execute the steps we carry in our minds. But knowing that this is how we solve problems only raises other issues. Without accounting for time constraints and class sizes, how does one teach an abstract skill such as categorizing a problem as one of the hundreds of types a student ought to know, and then successfully following through on each of the steps? How, for that matter, do I teach students whose problem might be poor reading comprehension how to read better?

I'm not going to pretend I have the answers to these questions. This is kind of where I am right now, trying to figure out if anyone else has the answers or if there even are answers. I'm trying a few techniques out to see how they work. For a while I have been telling students that when they get a problem wrong, they need to understand why they got it wrong. That, in other words, practice without improvement is useless, that it doesn't matter how many questions you do if you don't learn to do any of them better. But as Dunning has suggested, I'm always frightened to find out, in one-on-one discussions, that students haven't listened to my advice and continue to plow through practice problems unreflectively, valuing quantity over quality. Of the students who do listen, many don't really understand the advice. They look for the reason they got the problem wrong, but they don't actually try to go through the right method to solve the problem step by step to get that routine in their mind. It's a subtle, but extremely important difference, knowing what you did wrong versus knowing how to do it right the next time.

I've also taken to more explicitly telling students the steps I'm taking as I solve a problem. For all MCAT questions the first step is determining if the answer was in the test passage, if it draws on things the student is expected to know, or if it is a fusion of the two. But there are more steps past that. What concept is being tested? What are the relevant equations, facts and rationalizations surrounding that concept? This technique has helped a little in that while most often a student who can't get started just can't figure out what the question is asking (a reading comprehension issue), a significant  portion of the time I'll uncover that the student simply doesn't know something he ought to know. But this leads back to another thing I've had difficulty understanding. If I tell students that they ought to memorize things, why do they not memorize them? When I took a science test, I always made a list of things I ought to know (and memorized them) and things I ought to understand (and made sure I could explain them). Many students seem to lack this skill. More, as Dunning-Kruger teaches, they seem to be unaware that they ought to know this skill. Basically, every time I find a trick that seems to help, I'm brought back to the fact that these students need to learn to help themselves, my ability to help them is limited.

This need to learn to help themselves, and the centrality of reading comprehension issues in the difficulties these students face, brings me to the post I made a few days ago. Good teachers, especially early in education, are immeasurably important. While I would love to be able to address these issues, I simply can't given the format of my interaction with these students and the amount of time we have together. Typically I have students either in a one-on-one setting for eight hours or in groups of ten for twenty hours. In that time, I have to teach them material which it takes college professors 45 hours to cover, while also doing a fair number of sample problems together, discussing time management and other test strategies, and covering other logistical issues of the class and of the MCAT. Given these restrictions, how much can I really expect to do in terms of re-teaching these students how to approach a test? More broadly, if we have no restrictions, how do we teach young students or re-teach older students how to solve problems and how to read effectively?

I think the answer is we simply don't know. Good elementary school teachers and good parents teach these things, but we don't know how they do it. More, I've over-generalized in my description of students. I've seen students who come in with an initial 24 and leave scoring in the mid 30's by, among other things, re-teaching themselves how to pay more attention to the wording of questions and re-teaching themselves using the problem-solving techniques I preach that don't reach the other students. I've seen students who come in scoring a 24 and don't improve through the length of the course. I refuse to believe that the 24-to-28 students, who are literally as close as they can be to going to a quality medical school, are beyond hope. If we want to do something that will improve our society more than any scientific advance in the history of the world, all we have to do is answer one simple question: How do we raise children, how do we re-teach older students, so that they are not subject to the Dunning-Kruger effect?

Dunning-Kruger: Pre Med Students

This is a continuation of the post here on the Dunning-Kruger effect, which can be summarized as follows:

"When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it."

In order to ground our discussion of the Dunning-Kruger effect and show its practical side, it is time to make the effect personal. One of the many things I do to earn money is teach MCAT, the standardized test all medical school students take, a sort of SAT for the pre-med. The MCAT is broken into three parts, a verbal part, a physical sciences part which tests the students' knowledge of two semesters of college chemistry and two of physics, and a biological sciences part which tests the easier parts of about seven semesters of college organic chemistry and a variety of biology disciplines. Each section is scored from 1-15 for a possible total of 45 points. Generally 10's on each section, a 30, is required to get into a decent school. A score lower than a thirty or a poor split (i.e. 11/11/8) often relegates a student to a lesser school, some of which only grant a DO degree (Doctor of Osteopathy) instead of an MD. (You'd be surprised how many general practitioners, the doctor you're most likely to see first in the course of an illness, do not have an MD.)

I get a wide swath of students. The best come in scoring about a 30, while the worst come in scoring about a 15. The majority sit right around 24. This most common type of student would probably score about a 27 if she studied on her own outside of a formal review course. She's right on the cusp. She knows she isn't the smartest student, she knows she needs help and that's usually why she's signed up for the course I teach.

In my years teaching MCAT, I've observed a strange phenomenon. My students work very hard, they are very motivated. They really want to become doctors, to the point that they're willing to devote eight hours a day for a whole summer and $2000 or more to getting into medical school. But the most common outcome for this student who comes in scoring a 24, and who could score a 27 studying on her own, is that she scores a 28. The smallest of improvements. Watching this story play out course after course has led me to wonder what's going wrong, why is the course not helping a student who is so motivated to succeed?

Before I answer that, let me discuss the students who come in scoring a 30 or a 15. The typical student who comes in scoring a 30 is a good student who really doesn't need my class to get into medical school, but who prefers the structure of a class to studying on her own. She asks few questions during lectures, nods her head often, is less stressed then everyone else, and typically scores about a 36 on the real MCAT, an impressive 6 point improvement, all the more impressive because she has less to learn. In other words, the student who goes from a 24 to a 28 does so by improving on medium difficulty questions whereas the one who goes from 30 to 36 learns to answer the difficult questions right.

On the other side of the equation is the student who comes in scoring a 15. This student usually doesn't end up taking the MCAT. He will end the course still scoring about a 15, realize that it isn't good enough, and decide to save the $200 test fee. The typical 15-scorer always comes to any extra help sessions I give and sometimes even takes the course again. From what I can gather, motivation is not the issue for this student.

Thinking about the best and the worst students in my class has led me to realize is that most students who come in scoring a 24 think that they are not getting the score they want either because they need more practice or because they don't know the material well enough. Depending on their perception, they will either spend countless doing practice exams and practice questions or spend countless hours studying. They are partially right, and that is why they improve from a 24 to a 28. But they are mostly wrong. What they miss is that there is some deficiency either in the way they take a test or the way they study. It is not a matter of practice, but a matter of approach. Almost all of them are poor problem solvers. Some calculate carelessly. Others read the questions too fast and answer a different question than the one they are asked. Others lack confidence, or have jumpy minds, and are always second-guessing the answer they get. Many have poor reading comprehension, they study the textbooks but the information doesn't get into their head. Or they don't understand that reading and memorization are different tasks and mistakenly assume that if they read everything, of course they must know it.

Thus, the problems these students face are entirely results of a sophisticated form of the Dunning-Kruger effect. They know, as they should, that their scores are lacking. But they assume the flaw is that they are not working hard enough. This assessment leads them in a circle. When their score doesn't improve, they simply put their head down and decide they must work harder still. However, what they need is to change the way they work, to lift their head up and learn to work smarter instead of harder. This is what they should know, but don't.

Hopefully that is where I come in, but it is not easy to get them to make the next step. These students have been at the back of the pre-med pack for three or four years of college. They know there are students who are more advanced than they are but often they've chalked it up to simply not being as smart as those other students. They need to have it rammed into their heads that those other kids aren't any smarter, intrinsically, they just know how to learn. They need to be reached by someone, hopefully me, who can teach them the best methods to solve problems, how to read and retain information better, exactly which pieces of information must be memorized. They need, in other words, to re-evalute their self-image and to re-learn how to learn. It is not easy, it is not the work for a single course. Still, if they succeed, they will be better for the rest of their lives. Med school will be easier for them, they will become better doctors, hopefully they will even become better citizens.

That is my analysis of my students. In a reassuring yet forceful manner, I try to convey my observations to them, to point out to them that the reason they continue to perform poorly despite hard work is a flaw in they way they evaluate themselves. My results are, to say the least, mixed. Next time, I'll explain how, despite understanding what is going on, as a teacher I too am caught up in the Dunning-Kruger effect and its consequences.