Wednesday, July 28, 2010

The Value of a Good Teacher

In the middle of this discussion on how to really help MCAT students, the New York Times drops this article on how much a good kindergarten teacher is actually worth.

This is too rich a topic to interrupt the discussion, so I'll keep my comments brief and let my readers do the unpacking. First, does anyone really think that standardized test scores are a good way to measure student progress? If the folks in the Obama administration are really interested in improving public education, start by coming up with a better metric. I'd suggest testing for creativity, but I think the irony of the suggestion might be lost on those who insist on sticking by standardized tests.

Which leads me to the second and last thing I want to say. Nothing is more important to our country's future than the quality of our children's education. Of all the things academics are trying to do, how about some of them follow this study's lead and really try to probe the value of good teachers at all levels, the value of various class sizes, of having computer training for elementary school students (I'm guessing negative value) and so on? Then, and I know it is a lot to ask, perhaps we could have an informed discourse on the proper spending levels for public schools.

Monday, July 26, 2010

Dunning-Kruger: Pre Med Students

This is a continuation of the post here on the Dunning-Kruger effect, which can be summarized as follows:

"When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it."

In order to ground our discussion of the Dunning-Kruger effect and show its practical side, it is time to make the effect personal. One of the many things I do to earn money is teach MCAT, the standardized test all medical school students take, a sort of SAT for the pre-med. The MCAT is broken into three parts, a verbal part, a physical sciences part which tests the students' knowledge of two semesters of college chemistry and two of physics, and a biological sciences part which tests the easier parts of about seven semesters of college organic chemistry and a variety of biology disciplines. Each section is scored from 1-15 for a possible total of 45 points. Generally 10's on each section, a 30, is required to get into a decent school. A score lower than a thirty or a poor split (i.e. 11/11/8) often relegates a student to a lesser school, some of which only grant a DO degree (Doctor of Osteopathy) instead of an MD. (You'd be surprised how many general practitioners, the doctor you're most likely to see first in the course of an illness, do not have an MD.)

I get a wide swath of students. The best come in scoring about a 30, while the worst come in scoring about a 15. The majority sit right around 24. This most common type of student would probably score about a 27 if she studied on her own outside of a formal review course. She's right on the cusp. She knows she isn't the smartest student, she knows she needs help and that's usually why she's signed up for the course I teach.

In my years teaching MCAT, I've observed a strange phenomenon. My students work very hard, they are very motivated. They really want to become doctors, to the point that they're willing to devote eight hours a day for a whole summer and $2000 or more to getting into medical school. But the most common outcome for this student who comes in scoring a 24, and who could score a 27 studying on her own, is that she scores a 28. The smallest of improvements. Watching this story play out course after course has led me to wonder what's going wrong, why is the course not helping a student who is so motivated to succeed?

Before I answer that, let me discuss the students who come in scoring a 30 or a 15. The typical student who comes in scoring a 30 is a good student who really doesn't need my class to get into medical school, but who prefers the structure of a class to studying on her own. She asks few questions during lectures, nods her head often, is less stressed then everyone else, and typically scores about a 36 on the real MCAT, an impressive 6 point improvement, all the more impressive because she has less to learn. In other words, the student who goes from a 24 to a 28 does so by improving on medium difficulty questions whereas the one who goes from 30 to 36 learns to answer the difficult questions right.

On the other side of the equation is the student who comes in scoring a 15. This student usually doesn't end up taking the MCAT. He will end the course still scoring about a 15, realize that it isn't good enough, and decide to save the $200 test fee. The typical 15-scorer always comes to any extra help sessions I give and sometimes even takes the course again. From what I can gather, motivation is not the issue for this student.

Thinking about the best and the worst students in my class has led me to realize is that most students who come in scoring a 24 think that they are not getting the score they want either because they need more practice or because they don't know the material well enough. Depending on their perception, they will either spend countless doing practice exams and practice questions or spend countless hours studying. They are partially right, and that is why they improve from a 24 to a 28. But they are mostly wrong. What they miss is that there is some deficiency either in the way they take a test or the way they study. It is not a matter of practice, but a matter of approach. Almost all of them are poor problem solvers. Some calculate carelessly. Others read the questions too fast and answer a different question than the one they are asked. Others lack confidence, or have jumpy minds, and are always second-guessing the answer they get. Many have poor reading comprehension, they study the textbooks but the information doesn't get into their head. Or they don't understand that reading and memorization are different tasks and mistakenly assume that if they read everything, of course they must know it.

Thus, the problems these students face are entirely results of a sophisticated form of the Dunning-Kruger effect. They know, as they should, that their scores are lacking. But they assume the flaw is that they are not working hard enough. This assessment leads them in a circle. When their score doesn't improve, they simply put their head down and decide they must work harder still. However, what they need is to change the way they work, to lift their head up and learn to work smarter instead of harder. This is what they should know, but don't.

Hopefully that is where I come in, but it is not easy to get them to make the next step. These students have been at the back of the pre-med pack for three or four years of college. They know there are students who are more advanced than they are but often they've chalked it up to simply not being as smart as those other students. They need to have it rammed into their heads that those other kids aren't any smarter, intrinsically, they just know how to learn. They need to be reached by someone, hopefully me, who can teach them the best methods to solve problems, how to read and retain information better, exactly which pieces of information must be memorized. They need, in other words, to re-evalute their self-image and to re-learn how to learn. It is not easy, it is not the work for a single course. Still, if they succeed, they will be better for the rest of their lives. Med school will be easier for them, they will become better doctors, hopefully they will even become better citizens.

That is my analysis of my students. In a reassuring yet forceful manner, I try to convey my observations to them, to point out to them that the reason they continue to perform poorly despite hard work is a flaw in they way they evaluate themselves. My results are, to say the least, mixed. Next time, I'll explain how, despite understanding what is going on, as a teacher I too am caught up in the Dunning-Kruger effect and its consequences.

Wednesday, July 21, 2010

The Difficulty of Self-Assesing Competency

There was a thought-provoking but poorly organized essay by Errol Morris over at the New York Times a few weeks ago. There's so much in it and so much that comes out of thinking about it that I'm having difficulty coordinating everything I want to say about it. This first post on it is going to be a little dry, so bear with me. The resulting discussions and insight will be worth the initial dryness.

The essay itself is broken into five parts. I found it worthwhile to read the first, fourth and fifth parts as well as the first four paragraphs of the second. I strongly encourage you to read those portions, but I will do my best to summarize the relevant parts as we go.

Before I begin the summary and clarification, I want to offer a brief criticism of the article for those who do read it. Many people will probably find the second and third sections of the essay interesting as well but I found that the majority of the second portion was a somewhat obscure history lesson that confused rather than illuminated what Morris was driving at. Along the same lines, I found that while part 3 was a very vivid and historically important example of anosognosia, it also didn't illuminate Morris's points in any significant way and was more of a sidetrack. The presence of these sections forced Morris to, somewhat awkwardly, resummarize the beginning of the essay again in parts 4 and 5. And the digression away from the core of the problem of anosognosia leads me to my second criticism of the essay. Until the epilogue, Morris doesn't answer any of the questions he so provocatively raises. In fact, by the time we get to the epilogue, Morris is answering different questions from those he had been asking throughout the course of the essay and, to my mind, giving very unsatisfactory answers to those questions he does directly address.

Connected to this, the most interesting parts of the essay by far are the responses he elicits from the people he interviews. While they sometimes answer the questions he raises, their answers are at best incomplete. I know some people like the journalist to stay out of things and just report (whatever they think that means in an essay piece) but I think when you're writing an essay on philosophy, psychology and neuroscience like this, you need to give an opinion. The majority of the time you will know the subject far better than your reader and by not telling him what you learned and where you stand, you are robbing him of your expertise. Most people will just let an article like this pass in and out of their head and you cheat them of the biggest insights when you deny them the answers to the questions you raise, even if your answers are poor and incomplete. The advanced reader will still be able to see past your conclusions and come to his own.

I have other, more quibbling, problems with the essay which I'll get to as they come up. 

To briefly summarize the article, Morris is chiefly interested in what's called the Dunning-Kruger effect, which is summarized as follows:

"When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it." 1

To illustrate this phenomenon, Morris offers the case of a bank robber who was told that if he rubbed lemon juice on his face, the cameras in the banks wouldn't be able to record his face. Not quite believing this, he rubbed lemon juice on his face, took a picture, and found that his face was not in the picture. Turns out that, with lemon juice in his eye, he mis-aimed the camera and snapped a photo of some part of the room where he wasn't. Regardless, now a convert to the powers of lemon juice, he proceeded to rob two banks squinting all the while and looking squarely in the cameras. He was caught rather quickly. This is certainly not the only case you'll run into in your life of people being almost unbelievably stupid, but it does raise some pressing questions.

How can people be so stupid? And, by extension, are even the most successful of us doing really stupid things and simply not realizing it? Or, perhaps more directly, what really stupid things are even the most successful of us doing and how do we find out?

The need to explain the Dunning-Kruger effect brings Morris to the concept of anosognosia. There are many definitions he uses throughout the article, but I'm going to go with this one:

"The state of not knowing something you should know."

To illustrate anosognosia he cites an example of a woman paralyzed on her left side who is not aware that she is paralyzed. He asks her to touch his nose with her right hand, which she does. Then he asks her to touch his nose with her left hand. Though she claims that she is raising her left hand towards his nose, it actually remains lying in her lap. She really has to know at this point that her arm isn't moving, but she's in a state of anosognosia. He leaves her for a time and then returns, again asking her to touch his nose with her left hand. This time, she uses her right hand to pick her left arm up so that her left hand touches his nose. You can see the confusion in her mind. Some parts of her clearly know what is going on, and some parts clearly don't.

In the fifth part of the essay, Morris quotes Dunning tying anosognosia and the Dunning-Kruger

The road to self-insight really runs through other people. So it really depends on what sort of feedback you are getting. Is the world telling you good things? Is the world rewarding you in a way that you would expect a competent person to be rewarded? If you watch other people, you often find there are different ways to do things; there are better ways to do things. I’m not as good as I thought I was, but I have something to work on. Now, the sad part about that is — there’s been a replication of this with medical students — people at the bottom, if you show them what other people do, they don’t get it. They don’t realize that what those other people are doing is superior to what they’re doing. And that’s the troubling thing. So for people at the bottom, that social comparison information is a wonderful piece of information, but they may not be in a position to take advantage of it like other people."

I think this is one of those areas Morris screws up in not refining Dunning's answer. Dunning basically says that incompetent people don't realize the difference between a good way to do something and a bad way. But I don't think that is quite right. I think first that the incompetent have trouble figuring out whether or not they are being rewarded as a competent person ought to be rewarded. They ought to know, for instance, that they are poor or are getting bad grades. But they don't want to admit it, they're convinced that the problem is not them but the world's perception of them.

That's one way to see it. Another way is that competent people realize when the screw things up, and do things so as not to repeat the same mistake in the future. I don't think competent people need outside confirmation to know when they've done things poorly or well. On the other hand, incompetent people don't realize when they screw things up. Perhaps another way of putting it, the incompetent don't see the difference between the better way to do things and the way they are doing things. And if you don't see the difference, of course you can't realize one way is better than another. But that is only part of the answer. I think some part of them does know the difference, but for a variety of reasons, they don't want to acknowledge the difference.

In failing to make this explicit, in leaving Dunning's incomplete answer as the last word, Morris does his essay and his topic a major disservice.

Before I move on to what all of this has led me to think about, I want to make explicit some things Morris confuses a bit. First, there are three relevant types of knowledge which Morris doesn't break out but which he probably ought to have:
  1. Factual knowledge
  2. Knowing how to do something
  3. Knowing about oneself, having an accurate self-representation
These three types have a good bit of overlap, but they also have particular differences. I'm not going to go into even more dry detail about why I've separated these types of knowledge as I have, hopefully I'm competent enough that my meaning is clear.

Second, Morris specifically breaks potential questions into different types. Expanding on his essay a little we have:
  1. Answered questions: questions whose answer we either have memorized or know where to find
  2. Unanswered questions: questions we have thought to ask but don't yet know how to answer
  3. Unasked questions: questions we haven't yet thought to ask
  4. Unanswerable questions: questions that we will never be able to answer
There are of course, several permutations of these (for example, are there unaskable questions?, partially unanswered questions?) but the above is sufficient for our purposes.

1. Justin Kruger and David Dunning, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” Journal of Personality and Social Psychology, 1999, vol. 77, no. 6, pp. 1121-1134.

Friday, July 16, 2010

Philosophy for Daily Life

I mentioned in my last post that I'm an equal-opportunity critic. It's now philosophy's turn to play the whipping boy, though with the tremendous admiration science garners in our society, this is a bit like asymmetric warfare.

The New York Times recently asked two philosophers to defend philosophy from the criticism that philosophers are too far removed from the daily lives of most people. Both answered with lengthy essays. The first ended up being a discussion of the proper level of abstraction and complexity in philosophy. The second could be summarized as a short biography of Plutarch that attempts to argue that philosophy should be fun and should be practiced at dinner parties. If you feel like wasting your time, you can read the essays at a post titled (apparently without conscious irony) Lost in the Clouds.

Now, both essays did say some thing that were true, but neither actually addressed the criticism which they were supposed to address. I think this is, to a large degree, what is wrong with Philosophy. That is, if you ask a philosopher a question, he may say a lot of things, some portion of which may end up being either true or interesting, but he's unlikely to actually answer your question. This by no means is a problem confined to these essays. My personal experience having essentially double majored in Philosophy is that American Philosophy departments are full of people who like to argue about things, who are impressed by the cleverness of the mental loops they can execute, but who can't actually answer any questions or address any problems. In particular I've seen a disturbing tendency to argue about what a specific word means when it's obvious to everyone that words have multiple meanings. As a specific example, I took a class where we spent a whole semester reading philosophers who were trying to give a single, unifying definition to the word good such that it would have the same meaning when you said "good dog", "he does good", "he is a good friend", or "this food tastes good". The professor was so lost in the trees, he couldn't see that these philosophers he so admired were making a mistake a fifth grader would catch.

Now, I'm not one of those who feels he can criticize without offering an alternative. While the real answer is far beyond the scope of a blog post (I have a finished 7000 word essay on the purpose of the humanities which I may extend into a book someday), I can offer some pointers here as to the right direction. To that end, there are a number of great moral questions in our society that are becoming more pressing but which continue to go unaddressed. What is the best way to use money, when do you have too much, and what should you do with the extra? Given the other needs in our society, how much is it worth to add one year to the end of our lives and who should make the decision about when to pay and when to stop paying for treatments? If philosophers began to address these and many other practical, thorny, and important questions, they would gain more respect in society. In addition, our society dreadfully needs to be educated on how to spot arguments with logical fallacies and arguments made in bad faith, areas that philosophers are uniquely qualified to address, and not coincidentally, areas I'll touch on a good bit on this blog.

Wednesday, July 14, 2010

The Purported Science of Wisdom


One of the common themes that is going to keep coming up on this blog is how there are specific areas that science just doesn't belong. This is not to say, however, that we should ignore the conclusions of science or that the humanities and religious thought aren't also susceptible to overreach, but those are topics for another day. An example of scientific overreach is highlighted in this review of the book Wisdom: From Philosophy to Neuroscience.

There are, according to Hall and the researchers he meets, eight attributes of wisdom: Emotional Regulation, Knowing What's Important, Moral Reasoning, Compassion, Humility, Altruism, Patience, and Dealing with Uncertainty.

The huge flaw in breaking out wisdom in this way is that the eight attributes really all boil down to one: Understanding What's Important (Knowing is such a science word), especially because displaying any of the other traits in excess is downright unwise. For example, Humility is understanding that the things we don't know are often more important than the things we do know. But the truly wise will recognize that humility in the face of foolishness is overrated. Likewise, Patience is understanding that waiting is often more important than not waiting. Compassion is understanding that it is important to acknowledge the suffering of others and commiserate with them. Still, wisdom sometimes dictates knowing when people need to be told to just suck it up and keep on trucking. Emotional Regulation is understanding it is important to control your emotions or, sometimes, to let them out. I'll leave it to you to construct the rest, but it really seems that in naming Knowing What's Important as a single component of wisdom, scientists really just end up demonstrating that they don't know what's important.

If all you're left with, then, is that wisdom is understanding what's important, well, you haven't really defined anything, you've just restated the problem.

That said, I think the most disturbing oversight in this attempt to break wisdom into its constituent pieces is the failure to examine where wisdom comes from and why it's important. I mean, watching "wisdom" flash through an fMRI is great and all, but it's really just a parlor trick unless you can actually show us how to get more of it. To that end, I think a large unmentioned aspect of wisdom is understanding that sometimes suffering enriches our experience of life. I can tell you from personal experience that living for a year in China, going through break-ups, and studying difficult texts were not the most pleasant experiences in my life. But they were the experiences that shaped me the most, and had the most to do with teaching me what is valuable and how I could become a better person. I would not give them up for anything. But I think society as a whole would be very reluctant to embrace this aspect of wisdom, and if science is going to do the humanist any favors, it could attempt to understand why.

Monday, July 12, 2010

Did LeBron James Wimp Out?

There's a lot of talk in the media that LeBron James made a mistake by going to the Miami Heat, that he took the easy way out and that if he wins a title it will be perceived as tainted. Let me be clear, from a publicity standpoint, I think he handled this terribly. From a personal standpoint, I would have liked to see him as a Net or a Knick But anyone who says that going to the Heat is a mistake or a cop-out is just dead wrong.

We don't quite know yet how good LeBron the basketball player is. The average model says that he's been worth approximately 25-30 wins for Cleveland in the regular season. Consensus also seems to be that no matter how many wins he gets in the regular season, without supporting players as good as those that Kobe has in Los Angeles, it will be very difficult for him to win an NBA title. The truth is, we will never know what the future would have held for him had he decided to play for the Cavaliers, the Knicks or the Bulls. But that doesn't mean that we can't draw certain conclusions about his various potential futures.

For the purposes of this thought exercise, I believe that however good LeBron is, it doesn't matter where he plays. LeBron's skills are set. Maybe he's worth twenty regular season wins, maybe forty. Maybe he can lead mediocre teams to a title, maybe he requires another All Star playing alongside him to get a title, maybe he requires another Hall of Famer alongside him. Whatever his value, if we don't know it now it is not because his talents are going to change, but because we don't have yet have enough information to accurately assess them.

The point is, I believe there is a high estimate scenario for LeBron and a low estimate scenario and that we actually know what they are. We're going to think about the Cavaliers, the Knicks, and the Heat assuming that the Cavaliers represent a scenario where he'd probably rarely have another All-Star alongside him, the Knicks a scenario in which he would have had at least one and possibly two All-Stars on his team, and the Heat a situation in which he plays each year with a future top-notch Hall of Famer and another All-Star. If LeBron is as talented as billed, the high estimate scenario, I believe he would win two titles with Cleveland. He'd be billed a national hero by the Midwest press, but when you compare his legacy to Kobe and Jordan he'd fall short. Two titles simply don't compare with six, no matter how much work those two required. If he played with the Knicks in this high-talent scenario, I believe he'd win three to five titles depending on who he was surrounded with. Again, four titles would put him in the conversation about who the greatest post-1990 player was, but probably not at the top. Playing with the Heat, assuming no injuries to Bosh or Wade, I think LeBron will win seven or eight titles. If those three players are as good as we think they are, they simply shouldn't lose as long as they keep playing together. In this scenario, LeBron is squarely in the conversation with Kobe and Jordan. He'll have more titles, but with a better supporting cast. It will be almost impossible not to consider all three as equals with the best being a matter of opinion.

Then there is the low-talent scenario. Here, it turns out that LeBron isn't nearly as good as we think, perhaps he shows a habit of poor play in the playoffs, perhaps his hunger fades after a few years. If he stayed with Cleveland and this were his true talent level, he would never win a title. He'd be written off as a tremendous disappointment, a player who fell short of expectations in a way no one ever has before. In New York I think he'd win a title or two, making him a very good player, a hero to Knicks fans, but someone clearly not in the realm of the greats and perhaps no more highly regarded than a player like Stoudemire or Ewing. In Miami, alongside Wade who's already won on his own, I find it hard to believe that this diminished LeBron wouldn't still win at least three or four titles. Again, he'd be perceived as something less than what he could have been, but it's tough to argue with that many titles, even with so much help. He wouldn't be in the conversation with Jordan and Kobe, but he'd be in the next tier of greats.

Looking at these alternatives, I think it is clear that for the high and low talent cases I've outlined as well as for the in-betweens, LeBron's legacy is best served by playing in Miami, with the Knicks a close second. However talented he turns out to be, he'll win the most titles there, and I think history will remember that Jordan had Pippin and an excellent supporting cast while Kobe had Shaq/Gasol and a similarly talented team. That LeBron choose his cast himself rather than waiting, like Ewing, for it to materialize around him only shows his understanding of the way basketball greatness is evaluated. Those who decry his move to Miami by saying that if he wins there his title will be tainted will do well to remember that the number of titles he wins depends more on who he plays with than on his own talent and that he's more interested in how the last title he wins will look than the first.