HN Books @HNBooksMonth

What about What Intelligence Tests Miss: The Psychology of Rational Thought by Stanovich [1], is it better?

[1] https://www.amazon.com/dp/0300164629

Reminds me of 'What Intelligence Tests Miss: The Psychology of Rational Thought,' the most interesting cognitive science book I've read. http://www.amazon.com/What-Intelligence-Tests-Miss-Psycholog...

In short, rationality is not the same thing as intelligence and general intelligence does not correlate with rational behavior nearly as well as we might like it to. Individuals with high intelligence can and do put their mental endowments to work to argue for mistaken ideas and execute irrational plans.

I'm now thinking of an intelligent individual who made a small misstatement in conversation, then, when challenged on that point, proceeded to explain why he was not mistaken for several minutes... then finally wised up and admitted that his original statement was wrong.

A book that helped change the way I think is What Intelligence Tests Miss: The Psychology of Rational Thought

http://yalepress.yale.edu/yupbooks/book.asp?isbn=97803001646...

http://www.amazon.com/What-Intelligence-Tests-Miss-Psycholog...

by Keith R. Stanovich. I'll quote here from a review of the book I wrote for friends on an email list about education of high-IQ children, and sum up an answer to your question in my last paragraph:

"For many kinds of errors in cognition, as Stanovich points out with multiple citations to peer-reviewed published research, the performance of high-IQ individuals is no better at all than the performance of low-IQ individuals. The default behavior of being a cognitive miser applies to everyone, as it is strongly selected for by evolution. In some cases, an experimenter can prompt a test subject on effective strategies to minimize cognitive errors, and in some of those cases prompted high-IQ individuals perform better than control groups. Stanovich concludes with dismay in a sentence he writes in bold print: 'Intelligent people perform better only when you tell them what to do!'

"Stanovich gives you the reader the chance to put your own cognition to the test. Many famous cognitive tests that have been presented to thousands of subjects in dozens of studies are included in the book. Read along, and try those cognitive tests on yourself. Stanovich comments that if the many cognitive tasks found in cognitive research were included in the item content of IQ tests, we would change the rank-ordering of many test-takers, and some persons now called intelligent would be called average, while some other people who are now called average would be called highly intelligent.

"Stanovich then goes on to discuss the term 'mindware' coined by David Perkins and illustrates two kinds of 'mindware' problems. Some--most--people have little knowledge of correct reasoning processes, which Stanovich calls having 'mindware gaps,' and thus make many errors of reasoning. And most people have quite a lot of 'contaminated mindware,' ideas and beliefs that lead to repeated irrational behavior. High IQ does nothing to protect thinkers from contaminated mindware. Indeed, some forms of contaminated mindware appeal to high-IQ individuals by the complicated structure of the false belief system. He includes information about a survey of a high-IQ society that find widespread belief in false concepts from pseudoscience among the society members."

So Stanovich, based on the studies he cites in his book, concludes that the cognitive strategy of being a cognitive miser (using the minimal amount of information and thinking possible, even if it is too little) is such an inherent part of the human condition that external incentives and societal processes of decision-making are necessary to overcome that weakness. He has a fair amount of optimism about filling mindware gaps through educational processes that would train more thinkers in correct reasoning (as, for example, the kind of statistical training that some but not all hackers receive during higher education). He suggests that actively counteracting contaminated mindware (which is something I have a penchant for doing here on HN) is considerably more difficult, because it is precisely high-IQ individuals who are best able to defend their irrational beliefs.

⬐ lifeisstillgood

Fascinating - sounds like the description Charlie Munger gives to his process. With less jokes :-)

AFTER EDIT: Thanks to all who have replied for the interesting comments. I discovered this link while digesting replies I received on three different email lists to a request to name experts on mathematically precocious young people. (That was for work.) Tanya Khovanova, the author of the blog post submitted here, was one name suggested to me as an expert on precocious mathematics learners. When I saw her personal website,

http://www.tanyakhovanova.com/

I remembered that I had seen her blog post "Should You Date a Mathematician?"

http://blog.tanyakhovanova.com/?p=319

posted to Hacker News (and other sites I read) before. I'll read more of her more purely mathematical blog posts over the next few days. I see one I can use right away in the local classes I teach to elementary-age learners.

On the substance of the post, I'm seeing several comments that equate "genius" to "person with a high IQ score." That was indeed the old-fashioned way that Lewis Terman (1877 to 1956) labeled a person with a high IQ score as he developed the Stanford-Binet IQ test. But as Terman gained more experience, especially with the subjects in his own longitudinal study of Americans identified in childhood by high IQ scores, he didn't equate high IQ to genius, and he became more aware of the shortcomings of IQ tests. Terman and his co-author Maude Merrill wrote in 1937,

"There are, however, certain characteristics of age scores with which the reader should be familiar. For one thing, it is necessary to bear in mind that the true mental age as we have used it refers to the mental age on a particular intelligence test. A subject's mental age in this sense may not coincide with the age score he would make in tests of musical ability, mechanical ability, social adjustment, etc. A subject has, strictly speaking, a number of mental ages; we are here concerned only with that which depends on the abilities tested by the new Stanford-Binet scales."

Terman, Lewis & Merrill, Maude (1937). Measuring Intelligence: A Guide to the Administration of the New Revised Stanford-Binet Tests of Intelligence. Boston: Houghton Mifflin. p. 25. That is why the later authors Kenneth Hopkins and Julian Stanley (founder of the Study of Exceptional Talent) suggested that is better to regard IQ tests as tests of "scholastic aptitude" rather than of intelligence. They wrote

"Most authorities feel that current intelligence tests are more aptly described as 'scholastic aptitude' tests because they are so highly related to academic performance, although current use suggests that the term intelligence test is going to be with us for some time. This reservation is based not on the opinion that intelligence tests do not reflect intelligence but on the belief that there are other kinds of intelligence that are not reflected in current tests; the term intelligence is too inclusive."

Hopkins, Kenneth D. & Stanley, Julian C. (1981). Educational and Psychological Measurement and Evaluation. Englewood Cliffs, NJ: Prentice Hall. p. 364.

So on the one hand there is the acknowledged issue among experts on IQ testing that IQ scores don't tell the whole story of a test subject's mental ability. A less well known issue is the degree to which error in estimation increases in IQ scores as IQ scores are found to be above the norming sample mean. Terman and Merrill wrote,

"The reader should not lose sight of the fact that a test with even a high reliability yields scores which have an appreciable probable error. The probable error in terms of mental age is of course larger with older than with young children because of the increasing spread of mental age as we go from younger to older groups. For this reason it has been customary to express the P.E. [probable error] of a Binet score in terms of I.Q., since the spread of Binet I.Q.'s is fairly constant from age to age. However, when our correlation arrays [between Form L and Form M] were plotted for separate age groups they were all discovered to be distinctly fan-shaped. Figure 3 is typical of the arrays at every age level.

"From Figure 3 [not shown here on HN, alas] it becomes clear that the probable error of an I.Q. score is not a constant amount, but a variable which increases as I.Q. increases. It has frequently been noted in the literature that gifted subjects show greater I.Q. fluctuation than do clinical cases with low I.Q.'s . . . . we now see that this trend is inherent in the I.Q. technique itself, and might have been predicted on logical grounds."

Terman, Lewis & Merrill, Maude (1937). Measuring Intelligence: A Guide to the Administration of the New Revised Stanford-Binet Tests of Intelligence. Boston: Houghton Mifflin. p. 44

Readers of this thread who would like to follow the current scientific literature on genius (as it is now defined by mainstream psychologists) may enjoy reading the works of Dean Keith Simonton,

http://www.amazon.com/Dean-Keith-Simonton/e/B001ITRL1I/

the world's leading researcher on genius and its development. Readers curious about what IQ tests miss may enjoy reading the book What Intelligence Tests Miss: The Psychology of Rational Thought

http://www.amazon.com/What-Intelligence-Tests-Miss-Psycholog...

by Keith R. Stanovich and some of Stanovich's other recent books.

Readers who would like to read a whole lot about current research on human intelligence and related issues can find a lot of curated reading suggestions at a Wikipedia user bibliography

http://en.wikipedia.org/wiki/User:WeijiBaikeBianji/Intellige...

occasionally used for the slow, pains-taking process of updating the many Wikipedia articles on related subjects (most of which are plagued by edit-warring and badly in need of more editing).

⬐ anothermachine

""Most authorities feel that current intelligence tests are more aptly described as 'scholastic aptitude' tests"

Hence the S, A, and T in "SAT".

⬐ cobrophy

All an IQ tests, is your ability to do well on an IQ test.

It's not a particularly great measure of intelligence. The fact that you can often improve your IQ by 20 points (or more) just by practicing the types of questions that turn up should be evidence of this. Did you just become massively more intelligent relative to the population in the 2 days you spent practicing the questions?

⬐ JoeAltmaier

Be fair; this is a correct comment, in line with the OP. Terman admited that IQ tests are only good at testing what is covered in the IQ test. A circular definition of IQ if there ever was one.

⬐ scott_s

I also find it odd when people equate a high IQ with being a genius. In how I view the concept, one has to be a genius at something. I don't think there is a general "genius" category. In other words, I don't view it as a description of potential, but of unparalleled achieved mastery and accomplishment in something.

A lot of the comments here are related to the idea of whether or not the SAT can be regarded as being much like an IQ test. It can, and psychologists routinely think of the SAT that way. Despite a number of statements to the contrary in the various comments here, taking SAT scores as an informative correlate (proxy) of what psychologists call "general intelligence" is a procedure often found in the professional literature of psychology, with the warrant of studies specifically on that issue. Note that it is standard usage among psychologists to treat "general intelligence" as a term that basically equates with "scoring well on IQ tests and good proxies of IQ tests," which is the point of some of the comments here.

http://www.iapsych.com/iqmr/koening2008.pdf

"Frey and Detterman (2004) showed that the SAT was correlated with measures of general intelligence .82 (.87 when corrected for nonlinearity)"

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144549/

"Indeed, research suggests that SAT scores load highly on the first principal factor of a factor analysis of cognitive measures; a finding that strongly suggests that the SAT is g loaded (Frey & Detterman, 2004)."

http://www.nytimes.com/roomfordebate/2011/12/04/why-should-s...

"Furthermore, the SAT is largely a measure of general intelligence. Scores on the SAT correlate very highly with scores on standardized tests of intelligence, and like IQ scores, are stable across time and not easily increased through training, coaching or practice."

http://faculty.psy.ohio-state.edu/peters/lab/pubs/publicatio...

"Numeracy’s effects can be examined when controlling for other proxies of general intelligence (e.g., SAT scores; Stanovich & West, 2008)."

As I have heard the issue discussed in the local "journal club" I participate in with professors and graduate students of psychology who focus on human behavioral genetics (including the genetics of IQ), one thing that makes the SAT a very good proxy of general intelligence is that its item content is disclosed (in released previous tests that can be used as practice tests), so that almost the only difference between one test-taker and another in performance on the SAT is generally and consistently getting all of the various items correct, which certainly takes cognitive strengths.

Psychologist Keith R. Stanovich makes the interesting point that there are very strong correlations with IQ scores and SAT scores with some of what everyone regards as "smart" behavior (and which psychologists by convention call "general intelligence") while there are still other kinds of tests that plainly have indisputable right answers that high-IQ people are able to muff. Thus Stanovich distinguishes "intelligence" (essentially, IQ) from "rationality" (making correct decisions that overcome human cognitive biases) as distinct aspects of human cognition. He has a whole book on the subject, What Intelligence Tests Miss, that is quite thought-provoking and informative.

http://www.amazon.com/What-Intelligence-Tests-Miss-Psycholog...

(Disclosure: I enjoy this kind of research discussion partly because I am acquainted with one large group of high-IQ young people

http://cty.jhu.edu/set/

and am interested in how such young people develop over the course of life.)

⬐ spitfire

Just a comment, because I know you get some stick when you bring this material up.

I do appreciate it, and think it really does need to be drummed into people. Particularly the material on job performance indicators(IQ and work samples everyone). Also, you should be consulting with this if you aren't already.

⬐ None

None

⬐ WildUtah

West Germany and East Germany, North Korea and South Korea, China and Taiwan

These nations exhibit a strong difference in adult height also. The reason is well known to be mass childhood malnutrition in command political-economies.

Ethical studies cannot reproduce the effect because malnourishing children is not ethical. Relationships between environment and heritable factors in human development are heavily dependent on context and cannot be objective by definition. When that context includes an overwhelming factor like childhood malnutrition or childhood lead exposure, the results will be extreme. The usual randomized controlled trials or twin studies or genetic marker studies cannot adequately deal with that kind of effect and are not really intended to.

Modern academic studies of IQ seem to refer to populations of well fed, healthy, well cared-for, children raised with free education according to a uniform curriculum in free liberal nations. That is very much a formula for shrinking environmental effects by shrinking environmental variance. It's the reverse of a twin study; you make the environment uniform so all the difference in outcome must be a result of heritable factors. Such studies often indicate that g is 60% heritable.

If you threw in some lead poisoning -- as was near universal in the 1950-1975 generation -- or childhood malnutrition -- very common before the twentieth century everywhere -- you would get a very different result. That's not a defect; it's built into the nature of these studies.

⬐ fosap

I am from Germany and you are the first person to tell me about malnutrition in the GDR. After a quick google i call it BS. Yes, the GDR was a pretty shitty state, but socialism per se does not eat babies.

⬐ Retric

Malnutrition is not a simple binary condition. However, while East Germany was one of the wealthiest areas of the Soviet Union and economically better off than many nations today the average diet was lacking by current western standards.

Don't forget this was the middle of the Green Revolution, but the USSR was still slightly behind the curve. http://en.wikipedia.org/wiki/Green_Revolution

⬐ jlgreco

Has the SAT been found to have a disparate impact on protected minorities, like other "IQ" tests have been? I know employers usually consider any sort of "IQ test" in America to be legally risky for discrimination reasons; I wonder if Universities could face similar issues.

A lot of those things, particularly the reading comprehension sections, seem like they could be heavily influenced by culture.

⬐ pcwalton

There have been a lot of accusations of that over the years, which was part of the reason why the analogies section was dropped. Here's an article from 2003 mentioning probably the most infamous SAT question in this regard ("runner : marathon :: regatta : oarsman" -- the issue of course being that wealthy students are more likely to know what a regatta is): http://articles.latimes.com/2003/jul/27/local/me-sat27

⬐ rdl

How did you get involved with SET? The people who ran that back when I was a student/study participant (20 years ago!) were amazing. I got a free college CS class when I was 12-13 out of the deal, which got me my first reliable dialup access to the Internet and UNIX/VMS shells for the next few years. It's hard to thank them enough.

⬐ imjk

How do you reconcile the large leaps in SAT scores that people achieve through preparing for the test? For example, while I had a fairly respectable score in high school, after I spent a little time working as an SAT tutor in college, I was regularly scoring perfectly on all practice tests and recently release new tests. Surely my IQ hadn't jumped drastically. Just curious for your take on this.

⬐ yummyfajitas

According to studies not performed by a company selling test prep, there is no large leap in SAT score.

http://online.wsj.com/article/SB124278685697537839.html

⬐ Evbn

Yet practicing old tests clearly does increase scores, and token adult's comment above did not account for that. He assumed that everyone practices and so the measurements are unbiased, or that practice is highly correlated with IQ (which may be true)

⬐ andylei

correlations are aggregate data measures that are only meaningful on large datasets. a high score on the SAT for one person does not imply that that person is smart. it just means that if you had a large population of people and you wanted to predict their intelligence, you could use SAT scores and get pretty good results.

⬐ tankbot

You don't reconcile. Tests of any kind have this problem as intelligence is not quantifiable, only knowledge and age. The GP is merely showing that the SATs are referenced as IQ tests. Since the same argument applies to IQ tests, even in this respect they are similar.

⬐ seanlinehan

What I find interesting is that I achieved a massive leap in my SAT scores without studying. As a disclaimer, I grew up in a lower-income area with terrible college prep and neither of my parents knew the college admission process. In short, I didn't actually know that you could study for the SAT. So the first time I took it, I did so completely blind and got an 1800. I took the test, again without studying, two months later and got a 2100. Given that both times I took the exam I was neither stressed out, tired, or even the slightest bit prepared, I'm not sure how to reconcile the 300 point leap.

⬐ aggie

Your IQ could see a similar jump if you spent a lot of time practicing for and taking IQ tests.

IQ is a psychometric measure of a construct, intelligence, so when you increase your IQ by getting better at taking the test, you are not actually increasing intelligence, just influencing the measure of it.

This is an interesting first submission by a Hacker News participant who joined the community 551 days ago. The core idea in the submitted blog post (by the submitter here) is

"Contrarian anecdotes like these are particularly common

http://news.ycombinator.com/item?id=4076643

http://news.ycombinator.com/item?id=4076066

in medical discussions, even in fairly rational communities like HN. I find this particularly insidious (though the commenters mean no harm), because it can ultimately sway readers from taking advantage of statistically backed evidence for or against medical cures. Most topics aren’t as serious as medicine, but the type of harm done is the same, only on a lesser scale."

The basic problem, as the interesting comments here illustrate, is that human thinking has biases that ratchet discussions in certain directions even if disagreement and debate is vigorous. The general issue of human cognitive biases was well discussed in Keith R. Stanovich's book What Intelligence Tests Miss: The Psychology of Rational Thought.

http://yalepress.yale.edu/yupbooks/book.asp?isbn=97803001646...

http://www.amazon.com/What-Intelligence-Tests-Miss-Psycholog...

The author is an experienced cognitive science researcher and author of a previous book How to Think Straight about Psychology. He writes about aspects of human cognition that are not tapped by IQ tests. He is part of the mainstream of psychology in feeling comfortable with calling what is estimated by IQ tests "intelligence," but he disagrees that there are no other important aspects of human cognition. Rather, Stanovich says, there are many aspects of human cognition that can be summed up as "rationality" that explain why high-IQ people (he would say "intelligent people") do stupid things. Stanovich names a new concept, "dysrationalia," and explores the boundaries of that concept at the beginning of his book. His shows a welcome convergence in the point of view of the best writers on IQ testing, as James R. Flynn's recent book What Is Intelligence? supports these conclusions from a different direction with different evidence.

Stanovich develops a theoretical framework, based on the latest cognitive science, and illustrated by diagrams in his book, of the autonomous mind (rapid problem-solving modules with simple procedures evolutionarily developed or developed by practice), the algorithmic mind (roughly what IQ tests probe, characterized by fluid intelligence), and the reflective mind (habits of thinking and tools for rational cognition). He uses this framework to show how cognition tapped by IQ tests ("intelligence") interacts with various cognitive errors to produce dysrationalia. He describes several kinds of dysrationalia in detailed chapters in his book, referring to cases of human thinkers performing as cognitive misers, which is the default for all human beings, and posing many interesting problems that have been used in research to demonstrate cognitive errors.

For many kinds of errors in cognition, as Stanovich points out with multiple citations to peer-reviewed published research, the performance of high-IQ individuals is no better at all than the performance of low-IQ individuals. The default behavior of being a cognitive miser applies to everyone, as it is strongly selected for by evolution. In some cases, an experimenter can prompt a test subject on effective strategies to minimize cognitive errors, and in some of those cases prompted high-IQ individuals perform better than control groups. Stanovich concludes with dismay in a sentence he writes in bold print: "Intelligent people perform better only when you tell them what to do!"

Stanovich gives you the reader the chance to put your own cognition to the test. Many famous cognitive tests that have been presented to thousands of subjects in dozens of studies are included in the book. Read along, and try those cognitive tests on yourself. Stanovich comments that if the many cognitive tasks found in cognitive research were included in the item content of IQ tests, we would change the rank-ordering of many test-takers, and some persons now called intelligent would be called average, while some other people who are now called average would be called highly intelligent.

Stanovich then goes on to discuss the term "mindware" coined by David Perkins and illustrates two kinds of "mindware" problems. Some--most--people have little knowledge of correct reasoning processes, which Stanovich calls having "mindware gaps," and thus make many errors of reasoning. And most people have quite a lot of "contaminated mindware," ideas and beliefs that lead to repeated irrational behavior. High IQ does nothing to protect thinkers from contaminated mindware. Indeed, some forms of contaminated mindware appeal to high-IQ individuals by the complicated structure of the false belief system. He includes information about a survey of a high-IQ society that found widespread belief in false concepts from pseudoscience among the society members.

Near the end of the book, Stanovich revises his diagram of a cognitive model of the relationship between intelligence and rationality, and mentions the problem of serial associative cognition with focal bias, a form of thinking that requires fluid intelligence but that nonetheless is irrational. So there are some errors of cognition that are not helped at all by higher IQ.

In his last chapter, Stanovich raises the question of how different college admission procedures might be if they explicitly favored rationality, rather than IQ proxies such as high SAT scores, and lists some of social costs of widespread irrationality. He mentions some aspects of sound cognition that are learnable, and I encouraged my teenage son to read that section. He also makes the intriguing observation, "It is an interesting open question, for example, whether race and social class differences on measures of rationality would be found to be as large as those displayed on intelligence tests."

Applying these concepts to my observation of Hacker News discussions after 1309 days since joining the community, I notice that indeed most Hacker News participants (I don't claim to be an exception) enter into discussions supposing that their own comments are rational and based on sound evidence and logic. Discussions of medical treatment issues, the main concern of the submitted blog post, are highly emotional (many of us know of sad examples of close relatives who have suffered from long illnesses or who have died young despite heroic treatment) and thus personal anecdotes have strong saliency in such discussions. The process of rationally evaluating medical treatments is the subject on entire group blogs with daily posts

http://www.sciencebasedmedicine.org/index.php/about-science-...

and has huge implications for public policy. Not only is safe and effective medical treatment and prevention a matter of life and death, it is a matter of hundreds of billions of dollars of personal and tax-subsidized spending around the world, so it is important to get right.

Blog post author and submitter here tylerhobbs suggests disregarding an individual contrary anecdote, or a group of contrary anecdotes, as a response to a general statement about effective treatment or risk reduction established by a scientifically valid

http://norvig.com/experiment-design.html

study. With that suggestion I must agree. Even medical practitioners themselves do have difficulty sticking to the evidence,

http://www.sciencebasedmedicine.org/index.php/how-do-you-fee...

and it doesn't advance the discussion here to bring up a few heart-wrenching personal stories if the weight of the evidence is contrary to the cognitive miser's easy conclusion from such a story.

That said, I see that the submitter here has developed an empirical understanding of what gets us going in a Hacker News discussion. Making a definite statement about what ought to be downvoted works much better in gaining comments and karma than asking an open-ended question about what should be upvoted, and I'm still curious about what kinds of comments most deserve to be upvoted. I'd like to learn from other people's advice on that issue how to promote more rational thinking here and how all of us can learn from one another about evaluating evidence for controversial claims.