SIGMA SOCIETY

​

However, the most difficult questions on the Stanford-Binet V can be easily solved by people with an IQ of 135 to 140. This produces a very large disparity between measured IQ and true IQ. Anyone with an IQ of 140, as long as they are fast enough and have a good cultural level, can reach over 200 IQ on this test, generating a gigantic amount false diagnoses of genius. This does not mean that distortions are always upwards. The way in which standardization is done, this would not be possible, because if it were like that, the average would be displaced. Therefore, upward distortions occur at approximately the same frequency and magnitude as downward distortions. As a result, really great people can score far below their true potential on this test and this has been proven several times. In the study carried out by Lewis Terman, starting in the year 1921, 1528 children with an IQ above 135, none of the 1528 selected children won a Nobel Prize, nor any other international prize of great importance in scientific areas or in Mathematics. But among the children who failed the test, two of them were awarded the Nobel Prize in Physics. This makes it evident that the Stanford-Binet, while very good and accurate for measuring IQs between 70 and 130, is not appropriate for higher levels. Terman's group included about 100 children with an IQ over 175, but none of them won 1 Nobel, with the average IQ of Nobel laureates in Science being 154. This is another serious inconsistency in the scores produced by the Stanford-Binet at the highest scores.

 

How Terman's study was carried out with people screened as children, it could be argued that the problem was not inherent in the test, but in the fact that they were screened too early. In fact, this is also one of the problems, but it is not the only one and it is not enough to explain all the observed anomalies. To better clarify this point, it is worth citing the cases of people registered in the Guinness Book for having the highest IQ in the world based on Stanford-Binet scores applied at different ages:

 

The first record of this modality in the Guinness Book occurred in 1966, in which Chris Harding was presented as the person with the highest IQ in the world, for having obtained a score 196-197 in the Stanford-Binet (I believe that the 1960 standardization form was used, Stanford-Binet L-M). In a normal distribution with a mean of 100 and a standard deviation of 16, only one in 1 billion people have an IQ above 196. However, the number of people screened with the Stanford-Binet was in the few thousand. In the standardization process, the samples were also in the few thousand. Thus, the best that could be done was to place the test ceiling close to 155 to 160, and even then there would still be the problem that the most difficult questions were at a difficulty level close to 140, so scores 160 would only indicate higher speed to solve problems level 140, instead of indicating an intellectual level of 160.

 

In the 1970s and 1980s, Kevin Langdon and several other people started showing up with scores of 196-197, claiming to share the record for the highest IQ. Some of the people who applied for registration as the person with the highest IQ in the world between 1966 and 1978 were:

 
•    Christopher Philip Harding
•    Kevin Langdon
•    Bruce Whiting 
•    Robert Bryzman 
•    Leta Speyer 
•    Johannes Douglas Veldhuis 
•    Ferris Eugene Alger

​

There were also other cases after 1978 claiming the record, with nominal IQs above 197: 

​

•    Kim Ung-yong with IQ 210
•    Marilyn vos Savant with IQ 230, then corrected to 228, then corrected to 218, then corrected to 186, then 190
•    Keith Raniere, with IQ 242 

​

Finally Guinness removed this modality. One of the likely reasons is that it became clear that there was not adequate standardization that would allow a fair comparison. The adjustment metrics from childhood to adulthood scores were skewed, the use of different tests also produced very different scores. Another reason that may have aggravated this situation was the controversy over Marilyn being accused of falsifying the dates in her report and Keith Raniere being arrested, accused of several crimes, including murder. In Marilyn's case, I think her version is very plausible. She claims she took the test at age 10, but it was incorrectly recorded on her chart as if she had been tested at 11 years and 4 months. About this controversy, to the point of knowing the facts, I side with Marilyn and I explain the reason: in 2004 and 2005, I worked as a consultant at the main Psychology publisher in Brazil, I standardized and revised several tests of IQ, and I could see that the number of registration errors in the data of the people examined was absurdly large, reaching more than 5%. It was very common for people registered with birth year 2040, birth month greater than 12, among others. So I think it's much more likely that the psychologist who examined her actually got the date incorrectly than that Marilyn lied about it. Considering Marilyn's history, I have no reason to question her sincerity, while the history of recording errors in psychometric reports is very frequent. In Keith's case, the facts and evidence against him are plentiful and unquestionable.

 

The important point is that a test applied to a few thousand people in the standardization process, does not allow to establish a ceiling above 160 with the aggravating factor that the ceiling of difficulty does not exceed 140. But even if the test was really able to measure correctly at the level up to 196 and even if everyone in the world had been examined with the Stanford-Binet (considering that some people would be children and others would be very old), it wouldn't expect to find more than 3 or 4 in the world with an IQ above 196. However, in a sample of a few thousand people there were 10 people with an IQ above 196, some reaching 242, whose rarity is many orders of magnitude outside the limit of the number of people ever born, with a rarity level of 1 in 2.86 × 10^15 where the number of people already born is about 10^11.

 

It is a fact that this sample of a few thousand is not representative of the general population.  Therefore it is natural that more people with high IQs would be found in this sample than in a random sample of the population. If you apply an IQ test to Harvard or Cambridge students, it is natural that the average score is much higher than the average score of the general population, and it is also very likely that some of the 10 smartest people in the US or the UK are in these institutions. The main problem is not the statistical anomaly. The biggest problem is that 100% of those people with IQs above 196 didn't stand out as scientists, mathematicians or authors of brilliant intellectual works that matched the measured IQs.

 

Marilyn herself, in an interview on the David Letterman show, made the following comment (excerpt from the interview):

D: I have uh, I have miserable teeth. I mean, they're healthy... [Paul laughs aloud] They're just odd, they're odd. You know, I can eat things through fences. [laughter] Not that there's any call for that, but uh... Allright, now Marilyn, let's get back to you and your... uh... head. [laughter] Uh, what uh... now how do we know you're the smartest woman in the world?

M: Well, you probably don't know that, I don't think anyone really knows that, not that many people have taken an IQ test. And so I had the highest score on the Benet... so far... but this very...

D: [trying to interrupt] Now when did you...

M: ...small minority of people in the world have taken a test, and... [dramatically] what did Benet know, for heaven's sake? [Paul & Dave both chuckle as Marilyn rambles] I mean back in 1904, he didn't... [laughter] he didn't stumble over a Rosetta Stone, he said, "This is what I think I'm gonna do," and everybody's been imitating him ever since. 

​

Chris Harding, in a 2013 article stated:

​

“Genius is not intelligence. Genius is creative ability of the highest possible kind. True, most geniuses are highly intelligent, but this depends on the field their genius was recognized in. And here there is a plethora of problems. Recognized by whom; which people, what society, when and where. There is an old joke that goes something like I will believe in psychologists devising tests from geniuses when monkeys devise tests for psychologists. I do have ideas of my own on this, but so far no one seems interested in this. I was listed in the Guinness Book of World Records seven editions 1982-88 under "Highest IQ" and was given a certificate for this. I was also listed in 500 Great Minds of the Early 21st Century in 2002. All such lists-comparisons are temporary. There appears less and less match between persons and outcomes these days. Humanity hangs by it’s intellectual neck on the tree of tragedy –there are no Leonardo’s in the 19th, 20th, and so far in the 21st Century. Yet he/she must still exist we should think? With mass education has come the noisy ones but no Geniuses to show for it all. Bad money has driven out good money, bad people good people. The masses have come to judge the best and are part of this process to drive out the very people they need most, all in the name of incorrectly accessed political correctness. Today the system has driven down performance; today big institutional science has been a spoiler of great insights delaying progress everywhere. Today it is business as usual. The criminal come to the top. My greatest fear is that an end is coming to the centuries of progress that mankind has grown use to. The age of genius may be at an end. I’m sorry to ramble on this in such a `scatter gun’ way.”
 

 

Marilyn's statement is superficial because it is compatible with the TV show aimed at the mass audience, but her columns in Parade magazine are very high and deep, consistent with the IQ 186-190 that she got on the Mega Test. Chris Harding's statement, although short and on a topic that doesn't offer much depth, also reveals a very high intellectual level. His opinion on the meaning of “genius” is questionable, but for a one-paragraph text it is acceptable. And the key point is that both recognize that scores measured by conventional IQ tests present several problems and cannot be taken too seriously when used to try to assess intelligence at higher IQ levels.

 

This shows that, although scores in the range of 70 to 130 are able to measure intelligence reasonably well, as the scores move away from the mean, what the tests measure gradually ceases to be intelligence and becomes something more shallow, such as reasoning speed for trivial questions or mechanical repetition of tasks. The problem is that as the IQ to be measured increases, the test continues to measure the same variable, but the meaning of intelligence changes. For children aged 8 to 12 with an IQ between 80 and 130 it may be appropriate to measure the ability to spell words without making mistakes as a satisfactory criterion for determining written communication aptitude, but if applying this same method to try to estimate communication aptitude writing by Shakespeare or by Dostoevsky, it is evident that the result will be skewed, it not because these writers are too quick at spelling nor because they are infallible at it. They can even make more mistakes than a well-trained year-old who has “talent” at spelling. The point is that this criterion is no longer useful at the levels of Shakespeare, Goethe or Dostoevsky. In fact, it ceases to be useful at much lower levels, close to 125 or 130. The same problem occurs when trying to use elementary questions like the Stanford-Binet ones to measure intellectual levels above 140.

 

The fact that the Nobel prize-winner average IQ is at the rarity level of one in 3000, while the frequency of Nobel prize-winner in the population is less than one in 1 million, also corroborates that scores above 130 on the Stanford-Binet are highly distorted, dramatically failing to “let go” of the brightest people, while at the same time incorrectly selecting several who are not really bright, but just quick at performing trivial tasks.

 

This is not a defect unique to the Stanford-Binet. All the best IQ tests including WAIS, Raven, Cattell, DAT, D70 etc. have this same problem (and obviously there are more and worse problems in tests that aren't the best). One of the main reasons for this is the same as already mentioned: these tests attempt to measure IQs at levels well above 140, but do not include questions with a difficulty level above 135.

 

To solve this problem, in 1973 Kevin Landgon created the LAIT (Landgon Adult Intelligence Test), the first really difficult intelligence test, capable of measuring correctly until close to 165. In 1982, Ronald Hoeflin published his Mega Test, later the Titan Test, Ultra Test and Power Test. The Hoeflin tests could correctly measure up to about 170 or even 180 IQ.

 

Thus, a new era of intelligence testing has been inaugurated. The traditional tests used in clinics to measure in the range of 70 to 130 continued to exist, covering more than 95% of the population, and it also became possible to measure intelligence at much higher levels. However, these tests have not yet reached the “critical point” that allows us to correctly identify genius minds. The people with the highest scores on the Hoeflin tests are undoubtedly very smart: Rick Rosner, Chris Langan, Marilyn Vos Savant, etc. with scores of 190 or above. But when you compare the intellectual output of these people with that of a Nobel laureate with an IQ of 160, the difference is blatantly favorable to the Nobel Prize. Something was still missing from the variables to be measured at the top of the difficulty level. In the years and decades that followed, other tests were created, including the Eureka, Logima Stricta, and the Sigma Test.

 

Sigma Test, since its first version, tries to be innovative in several aspects. This does not mean that it has achieved this purpose, but at least we are trying, and some of the results obtained have been encouraging. There is controversy over how much difficulty the Sigma Test can actually measure correctly. Some people think the actual ceiling is no more than 180, others think it goes up to 200 or a little more. This is difficult to determine until the number of people evaluated is large enough or until there is some great genius internationally recognized as such (Fields Medal, Abel Prize, Nobel Prize in Physics) who is evaluated by the Sigma Test. But regardless of the difficulty ceiling, Sigma Test also brings other relevant innovations and some of them have already been experimentally corroborated. Among these innovations, the most important is the new standardization method, first introduced in 2000 and first applied in 2003.

 

The new normalization method used in Sigma Test is distinguished from all others by generating scores whose antilogs are on a scale of proportion. Furthermore, this method makes it possible to correctly calculate the corresponding percentages, avoiding the inflated results that are produced by traditional methods. This topic is covered in more detail in other articles.

 

Another differentiator is the variety of cognitive processes required to solve the questions. This is extremely important for measuring intellectual capacity in a wide range of settings. The ability to play chess, for example, measures a very specialized and very narrow latent trait, which cannot be interpreted as representative of general intelligence. Chess skill is positively correlated with intelligence, but as the rating moves away from the average, this score is determined more and more by chess-specific skill and less and less by general intelligence. The same happens if a test uses exclusively series of figures, or if it uses exclusively series of numbers. The measured variable cannot be interpreted as representative of general intelligence. This statement runs counter to some “psychometric mantras” that have been repeated for decades – in particular about homogeneity (the higher the better) and about g saturation – so it requires a little more detailed analysis:

 

Series of figures have the virtue of minimizing the requirement for knowledge, preventing cultural and age factors from interfering with the result and this is a good thing. On the other hand, they limit the ceiling of difficulty and complexity, but the main problem is excessive homogeneity.

 

There are many different ways to measure homogeneity. One of the best and most common is through Cronbach's α.

 

In order to understand how Cronbach's α works, first it is worth explaining how the Kuder-Richardson works: the idea is quite simple, the test is divided into two equivalent halves and the score that each person obtained in each half is verified. . This division can be between odd and even questions, it can be by lottery, or by any other reasonable criterion. If the halves are equal, each person is expected to score approximately the same score on each half. The idea of ​​Cronbach's α is similar, but all covariances between all items are considered, making this measure independent of the criterion adopted to separate the two halves, this is almost equivalent to comparing all possible combinations of two halves.

 

It is positive and desirable that a good test has a high Cronbach's α (above 0.7), because it indicates that the test items are contributing to measure the same variable. This everyone knows and repeats religiously. On the other hand, it is bad if Cronbach's α is excessively high (above 0.9), because it indicates that the test items are not covering a sufficiently wide range of the characteristics that should be measured, that is, the items are excessively redundant and specialized. This fact is apparently neither known nor well understood, so it requires a little more detailed explanation. For this, I will use a didactic example:

 

A test consisting exclusively of 60 numerical series tends to present Cronbach's α greater than a test that includes 20 numerical series, 20 series of figures and 20 analogies. If the difficulty distributions are the same on both tests, then the one with 60 numerical series is likely to have a higher Cronbach's α, in which case having a higher Cronbach's α may be worse. In other words, a Cronbach's α of 0.85 may be better than 0.92.

 

An analogous effect can also produce illusions about g saturation, making a test appear to be more g-saturated than it actually is, simply because it is excessively homogeneous. In a test that is too homogeneous, the first factor extracted may be sufficient to explain more than 80% or 85% of the variability, not because the test is in fact more saturated with the g factor, but because within the limits of what is being evaluated by this test. test, a leading factor common to all items accounts for 80% to 85% of the variance or even more.

 

In this context, pseudo saturation of g is a bad symptom, unless the ultimate goal is to measure the ability to solve series of numbers and figures. But this is usually not what you want to measure. The purpose of a good intelligence test is to gather an appropriate list of questions to assess your ability to solve real problems. The objective is not the score on the test itself, but to ensure that this score is able to reflect the ability to solve different problems in real life. And in this, STE stands out, as it includes several problems with a structure very similar to real problems.

 

The ability to solve series of pictures is also useful, because this same ability also contributes to solving other problems in other situations. However, directly measuring the type of skill you want to know is preferable to measuring a correlated attribute. To clarify this problem, let's analyze two more well-known variables: weight and height.

 

People's weight and height are moderately correlated variables. This means that by knowing a person's weight, one can estimate that person's height. However, if it is possible to directly measure one's height, this is better than measuring weight and trying to estimate height based on weight. If it is not possible to measure height, and the only information available is weight, it is possible to use this information to try to roughly estimate height, but the error can be very large, because there are short people with a lot of fat mass and there are tall very thin people.

 

Therefore, if there is a group of variables more closely related to height, such as femur size, foot size, arm size, then measuring these variables should provide a more accurate estimate of height than trying to estimate height with based on weight. Femur size is not exactly proportional to height, but variation in femur size preserves the proportion to variation in height much better than variation in weight to variation in height. The same applies to foot size and arm size. When you consider femur size, foot size, and arm size together, you can make a much more reliable estimate for height than if you tried to estimate height on the basis of weight.

 

So using a series of figures test to estimate intelligence is like using weight to estimate height, that is, it works, but the errors and distortions are large. Furthermore, as you get closer to the higher levels of weight, the error also increases and the same happens when you want to measure correctly at the highest levels of height, because the higher levels of height rarely correspond to the highest levels of weight. The tallest people in the world are not the same as the heaviest people in the world. Usually the heaviest ones are normal height or just a little above normal.

 

But if the measurement were based on the size of the femur, arm, and foot, estimating height based on each of these variables, then averaging the results, the estimate for height would be much closer to the correct value. Another detail to consider is that in addition to the correlation between femur size and height being much stronger than between weight and height, this correlation is preserved at the highest levels, so that the largest people in the world also have larger femurs, bigger arms and bigger feet. Therefore, the measurement of these body parts remains effective in estimating the correct height of people at all levels, from the average population to the tallest people in the world.

 

Likewise, the use of items with the properties of the Sigma Test questions, closely related to the cognitive processes that represent intelligence, covering a wide variety of cognitive characteristics and skill levels, provides a much more accurate and realistic estimate for intelligence.

 

There are also disadvantages to the Sigma Test, which produces less fair results if it is applied to rural groups or groups with a level of education far below the Middle School grade. But I don't see much need to create tests aimed at this audience, because there are already good tests for that, including Logima Stricta and some of the excellent tests by Iakovos Koukas and YoungHoon Kim. So my focus is on trying to fill a gap that has existed since the early days of IQ tests, which is trying to correctly measure the intellectual level in the higher strata. The Binet tests were able to measure correctly up to about 135, then the Langdon and Hoeflin tests were able to correctly measure up to about 170. The Sigma Test Extended aims to realistically and accurately measure above 190 and perhaps above 200.

 

As already mentioned, Hoeflin tests pioneered the correct measurement of IQ at levels far above the limits of traditional IQ tests, but as there was no proper method for calculating the corresponding percentiles, norms were calculated using the methods available to standardization, resulting in skewed estimates, especially near the ceiling.

 

The “correct” ceiling for Mega Test, based on data that was available on the Miyaguchi website and using the same standardization method as the Sigma Test 2003 standard is about 186, very close to the nominal ceiling of 190+ (~ 193) which was adopted by Hoeflin. The ceiling calculated by Grady Towers was 202. Bob Seitz also made an attempt to establish a new norm that would fit the correct levels of rarity, and he came up with around 170, very close to the rarity norm I found in 2003 for the ceiling of the MegaTest (168.5). This divergence between the results obtained by Towers and by Seitz already signaled a disparity between the true rarities and the rarities obtained based on the hypothesis that the scores were normally distributed. By the late 1990s, the problem was well established: the actual rarity did not agree with the IQ scores measured in the tests. But the solution to this was not yet clear.

 

The nominal IQ score does not present major problems. The Mega Test ceiling presents an error of 7 points, which is tolerable and for lower scores the error is smaller and smaller. However, the corresponding percentile is very different from the correct one. The theoretical level of rarity for IQ=193, assuming the distribution of scores were perfectly adherent to a Gaussian, would be one in 325,000,000, but the correct level of rarity, given the true shape of the distribution of scores, is about one in 435,000. If you consider the correct ceiling to be 186 instead of 193, then the rarity level is one in 130,000. So the true level of rarity differs from the level indicated in the standard by a factor greater than 2000. A huge mistake. The data on Miyaguchi's website is incomplete, so the 186 IQ value indicated as "correct" for the ceiling may be slightly different, perhaps close to 190. However, the percentile distortions are too large to be explained by some bias present in the data available on the Miyaguchi website. This is a serious methodological error that has been systematically repeated for decades.

 

Hoeflin's and Landgon's tests differ in some important points. Langdon's tests, as well as some tests by Cooijmans, Robert Lato and others, followed a similar line to Raven's tests (figure series), while Hoeflin's tests followed a more similar line to that of Binet and Wechsler (diversified).

 

At this point it is worth recapping how the first attempts to measure intelligence were. I won't be redundant with the Historical Summary article on Intelligence Tests; Anyone interested can access it for more details. Here I will give a much briefer summary focused on the topics we are covering.

 

The Binet test represents an important advance in the evolution of cognitive tests. After the attempts by Francis Galton (1884) and James Cattell (1890) to measure intelligence proved unsuccessful, Alfred Binet (1904) tried to approach the problem from a different perspective. Galton and Cattell believed that it would be possible to measure the elementary components of intelligence, while Binet decided to measure the combined result of these components in synergistic action, obtaining much more promising results, making it possible to identify mild deficiencies and some aptitudes. This suggests that the combined use of questions that require different types of thinking working together in solving complex problems may work better than questions that try to measure each type of thinking separately. The STE follows a similar line, betting on the measure of a combined set of skills to solve complex problems, with the differential of including questions that reach levels much higher of difficulty than the Stanford-Binet ceiling (140), reaching and surpassing 190 and even 200.

 

While Binet test is one of the best for correctly measuring IQs between 70 and 130, it fails badly by continuing to produce scores far above what it is actually capable of measuring. The same problem is present in the tests by Wechsler, Cattell Culture Fair and others. The extrapolated Stanford-Binet nominal ceiling reached 225, but the actual ceiling never went above 140. I'm not saying the IQ 140 is low; to say so would be a gross error. What I am saying is that a test with a ceiling of 140 would be like a clinical ruler to measure height with a maximum limit of 1.87 m. The 1.87 m threshold is not low, but it is also not enough to serve a considerable fraction of the population.

 

In fact, the problem with the Stanford-Binet standard is worse than that. It is as if it were a ruler with a nominal ceiling of 2.15 m, but that started to get crooked and with the 1 cm intervals getting smaller and smaller for heights above 1.80 m. On this ruler, the size of 1 cm in the range of 1.50 m to 1.80 m is approximately uniform, but above 1.80 m, each 1 cm interval becomes increasingly narrow. When it gets close to 2.15 m, every 1 cm is so short that it is less than half a 1 cm in the region between 1.70 m and 1.80 m. With a ruler skewed at this level, measurements are only reasonably reliable up to 1.80 m or, with a little optimism, up to 1.85 m.

 

The below image shows an example of a distorted (uneven interval) scale where up to a certain point (the first 10) the intervals are uniform, but then they get increasingly narrower:

​

​

Using a ruler that had each unit spaced this way would obviously produce big errors. This is basically what happens with almost all IQ tests, including the Sigma Test before 2003, because although this method for standardization had already been devised and published in 2000, first Sigma Test standard had to be determined by comparison with other tests already standardized by existing methods and the number of tests in Sigma Test in 2000 was still not enough for adequate standardization using the new method. Therefore, the first standard applying this method was in 2003.

 

Therefore, all traditional IQ tests and all high range IQ tests had this distortion up to 2003 and this distortion causes several problems.

 

If there were such an error in a ruler or a tape measure, where one part of the ruler was correct and another was distorted, it would be easy to correct it by using the “healthy” part to compare side by side with the anomalous part, and then repair the error. But on an IQ scale this is much more difficult and complex to correct. On a crooked ruler with a distorted scale, the problem is noticed visually, but the distortions in the scale of an IQ test are invisible and can only be detected with an adequate statistical treatment. In addition to the detection not being trivial, the correction is even more difficult because it is necessary to establish a reference scale that is invariant. Binet tried to do this using ages and it was an interesting initial idea, but it was quickly found that it didn't produce an interval scale. To produce a ratio scale is even more difficult.

 

The solution adopted in the 2003 Sigma Test standard manages to generate a proportion scale taking advantage of Bill McGaugh's idea of converting Chess rating into IQ. However, FIDE rating, USCF rating and especially online ratings are highly distorted, in addition to the inflationary effect. Therefore, before it was necessary to establish an appropriate rating scale from which a potential ratio scale would be established and then this could be applied to the IQ. It is clear that Bill McGaugh's formula could not be used in its original form either, just because it was calculated based on the FIDE rating, but it was an important inspiration.

​

The rating calculation method is described in this book https://www.saturnov.org/livro/rating and the distribution of scores using this method is this:

​