Great to Good The Diluted Value of High Test Scores

When admission letters came in last spring, there was more head-scratching than usual among counselors and families. Much of the discussion revolved around how unpredictable things seemed, especially when it came to test scores. With the release of class of 2018 results from both ACT and College Board, we can now say definitively that students saw the most competitive scores ever. In the last 10 years, the number of students scoring 14001600 on the SAT or 3136 on the ACT doubled. In just the last 5 years, the number of students scoring 15001600 or 3436 has doubled. Compass has examined, in-depth, how this trend has been shaped, why it accelerated in 2018, and what impact it has on admissions. A big shift in the SAT The SAT regained momentum last year, and once more became the most popular—if that’s the right word—admission test. The story that has been overlooked is the surge in top scores. In the class of 2017, College Board reported 85,000 students scored between 1400 and 1600. For the class of 2018, 145,000 fell in this range—a 71% jump in a single year! While the size of last year’s change was unusual, its direction was not. The increase fits a historical trend for both the SAT and ACT that shows no signs of abating. Students, parents, counselors, and even admission offices, may need to reconsider their confidence about what a â€Å"great† score is. A high score is no less of an achievement from an academic standpoint, but it is an achievement shared by a wider set of students. It has become more difficult to distinguish oneself with high scores. A note about â€Å"high scorers† or â€Å"top scores. College Board stopped reporting the number of students at each point along the scale. Instead, we must make do with ranges—1400 to 1600 for total score and 700 to 800 for Math and Evidenced-based Reading Writing (ERW). ACT does still produce figures for each composite and section score. In some cases, though, it is easiest to use the 31 to 36 range of composite scores because it best aligns with the 1400 to 1600 range on the SAT. A real world example UC Berkeley is one of the most competitive schools in the country and provides context because it—along with the other University of California campuses—publishes applicant and admission data by score range. The state and national trends we are seeing are reflected in Berkeley’s statistics. The number of applicants with SAT ERW and SAT Math scores at 700800 approximately doubled last year. While it is true that some of that change is a reflection of overall SAT and applicant growth, the percentage of applicants in the top score range also increased. As shown below, the percentage of students with Math scores at or above 700 went from 41% of applicants in 2017 to 56% of applicants in 2018. Nor was the change limited to applicants. Admitted students fell in the top range 54% of the time in the fall 2017 cycle, but 70% of the time in the 2018 cycle. ACT scores at Berkeley have also seen a steady uptick. Below is a graph over the last five years of the percentage of ACT applicants who have scores in the highest band of composite scores, 3136. The whole hides the parts News stories about SAT and ACT data tend to focus on the overall figures—total test takers or average scores. This approach masks shifts in test results and test taker behavior—particularly at the â€Å"tails† of the score distributions. For example, if the number of low-scoring and high-scoring students increase by equal numbers, the mean score may not move at all. Between 2008 and 2018, the average ACT composite score went from 21.1 to 20.8, an insignificant change. Yet nine times as many students achieved a 36 in 2018. Two very different trends help explain this divergence. In recent years, College Board and ACT have focused on selling statewide testing.   By having every student participate, states have brought college awareness and preparedness to wide swaths of students. As a group, students who test only because they are required to test score lower than students who opt-in. High scoring test takers are more likely to be free agents, deciding on the best test based on individual preferences. These students have become savvy consumers, and don’t feel the need to follow labels such as â€Å"ACT state† versus â€Å"SAT state.† They are increasingly likely to consider both tests and, in many cases, take both tests. Informed test choice often allows them to start at a higher point versus a coin flip decision and may allow them to make progress more quickly. Where were the high scorers? State-mandated testing explains why Illinois and Colorado come out on top in SAT taker growth. An additional 186,000 students took the SAT in those states in the class of 2018, because both states switched from funding the ACT for all students to funding the SAT for all students. The two states had a more muted impact on high scorers, accounting for just one-tenth of the increase in top scores. Free agent states such as California, New York, New Jersey, and Massachusetts made up much of the rest. International students also played a large role in the shake-up of scores at the high end. An additional 9,077 international students scored in the 14001600 range in the class of 2018, despite the fact that there were only an additional 28,568 SAT test takers. The students who chose to take the SAT expected to do well, and they did. Below is the data for all 50 states, District of Columbia, Puerto Rico, U.S. Virgin Islands, and international. State1400-1600 Scores (2017)1400-1600 Scores (2018)1400-1600 Scores (Chg)1400-1600 (% Chg)Testers (2017)Testers (2018)Testers (Chg)Testers (% Chg) California9,33120,79911,468123%226,699262,22835,52916% International13,18922,2669,07769%112,901141,46928,56825% New York4,98210,5985,616113%135,141162,55127,41020% Illinois1,8886,5594,671247%12,402145,919133,5171077% New Jersey2,8437,4634,620163%72,17384,67212,49917% Texas6,74610,3773,63154%204,409226,37421,96511% Pennsylvania3,4845,6232,13961%89,21896,7407,52208% Massachusetts3,9536,0752,12254%56,02459,3823,35806% Florida4,0715,8471,77644%147,058176,74629,68820% Virginia3,4505,1791,72950%57,45361,5764,12307% Maryland2,2473,8431,59671%42,88948,0405,15112% Colorado9612,4671,506157%5,89658,79052,894897% Georgia2,1323,5481,41666%63,80574,24010,43516% North Carolina2,0553,3911,33665%49,59554,9875,39211% Washington2,0483,1911,14356%43,91148,5744,66311% Michigan3,2734,2761,00331%110,082115,2815,19905% Connecticut2,1403,09795745%43,25244,7001,44803% Ohio1,8562,46160533%14,54522,9928,44758% Indiana1,5462,06652034%45,62248,9623,34007% Arizona1,3131,78947636%20,46620,188-278-01% Oregon8821,22934739%15,86617,4761,61010% South Carolina7511,08133044%22,29225,3903,09814% New Hampshire69289920730%14,75814,8347601% Hawaii33648314744%7,3527,88052807% Minnesota70083313319%2,0612,46440320% Tennessee76789713017%3,4954,18168620% Delaware30942611738%10,06011,05599510% District of Columbia28340011741%4,8014,98518404% Rhode Island31542611135%7,20510,1612,95641% Louisiana3384349628%1,6962,02733120% Maine4085019323%13,82614,31048404% Missouri6026959315%1,9902,42043022% New Mexico2092958641%2,3423,22588338% Alabama3634417821%2,3932,87848520% Nevada3754527721%6,2455,588-657-11% Alaska1131826961%2,9713,33436312% Kentucky4124756315%1,6081,92531720% Kansas3514146318%1,1991,41922018% Vermont2272896227%4,3034,3232000% Wisconsin6026615910%1,7801,92314308% Idaho4154655012%18,75720,4841,72709% West Virginia1171594236%2,4065,0582,652110% Puerto Rico43823991%2,9133,78387030% Arkansas2002383819%1,0651,59152649% Utah2783143613%1,2771,42514812% South Dakota45742964%2372602310% Montana1101342422%990952-38-04% Iowa2773002308%86199413315% Wyoming24452188%1441692517% Oklahoma3283461805%2,7763,33756120% Mississippi1801961609%7168069013% North Dakota38481026%1231482520% Virgin Islands97-2-22%752635-117-16% Nebraska199187-12-06%680688801% The Curious Case of California Last year California became, for the first time ever, the largest market for both the SAT and ACT. The states sheer size is not the only reason it provides a useful case study. California has never had state-mandated or state-funded college admission testing, and its test taker population is highly diverse. The case of California shows how test taker behavior played a role in the rise of high scores. In Californias class of 2017, there was actually a decline in the number of students in the top tier of scores. These students defected to the ACT. With the class of 2018, those students returned. And they brought their friends! The number of students scoring above 1400 on the SAT more than doubled in a single year. This flow out and back in is why Compass does not believe that the increase in top scores can be attributed solely to score inflation, an â€Å"easier† test, or more effective preparation. Many top scorers in the class of 2017 chose to avoid the uncertainty of the new SAT when they drew up their testing plans. For some, this meant squeezing in an old SAT before it disappeared. For others, it meant switching to a known commodity, the ACT. And for still others, it meant trying multiple test types. Don’t percentiles give an accurate accounting? Percentiles can be a useful way of describing test score distribution. They are, though, a lagging indicator. The figures shown on the class of 2018’s SAT score reports were based on the performance of students in the class of 2017. Percentiles were misleadingly optimistic—a high scoring student’s actual rank within the class of 2018 would have been lower than projected. Even upcoming SAT reports will show percentiles that understate the number of high scores. College Board will be reporting percentiles based on both the classes of 2017 and 2018 (it is phasing in the use of a 3-year running average). While basing percentiles off of two or three class years can smooth out minor ups and downs, the policy makes less sense when the trajectory is always in one direction. We won’t have full data on the class of 2019’s performance for another 11 months, but it will certainly be higher than what the percentiles indicate. Percentiles also do not reflect the impact of highest score or superscoring policies. Compass frequently encounters the misperception that standardized scores are always normally distributed about a mean. In fact, the scale is only meant to ensure that a 700 or a 33 means the same thing in one year as it does in the next. The distribution changes every year.   It is not designed so that the same number of students earn a certain score each year. That’s why any rule-of-thumb take on a good or great score can be outdated as soon as it is given. If scores become too clumped at the high end, a concern is that they will be less useful in distinguishing students or in predicting performance in college. From a student’s viewpoint, any mistake becomes more costly as score groups become more populated. [For optimistic students, any gain becomes more valuable!] Imagine a student in the class of 2017 who expected to score a 750 Math, but instead scored a 700. The shortfall would have put an additional 70,000 students ahead of her. Transplant that student to the class of 2018, and she would have seen closer to 120,000 students pass her—that is even before considering highest score or superscore policies. As the controversy over the June SAT demonstrated, even 2 or 3 missed problems can easily drop a student 50 points. It is understandable why students sometimes feel that they are on a knife’s edge. An easier experience Compass does not believe that the defection/re-adoption theory explains the entirety of the change in high scores on the SAT. The class of 2018 faced fewer obstacles than had the class of 2017. The new SAT is, loosely speaking, the only SAT the class of 2018 ever knew. They took the new PSAT as sophomores, repeated it as juniors, and all of their SAT scores are on the new exam. There was less noise around SAT versus ACT and more signal. Students could make optimal decisions without feeling like guinea pigs in a College Board experiment. They could test early or late without worrying about the change from old SAT to new. As more information became available for the SAT, they were better prepared than the class of 2017. In short, the class of 2018 didn’t necessarily face an easier test; they faced an easier experience. An accounting change The old SAT introduced an odd accounting problem for College Board. Its 2017 report included only students who took the new SAT. This excluded students who finished up their testing with the old SAT. Because that group skews toward high-performing students, the 14001600 total of 85,000 students likely undercounted top scores in the class of 2017 by 5,00010,000. This helps explain why the 2017 figure was so low. The source of the high score surge is not entirely academic. The concordance of SAT and ACT scores was based on the performance of class of 2017 students who took both exams. If the class of 2018 differed significantly from the class of 2017, then the concordance is less accurate—at least at the high end of the scale. Wouldnt an increase in SAT numbers be offset by a decrease in ACT numbers? An additional buyer of a Ford pick-up means one less buyer of a Chevy or Ram. The same cannot be said about the SAT. Many high-scoring students have â€Å"discovered† the ACT and added it to their portfolios rather than switching to it exclusively. The increase in top scores on the ACT slowed with the class of 2018, but it did not reverse, despite an overall drop of 176,000 test takers. The role of dual testing Taking both the SAT and ACT has played a role in the growth of high scores. It has added to the total number of tests taken by high scorers and allowed them to rack up more top scores. However, a student has only one best score (even a concordant tie means a single score can summarize things). This leads to some double-counting of high scores, something not unique to the class of 2018. For example, a student with a 33 ACT receives no benefit from having scored a 1420 on the SAT, even though the latter score would count in College Board’s figures. Neither College Board nor ACT report on cross testing. In fact, they don’t even have direct access to this data, as both must adhere to privacy restrictions. Based on data from competitive colleges, we estimate that approximately one-third of high scoring students take both exams at least once. This trend does not mean that students are at a disadvantage if they only focus on one test. It is difficult to be fully and equally prepared for two admission tests. What is important is that students increasingly consider both tests and try to find the optimal path. What about the most exceptional scores? Most of the discussion above has been about students in the 14001600 range because College Board has stopped reporting the number of students at each score point. Historical SAT trends and current ACT trends indicate that change at the very highest scores is even more extreme. While the number of ACT scorers at 31 or higher was only up slightly in the class of 2018, for example, the number of perfect scorers spiked by 36%. The top percentile of ACT scores has moved up a full point in the last 5 years. For each of the last 15 years, growth in the most elite scores has outpaced the already significant growth in great scores. Supersized scores Despite the gaudy growth numbers this year, the data actually understate the number of top scorers in one way. In their reported figures, College Board and ACT use a student’s most recent test performance. A student scoring a 1450 in June and then a 1420 in October would have been counted as a 1420. Since most colleges consider the best performance or participate in Score Choice, this student would have shown up as (at least) a 1450 scorer in admission offices. A large number of competitive colleges now superscore SAT results. A superscore combines a student’s best Evidence-based Reading performance with her best Math performance, even if the scores come from separate sittings.Research from College Board indicates that â€Å"most recent† accounting is, for students who take the SAT more than once, approximately 20 points lower than â€Å"highest single administration† scoring and 30 points lower than superscoring. The more students repeat the exams, the hi gher this figure grows. High-scoring students tend to repeat the SAT and ACT more than low-scoring students, so the impact of repeat testing is magnified at the upper levels. About two-thirds of re-testers have superscores that are higher than their best single administration scores. Compass estimates that highest combined and superscoring policies increase the number of students in the top score ranges by 1520%. While College Board may report that 145,000 students scored 14001600 in the class of 2018, college applications likely reflected at least 170,000 students in that range. ACT reports 135,000 students in the 3136 range. A real world estimate would be closer to 160,000. Isn’t this all just score inflation related to the new SAT? College Board completely overhauled the SAT in March 2016. By comparing how a pilot group of students performed when taking both old and new SATs, a concordance was developed between the two exams. Overall, the concordance showed—and further studies confirmed—that scores were higher on the new SAT. However, the changes seen in 2018 are separate from that score restructuring. Using the structure and scoring of the new SAT to explain this year’s developments has several flaws: The class of 2017 report is based on new SAT data. Score inflation should have already been accounted for. Score inflation decreases at the high end of the range. While a student scoring 9001100 on the old test would be likely to score 80 to 100 points higher on the new exam, the same cannot be said for students at 14001600. In fact, there is little difference at all above about 1450. The old SAT and the ACT have had an increasing number of high scoring students for many years. The trend is real; 2018 is an outlier because of the scale of the change, but it fits well with the overall pattern. What will happen for the class of 2019? The trend of more students reaching the top tiers is unlikely to reverse. Early indications, though, are that the giant leaps seen last year will not be repeated. PSAT scores can serve as a leading indicator of score changes. While the class of 2018 saw large changes at the top end of PSAT scores, the class of 2019’s PSAT results showed modest increases. Early SAT dates reflect small average improvements for this year’s class. As noted previously, though, average scores can mask changes in high and low scores. None of the available figures indicate that the number of top scores will decline. High score growth on the ACT also continues. While it slowed this year, that was partly the result of an overall decline due to lost state contracts. We expect the class of 2019 to show increases, particularly for elite scores. At the most competitive colleges, high test scores can be viewed as â€Å"necessary but not sufficient.† It is extremely difficult to gain admission to Stanford with a low SAT score, but getting a great score is far from a guarantee of admission. The net effect of the growth at the top ranges is to make a high score more essential but less sufficient. After the dust settles each April, we often hear that â€Å"this was the worst year ever.† For 2018, that assessment feels fair. ACT and SAT scores at colleges have trended up over time, but it’s not simply higher scores that create anxiety—it’s also the added unpredictability. The combination of increased applicant numbers at competitive colleges and a higher percentage of top scores magnify the uncertainty that students experience. Fortunately, hundreds of thousands of students who experienced this anxiety last April are off and thriving at college and looking forward to homecoming. And 2019 will be the worst year ever, until it isn’t.