Niche.com rankings compare schools locally, statewide, and nationally, and may help families to make decisions. But on closer examination for fairness, do these ratings hold up?

Although more known for college rankings, Niche.com recently announced their rankings for best all boys high schools for 2017. The ranking was broken down by numerous different categories. Malvern Prep was ranked 55th best all boys school in the country, with two other area all-boys schools ahead of Malvern: Haverford School and Devon Prep.

Niche, founded by Carnegie Mellon alumni Luke Skurman, provides statistics and reviews on communities, high schools, and colleges. Formerly known as CollegeProwler.com, Niche’s website states the program intends to “transform the way people make big life decisions” with insights from data analysis and “everyday experts.”

According to Head of School Mr. Christian Talbot, whose alma mater Regis High School ranked third, the mission statement for Niche is almost comical.

“Basically it’s self reporting from parents, students, and alumni,” Talbot said. “It’s almost like an opinion poll, and then they push these rankings out on the internet and social media.”

According to Talbot, Niche.com may be driven more by profit than by accurate ratings.

“What people may not know behind the scene is that Niche goes to each individual school who is identified and says we have a profile of your school, and says it’s in your best interest to buy a subscription,” Talbot added.

According to Talbot, an account executive at Niche sent him a solicitation email in August 2016 about monthly insight reports.

“Let us know what you think! We would love to discuss this report with you further and share more about our new Premium Profile offering, where you can enhance what prospective families are seeing when they visit your profile, and drive more traffic to your school as well,” the marketing email stated.

“This is ‘nod-nod-wink-wink,” Talbot said. “If you pay more and get more reports in your rank will go up.”

Niche Marketing Outreach Coordinator Jessica Hair said Niche does not only rely on marketing or self reporting.

“We try to pull together information from reviews and things like SAT scores, diversity, and college admissions to try to paint the best picture we can for each community,” she said.

Hair explained that some information for the rankings is reported by the school.

“All the data is pulled from government sites where schools are mandated to send information every year,” she said. “Private schools like Malvern report to the government on things like composite SAT/ACT scores, diversity and numerous other things.”

According to Talbot, a self-reported system driven by profits may lead to some misleading and confusing information that could affect a decision about where students go to school.

“I feel very strongly that any ranking system that’s trying to capture the meaningfulness or value of a learning institution by looking at such basic and unhelpful data like SAT/ACT scores, self-reported college placements, self-reported diversity, that’s worthless,” Talbot said. “In fact it’s worse than worthless. If it were worthless at least you could throw it in the garbage.”

Talbot believes that the most important aspects of the learning experience at Malvern cannot be measured by the sorts of scores that schools report to Niche.

“I don’t think that any Malvern student, alumnus, or Malvern parent would ever point to those things as having any critical or any important role in what it means to go to Malvern,” he said.

In a 2014 post on Independent School Magazine blog, published by the National Association of Independent Schools (NAIS), former NAIS president shared some observations in a post titled “Bad Data.”

Chubb described Niche as the least competent ranking system he had ever seen.

“The Niche ‘Best Private Schools’ list does nothing to distinguish quality,” Chubb stated. “Whether a school looks good or less good in this system, it would be best to help your community understand why the ranking just doesn’t matter.”

According to Chubb, Niche grades fewer than half of all private schools, and includes slightly fewer than that in its rankings. Because Niche lacks data on some schools, it simply omits them from the rankings. Chubb states that the data the site does collect for most private schools is self-reported, with sample sizes that are too small for accuracy.

Chubb notes that Niche ranks schools by SAT and ACT scores, college placement, culture and diversity, and parent and student surveys. The site requires that seven survey responses are required in order for data from the surveys to be used in grading schools. However, much of this data is self-reported.

“Niche says explicitly that if its survey yielded seven survey responses, the school’s student outcomes were estimated from that handful of responses,” Chubb wrote. “One does not have to be a professional pollster to know that the confidence intervals around such estimates are huge — actually, plus or minus 35 percent. The ranking system ignores confidence intervals. Schools that, in fact, differ by indistinguishable amounts statistically are lined up as if the differences matter.”

Chubb stated that Niche’s rankings do nothing to distinguish quality among private schools.

“NAIS is committed to providing our school community with better data. I believe that strong data systems can help all of us make far better decisions, as school leaders or parents. Niche is just not one of those systems,” he wrote.

Director of College Counseling Mr. Ian Harkness pointed out that the Niche rankings could sway families’ perceptions of private schools.

Devon Prep shares the school’s Niche rankings on their website.

“The rankings really can sway someone’s mind,” Harkness said. “If you’re looking at three all-boys schools in the area and based on rankings you choose the top school that doesn’t necessarily mean it’s the right fit.”

“It’s not about the prestige of the college you get into or the bumper sticker on the back of the car. It’s about finding a great place where you fit in,” Harkness said. “And the rankings can sometimes make that a much harder process.”

Upper School Head Mr. Ron Algeo thinks that Niche does not have a strong enough reputation to give these rankings much credence. “I don’t think they have been around long enough to say, ‘Wow that’s really honorable and hang your hat on that,’” he said. “They don’t have enough recognition to do that.”

According to Algeo, a school like Malvern must be careful about its goals. “If our goals were to move up the rankings we could possibly do that, but it may not really serve our community or our students.”

“We want to do what’s best for our student and our families. That’s why we don’t put a lot of stock into those rankings,” he said. “They don’t provide us feedback, which is why we put a lot more stock into our relationships with our students and families.”

Senior Zac Fernandez thinks that the rankings provide no clear reason why Malvern should be ranked lower than Haverford or Devon. “If there was a clear formula to rank these schools, I would accept them more, but they have no real meaning honestly,” he said.

Fernandez noted that a ranking system based on self-reporting does not seem to provide much of a basis for accuracy. “Every school is going to think they are better than the other because of pride,” he said. “So a ranking system of self reporting does not help at all, because you can easily get a bunch of people to put in great reports for the school to boost the ranking.”

According to Talbot, a ranking system that is reaching out to heads of schools for subscriptions or payment behind the scenes has no credibility.

“A ranking should be objective,” he said. “There should be no capacity for their rankings to be influenced by people who are in the rankings paying you money.”

“That’s the definition of a conflict of interest.”