– □ x
OUR MISSION OF
RADICAL TRANSPARENCY
How We Test The Testers
– □ x
THE PUBLICATION
TRUST SCORE
A Publication Trust Score assesses the trust of an expert site, and it serves as the bedrock for the weighting of the Expert Scores and Score Confidence. Expert sites can be complex and feature diverse ranges and types of content apart from each other, so to be as thorough and accurate in scoring as possible, we have to first calculate a publication’s Category Scores.
A Category Score is a weighted score composed of another two parts: a General Trust Score and Category-Specific Trust Score. A dedicated researcher calculates these scores by evaluating a wide variety of aspects on a site to determine how trustworthy the publication is on a whole and for a specific product category. Once the evaluation process is complete, we reach out to the publication, inform them of their Trust Scores, and ask if we missed anything.
We explain the Publication Trust Score – Phase 1 criteria in the following tables:
GENERAL TRUST CRITERIA – PHASE 1
This criteria assesses foundational content on each publication, such as an About Us page, sponsorship/paid promotion disclosure, and a scoring system used on every review regardless of product category.
ANALYSIS CATEGORY
WHO THEY ARE
About Us or Team Page
Does the expert site have an About Us or Team Page, and do they include names and photos of team members?
0.5
Author Social Media
Does the site provide links to their authors’ social media pages?
0.1
Business Address & Phone Number/Email
Does a business address and phone number/email exist on the site?
0.3
Ethics Statement
Do they have a statement on their honesty, transparency, and/or authentic reviews?
0.5
Privacy Policy
Do they have a statement on their honesty, transparency, and/or authentic reviews?
1.0
Overall “How We Test” Page
Does a page where the site goes over how they test and review all their products exist?
0.3
Publication Social Media
Does the site have its own LinkedIn Page?
0.5
CATEGORY-SPECIFIC TRUST CRITERIA – PHASE 1
This criteria assesses content on specific categories on each publication, such as how experienced an author is in the specific category, what type of media they present in their content, and how in-depth the product testing is.
ANALYSIS CATEGORY
WHO THEY ARE
Product Reviews and Buying Guides
Does the site feature at least three product reviews for the category?
Does the site feature buying guides for the category?
Does the publication claim to test the category being reviewed?
2.5
Review Author
Has the author on the most recent category-specific product review written at least 10 reviews for the category on the site?
Does the review author have a public LinkedIn profile?
Has the author been writing for at least one year?
2
Supporting Assets (photos and tests)
Do the product reviews contain real-world, non-stock photos?
Do the buying guides contain real-world, non-stock photos?
Do at least 3 product reviews or buying guides include images that show any of the following:
Hardware testing equipment; software testing screenshots; test results chart; multiple similar products together; or the journalist’s face with the product?”
1.5
Testing
Does a category-specific test method exist, i.e. How We Test Headphones,
Is the category-specific test method have a published date?
Is the above a year or older..or longer?
Does the writer/publication provide any names and types of testing equipment, software, or tools used?
Do they provide correct units of measurement to help support that they actually tested?
“Did the reviewer demonstrate that the product was tested in a realistic usage scenario?
(e.g. assessing a pair of headphones’ sound quality with different genres of music, riding an e-bike on various inclines, blending ice in a blender, etc.)”
5
Custom Questions for the Category
Criteria of this aare determined per category relating to relevant aspects that should be tested to demonstrate technical understanding of the products; these are often tests of performance criteria by way of measurement or use case, such a testing for brightness, how well something crushes ice, or how much debris a vacuum picks up)
2
Based on the custom questions and your own assessment, is their claim to test truthful?
Does a page where the site goes over how they test and review all their products exist?
-10% if “yes”
HOW WE CALCULATE A CATEGORY SCORE:
Once the above criteria is all evaluated, we then calculate the Category Score:
(General Trust Score x 0.3) + (Category-Specific Trust Score x 0.7) = Category Score
We believe that the category-specific review content is the core of all these publications, which is why we weigh that score to be worth 90% of the Category Score. General Trust Makes up the other 10%, and the two values are added together to become the Category Score. The average of all the Category Scores for a single publication is calculated, and that number becomes the Publication Trust Score.
Calculating a Publication Trust Score Example:
- Publication #1
- → General Trust Score: 86%
- → Category-Specific Trust Scores:
- → Best Laptops: 65%
- → Best Headphones: 71%
- → Best TVs: 93%
- Category Score:
- → Best Laptops: (86 x 0.1) + (65 x 0.9) = 67%
- → Best Headphones: (86 x 0.1) + (71 x 0.9) = 73%
- → Best TVs: (86 x 0.1) + (93 x 0.9) = 92%
- Publication Trust Score:
- → (67 + 73 + 92) / 3 = 77%
The Publication Trust Score determines the weight of the publication’s review score when calculating the Expert Score for a specific product. So in essence, the higher the Publication Trust Score, the more weight that publication’s review has in our overall TrueScore for that product.
Calculating An Expert Score Example:
In this scenario, five publications have their own reviews and scores on a specific OLED TV model. We’ve decided to use them as Expert Sources to calculate our Expert Score for that model.
| Publication # | OLED TV Model Expert Score |
|---|---|
| 1 | 93.4^ |
| 2 | 4.5/5 |
| 3 | 8.8/10 |
| 4 | 76% |
| 5 | 3/5 |
We then have to convert their review scores to our own
scoring system that uses a 1-100 logarithmic scale:
- Publication #1:
- 93.4% → down to 93%
- Publication #2:
- 4.5/5.0 = 0.9 → becomes 90%
- Publication #3:
- 8.8/10 = 0.88 → becomes 88%
- Publication #4:
- 76% already aligns with our scoring scale and remains 76%
- Publication #5:
- 3/5 = 0.6 → becomes 60%
Through our Trust Score research, we’ve found that the
Publication Trust Scores (PTS) for Publications #1-5 are:
| Publication # | OLED TV Model Expert Score | Converted Expert Score | Publication Trust Score (PTS) |
|---|---|---|---|
| 1 | 93.4^ | 93% | 80% |
| 2 | 4.5/5 | 90% | 83% |
| 3 | 8.8/10 | 88% | 95% |
| 4 | 76% | 76% | 91% |
| 5 | 3/5 | 60% | 88% |
Now we have everything we need to calculate the
weighted Expert Score of the OLED TV model using this formula:
- (Publication #1 Converted Expert Score x 0.__)
- + (Pub. #2 Conv. Expert Score x 0.__)
- + (Pub. #3 Conv. Expert Score x 0.__)
- + (Pub. #4 Conv. Expert Score x 0.__)
- + (Pub. #5 Conv. Expert Score x 0.__)
- = Weighted Expert Score (%)
The publications with the higher PTS get the higher weights in the score.
(93 x 0.__) + (90 x 0.__) + (88 x 0.__) + (76 x 0.__) + (60 x 0.__) = __%
The weighted Expert Score is then used alongside
the Customer Score to calculate the OLED TV model’s TrueScore.
Calculating score confidence Example:
We can also determine Score Confidence with the PTS values. Score Confidence signifies how confident we are in a product’s TrueScore. For example, consider the OLED TV model scenario above, take the Score Confidence formula below and plug the five publications’ PTS numbers into it:
- [(TrustScore1 + TrustScore2 + . . . + TrustScore) / (n x 100) ] = Percent Average Trust Score
- CONMOD (Confidence Modifier) = (1 – n) [where n ≤ 5, n = (5 – Total Sources) x 0.05]
- [(PATS) x (CONMOD)] x 100 = Score Confidence
- Example
- (87 + 94 + 74 + 66) / (4) = 80.25 (PATS) and 4 Sources Used
- [(PATS) x (CONMOD)] x 100 = Score Confidence
- → [(0.8025) x (1 – [(5- 4) x 0.05] )] x 100
- → [(0.8025 x (1 – 0.05))] x 100
- → [(0.8025 x 0.95)] x 100
- If this product had enough sources, it would have scored 80%.
- → 0.762375 x 100
- →76% Score Confidence
Therefore, we have 81% confidence and trust
aka “No Doubt” in the OLED TV model’s TrueScore.
PUBLICATION TRUST SCORE RESEARCH – PHASE 2
The Trust Scores are a work in progress, and since completing Phase 1 of Publication Trust Score Research, we’re now working within Phase 2, which will add onto the current Trust Scores and assess new criteria:
- How many products have they tested?
- How many Performance Criteria do they evaluate in reviews?
- Have they reviewed five of the top brands of a specific category?
- Do they review any newcomer brands’ products in a category?
- Do they cover how a product has evolved from previous models?
Future phases are to come so that we can further refine the Publication Trust Scores.