Last week Anandtech published an article that revealed recent Huawei devices alter their behavior to boost the performance of two popular benchmark applications – 3DMark and GFXBench.

After further investigation, Primate Labs has discovered that Huawei devices also alter their behavior when running Geekbench. We did this using our “private” version of Geekbench, which runs the same benchmark tests as the “public” version of Geekbench but has its identifying information (such as the package identifier) changed.

If both the “public” and the “private” versions of Geekbench return the same scores, then it is unlikely the device is boosting benchmark scores by altering its behavior. Conversely, if the “private” version returns significantly lower scores than the “public” version, then the device is likely boosting benchmark scores.

In Primate Labs’ preliminary testing, performance was identical between the “public” and “private” versions on several Android phones, including the Google Pixel 2, the LG G7, and the Samsung Galaxy S9. However, the Huawei P20 Pro produced scores, on average, 7.8% lower in the “private” version, and the Huawei Mate 10 Pro produced scores, on average, 10.2% lower.

Based on these results Primate Labs strongly suspects these two phones are altering their behavior to boost Geekbench scores. Furthermore, based on results from the Geekbench Browser, Primate Labs also suspects similar models (the Huawei P20 and the Huawei Mate 10) are altering their behavior.

Primate Labs expects devices to treat benchmarks no differently than any other application. Ryan Shrout of Shrout Research, when discussing Huawei’s behavior, explains why this is important:

Some have asked me why this issue matters; if the hardware is clearly capable of performance like this, why should Huawei and HiSilicon not be able to present it that way? The higher performance results that 3DMark, GFXBench, and now Geekbench show are not indicative of the performance consumers get with their devices on real applications. The entire goal of benchmarks and reviews is to try to convey the experience a buyer would get for a smartphone, or anything else for that matter.

If Huawei wanted one of its devices to offer this level of performance in games and other applications, it could do so, but at the expense of other traits. Skin temperature, battery life, and device lifespan could all be impacted – something that would definitely affect the reviews and reception of a smartphone. Hence, the practice of cheating in an attempt to have the best of both.

Primate Labs is taking several steps to prevent our users from being misled by “boosted” Geekbench results, and to discourage device manufacturers from adding this behavior to future devices.

Primate Labs will exclude the following Huawei phones from the Android Benchmark Chart and the Mobile Benchmark Chart, and will add an alert to individual results for the following Huawei phones that the phones attempt to manipulate benchmark results:

  • Huawei P20
  • Huawei P20 Pro
  • Huawei Mate 10
  • Huawei Mate 10 Pro

Primate Labs also will conduct a review of handsets from other manufacturers to determine if they are also manipulating benchmark results. An initial review which included several handsets from Google, HTC, Samsung, LG, and OnePlus is already complete, and no other handsets were discovered to be boosting.

Finally, Primate Labs will make the “private” build of Geekbench available to trusted journalists to discover and hopefully discourage this behavior by device manufacturers.

Primate Labs is committed to providing objective and realistic benchmark tests, and will continue to work to ensure that Geekbench meets these goals now and in the future. Primate Labs hopes Huawei acts quickly to remove the boost from their devices, and that they refrain from including the boost in future devices.