Even though they keep getting caught when they cheat on benchmarks, Android vendors keep doing it. Prominent handset makers that created some of the best iPhone rivals around cheated in benchmark tests in previous years, including Samsung and OnePlus, and they were all caught. Huawei now joins the sad list after several of its handsets, including the P20 Pro, were discovered behaving differently when a benchmark app was detected.
It all started with AnandTech, which first discovered the behavior and then talked to Huawei at IFA 2018 about their findings. Huawei acknowledged the discovery, stating that they’re doing it to compete against rivals from China that are also cheating in similar tests.
Huawei then provided a comment to Android Authority in which it explained its behavior:
Huawei always prioritizes the user experience rather than pursuing high benchmark scores – especially since there isn’t a direct connection between smartphone benchmarks and user experiences. Huawei smartphones use advanced technologies such as AI to optimize the performance of hardware, including the CPU, GPU, and NPU.
When someone launches a photography app or plays a graphically-intensive game, Huawei’s intelligent software creates a smooth and stable user experience by applying the full capabilities of the hardware, while simultaneously managing the device’s temperature and power efficiency. For applications that aren’t as power intensive like browsing the web, it will only allocate the resources necessary to deliver the performance that’s needed.
In normal benchmarking scenarios, once Huawei’s software recognizes a benchmarking application, it intelligently adapts to “Performance Mode” and delivers optimum performance. Huawei is planning to provide users with access to “Performance Mode” so they can use the maximum power of their device when they need to.
Huawei – as the industry leader – is willing to work with partners to find the best benchmarking standards that can accurately evaluate the user experience.
UL, the creators of the famous 3DMark test, performed their own research and decided to delist four Huawei phones that have been caught cheating in 3DMark, including the P20 Pro and P20, but also the Nova 3 and Honor Play.
The company tested all phones using the public version of 3DMark that’s available on Google Play, as well as an internal one that Huawei didn’t have access to it:
We found that the scores from the public 3DMark app were up to 47% higher than the scores from the private app, even though the tests are identical.
This is proof that Huawei’s software is actively bumping up performance when detecting benchmark apps, but also that the AI features Huawei promotes don’t always work, because they failed to increase performance in a similar way when the phones were tested using the internal builds.
It’s bizarre to see Huawei engage in benchmark testing and hope it won’t get caught. The findings are all the more annoying because Huawei just unveiled its next-gen processor, the 7nm Kirin that will compete against Apple’s A12 and Qualcomm’s Snapdragon 855. And the company used benchmark scores in its presentation to highlight the Kirin 980’s advantages over the current Snapdragon 845. When the Mate 20 Pro comes out next month, it’s likely the phone will be put through thorough benchmarks with testers looking for any signs of cheating.
For the second time in a matter of months, Huawei has had to defend itself against shady practices. A few weeks ago, Huawei was just caught using a DSLR camera for a commercial that implied the selfies in the ad were taken with a new Huawei handset.
Even though benchmark tests aren’t entirely reflective of the real-life performance of a phone, misleading customers isn’t the way to go, especially for a company looking to become the number one smartphone vendor in the coming years.
UPDATE: Huawei provided the following statement on behalf of Huawei and UL:
Huawei and UL (creators of 3DMark) have held comprehensive discussions on benchmarking practices this week, and have reached a positive agreement on the next steps in working together.
In the discussion, Huawei explained that its smartphones use an artificial intelligent resource scheduling mechanism. Because different scenarios have different resource needs, the latest Huawei handsets leverage innovative technologies such as artificial intelligence to optimize resource allocation in a way so that the hardware can demonstrate its capabilities to the fullest extent, while fulfilling user demands across all scenarios.
UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.
Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device.
Huawei and UL have also discussed current common benchmark testing methodologies in general. UL and Huawei would like to participate in an industry movement to develop benchmarking standards that best serve the needs of manufacturers, press, and consumers.
To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.
