After an attempt (failed one at best) to discredit tech reviewers at one point, including a brief Twitter battle between HardOCP and Roy Taylor (hence deactivating the account following the backlash), AMD has aimed its crosshair over BAPCo SYSmark benchmark, calling it bias. Same stuff, different player!
Round 2!! *ding ding*
On a serious note…
AMD posted a video presentation on Youtube to talk about SYSMark 2014 benchmark and its alleged unreliability, claiming that its bias towards Intel. The video starts with John Hampton, the director of Computer Client Products at AMD. In the video, he explains that SYSMark is one of the most commonly benchmark used in the world, but also follows up by saying that one should know how to to choose the right benchmark. It should be noted that at the start of the video, John says that they will be using facts and data throughout this video.
The presentation shows a comparison between BAPCo’s SYSmark 2014 with Futuremark’s PCMark 8, which is something that I use in many test along with Futuremark 3DMark benchmarks for graphic cards (Unfortunately, AMD India doesn’t send samples to Hardware BBQ and may have some exotic issue to reply emails in the near past).
Tony Salinas, AMD’s engineering manager explains with an un-named SKU Intel i5 notebook. With BAPCo SYSMark 2014 benchmark, it generates 987 as a score. It is then compared with an AMD FX powered notebook generating a score of 659, a difference of 50% (well- 49.77% to be exact). It is, then, switch to PCMark 8 benchmark which generated 4199 score on the Intel notebook and 3908 on the AMD equivalent, a difference of 7%.
Though the SKUs were not mentioned in the video, upon a closer look AMD’s equivalent uses AMD FX 8800p processor and Intel used i5 5200U processor. Both notebooks used the same SanDisk SSD, but Intel’s system detected the memory as 8GB while AMD’s equivalent detected as 0MB(?). Let’s consider both are practically the same, except the CPU.
AMD FX-8800p is a quad-core Carrizo APU with Radeon R7 graphics. Intel i5 5200u is a dual-core based Broadwell processor using Intel HD 5500 on-chip graphics.
It should be noted that AMD said BAPCo SYSMark 2014 runs CPU-only benchmark while Futuremark’s PCMark tests all the components in a system, including secondary storage if selected. According to AMD, SYSmark 2014 doesn’t reflect the general usage that can be related with every day-to-day life. Tony did point that out in the video as well.
Later, a custom script used with a benchmark. Unlike the previous tests that ran a battery of tests with different workloads and generating an evaluating a score, this was a time-based benchmark. The ‘WORD’ test indicated that Intel’s platform completed the task in 61 seconds, and AMD platform completed in 64 seconds. Again, the actual SKUs of both the processors were not mentioned.
AMD’s issue with BAPCo isn’t something new. In 2011, AMD quit BAPCo industry consortium because they deemed SYSmark 2012 was optimized and biased towards Intel. Back then, Nigel Dessau, the senior VP and Chief marketing officer at AMD back then said that they did not believe in SYSMark 2012 and, therefore, cannot endorse 2012 version benchmark and cannot be a part of the consortium. But it should also be noted that Nvidia also quit the group.
While Nvidia did not give any reason, AMD specified that the benchmark did not emphasize on a general-purpose computing using the graphic processing units. Back then, AMD introduced Llano APU. Knowing Nvidia being into graphic processing, you could figure out the reason of their departure from the group.
It should be noted that BAPCo’s consortium does not include only AMD, Intel and Nvidia, but other companies such as Microsoft, Dell, HP, Lenovo and few others.
It should also be noted that back then, Intel said no single benchmark can provide a definitive measure of PC’s performance, and, therefore, encourages all technical reviewers to run multiple benchmarks and applications. That’s the reason why reviewers run a lot of benchmarks from their end and present the numbers. I did the same with Intel i7 5960x review which showed that the HEDT processor doesn’t have any or significant advantage over the mainstream variants in certain specific workloads. Running the battery of tests and benchmarks, especially real-world based tests, helps the reviewer and its readers to understand who is thing product made or- and if it’s worth it.
It should be noted we did meet Intel who was introducing some benchmarks (more towards notebook and smartphone benchmarks, including a BETA benchmark to evaluate the camera image quality irrespective of the device- which didn’t seem to have worked from my end). BAPCo SYSmark and Futuremark PCMark were both talked with the same level of respect.
Intel was more specific compared to AMD’s presentation video. The company’s PECA communications manager Hiral Gheewala talked about Branch mispredictions, L2 Cache misses and L3 Cache misses. Although it’s for mobile processors, I am sure people can relate this with desktop variants as well.
While it is obvious that both AMD and Intel emphasized on real-world benchmarks (as they should), the opinion about BAPCo SYSmark is what separates them. But BAPCo SYSmark 2014 benchmark has three main test scenario- Office productivity, media creation and Data/Financial Analysis. PCMark8 benchmark evaluates using six scenario- Home, Creative, Work, Storage, Applications and Battery Life.
In any which case, while many manufacturers (not limited to CPU and GPU makers) market or bash or praise certain benchmarks, software and games, it’s always good to refer and judge based on multiple benchmarks. At the same time, reviewers need to understand what exactly the benchmark evaluates, use a real world benchmark or a testing scenario (even a time-based evaluation is going to be more accurate than synthetic benchmarks as long as same testing setup and BIOS settings are maintained to which users can relate with) and explain what does it do.
At this point, I should let people know that when I get storage devices and hard drives for review purposes, the previous reviewers have a habit of leaving some data, software and videos (well, there was this one time where a bunch of porn videos was found, whose thumbnail showed a skinny wearing a gas mask doing something- not that I am judging anyone’s sexual preferences as long as I am kept out of it). Many times, it was found to use the non-updated version of the benchmark, followed by a serial number generator. It’s weird to see a well-reputed review source couldn’t buy or request a media copy of the benchmark. Both BAPCo and Futuremark software were present, among other utilities.
— Hardware BBQ (@HardwareBBQ) January 15, 2016