So here we are on the day of the highly anticipated Apple iPhone 8 announcement and an iPhone 7 Plus DxOMark score has finally been released. If you’ve been following along, we reported on the lack of an iPhone 7 Plus DxOMark score numerous times and even reached out to the company for comment. Interestingly enough, we seemed to be the only ones doing so.
iPhone 7 Plus DxOMark Score — or lack thereof…
To recap, DxOMark stated in their iPhone 7 review that the iPhone 7 Plus score would be coming shortly:
The iPhone 7 Plus features additional capabilities based on its dual-camera architecture. It is not the first phone with a dual-camera, but it is the first to use the second camera for a 2x optical zoom, and computed depth information. Its new Portrait mode is designed to use the depth information gleaned from analyzing the images from both of its cameras to selectively-blur image backgrounds, while keeping the foreground subject sharp. This is intended to mimic the shallow DOF effect and pleasing “bokeh” that photographers using DSLRs and other standalone cameras have been able to achieve. The new features and technologies in the iPhone 7 Plus require additional time for testing, so its review will follow this one, and be published shortly.
Of course, we still had no score come November of last year, while the Pixel received its coveted DxOMark score on October 4th — the day it was announced. April came and went… the HTC U11 received its score in May… and still no iPhone 7 Plus DxOMark score. Each time we inquired with DxOMark, we received the standard canned response:
We do not communicate about the road map and can only invite you to stay tuned.
Stay tuned we did, but to be honest the staff here at Techaeris pretty much gave up on ever seeing a score for the iPhone 7 Plus. In fact, the past few weeks Alex and I have been tossing around the idea of an editorial questioning DxOMark’s relevance in light of this, and the fact they partnered with OnePlus on the OP5 to optimize the camera for that phone.
New DxOMark Mobile Score Criteria
I’ll admit I was a bit surprised to see the press release in my inbox this morning stating that an iPhone 7 Plus DxOMark score had been published yesterday — on the eve of the upcoming iPhone 8/X/whateveritsgoingtobecalled announcement.
DxOMark’s new Mobile scoring protocol addresses these new smartphone photo capabilities by providing:
- An all-new Zoom sub-score based on extensive testing at multiple focal lengths.
- An all-new Bokeh sub-score using a scene carefully designed to allow comparisons.
- Low-light testing down to 1 Lux.
- Motion added to the test suite for more accurate evaluation of camera performance in real-world situations.
- Expanded lab and outdoor test scenes.
As you can see, there are two major sub-scores added: Zoom and Bokeh — features which are very new to smartphones and mobile photography. As a result, existing DxOMark Mobile scores don’t really mean much anymore. The company has retested the top phone for each year since 2012 to act as a sort of “baseline” for scores going forward.
Several already-tested devices have been retested following the new protocol to provide a basis for comparison—including the #1 smartphone from each year since DxOMark began. These phones include the Nokia 808 (2012–2013), the iPhone 6 (2014), the Samsung Galaxy S6 edge (2015), the HTC U11 (2017), the Google Pixel (2016), as well as the iPhone 7 and iPhone 7 Plus. In addition, DxOMark will be announcing the new test results for several current models in the coming days. New devices will be tested only with the new protocol. (Existing scores and reviews will continue to be accessible, but aren’t valid for direct comparison.)
So just how did the new DxOMark Mobile Scores change over the existing ones? Since we’ve been waiting for the iPhone 7 Plus score, let’s start with that one.
- iPhone 7 Plus: n/a (old), 88 (new)
- iPhone 7: 86 (old), 85 (new)
- HTC U11: 90 (old), 90 (new)
- Google Pixel: 89 (old), 90 (new)
- Samsung Galaxy S6 edge: 87 (old), 82 (new)
- iPhone 6: 82 (old), 73 (new)
- Nokia 808: 77 (old), 61 (new)
As you can see, there are minor changes to most of the scores. The HTC U11, previously the top scored device on DxOMark Mobile, is now tied with the Google Pixel, the smartphone it beat out to gain the top spot. DxOMark mentions in the HTC retesting that “although the two devices have achieved the same overall ranking, they have done so in different ways (highlighted by the new test protocol).”
If you’re interested in the comparison review between the iPhone 7 and iPhone 7 Plus, you can check out DxOMark’s detailed analysis. DxOMark has also gone into great detail to explain their new testing protocols.
Final Thoughts
Now for the “editorial” section of this article, assuming you’re still with us. We get that technology changes and new features need to be taken into account. However, we here at Techaeris lost a lot of faith in DxOMark’s scoring after the very long delay and canned responses when we inquired about the iPhone 7 Plus — a flagship device — scores. What’s even more interesting is DxOMark felt they had to overhaul their scoring system in order to be able to rate the iPhone 7 Plus. Again, we understand that a process like that takes time, but as a side effect — and as DxOMark admits — existing scores and past reviews outside of the seven phones listed above are no longer valid for direct comparisons. In fact, I’d even go so far as to say that only the top four on the list would be valid scores with the new system as devices like the S6 edge, iPhone 6, and Nokia 808 dropped considerably because they don’t have both the new features DxOMark is scoring on.
As anyone in the tech industry knows, while benchmarks are great they don’t always reflect real world usage. There are industry “leaders” as far as benchmarks are concerned, and in the case of DxOMark, the editorial staff at Techaeris found their delay in releasing the iPhone 7 Plus score unacceptable. Yet again, we understand technology changes and existing scoring methods may not cover that new technology, but if other scores were revisited and adjusted after the new scoring system was released, the iPhone 7 Plus could just have easily been scored using the old system and readjusted alongside the likes of the iPhone 7, Google Pixel, and HTC U11.
If you’re an industry leading source of mobile camera rankings, you’d better be ranking just about every phone available in the same timely fashion. Sure, some companies might provide units ahead of time while others don’t but you’d think that for flagship devices, DxOMark would have a standard maximum time frame after the device officially launches to get the DxOMark Mobile score published. Not only that, their partnering with OnePlus on the OP5 calls into question the fact they refer to themselves as “the leading source of independent image quality evaluation for mobile devices.”
What do you think about DxOMark’s new mobile camera testing criteria? Do you think it justifies the delay in the iPhone 7 Plus DxOMark score? Do DxOMark scores even matter anymore now that the criteria has been changed and some past scores updated? Let us know in the comments below or on Google+, Twitter, or Facebook.
[button link=”https://www.dxomark.com/introducing-the-new-dxomark-mobile-test-protocol/” icon=”fa-external-link” side=”left” target=”blank” color=”285b5e” textcolor=”ffffff”]Source: DxOMark[/button]
