> But for a hardware test, I'd expect a single running app, executing shared native codebase, in no disruptions airplane mode, measuring specific part of the hardware. And that's not even close to measuring before/after improved branch prediction, which could be useful in both phones.

I would argue that would be a completely useless test unless you're explicitly testing something for development purposes only. Users use phones. Nothing else matters but the user experience. If you can make your phone beat another phone in your limited, no disruptions, network traffic off example that doesn't mean anything.

I feel this is a fair test. You test the phones at what they're supposed to be able to do. If one does it better it's legit to point that out regardless of what the underlying structure looks like.

I think you either misread my comment or missed the point of it. Overall tests matter, specific tests matter, unittests matter, microbenchmarks matter, user experience matters, perceived performance matters. But they don't always intersect.

Bringing up user experience when talking about branch predictors is meaningless. Just as bringing up branch predictors when talking about perceived performance. (unless you're proving that this exact feature being available/not available, with other things being controlled for, makes a major difference in the test)

Branch predictors made the phones faster and this affects the user experience. But if you want to test the performance of branch predictors you have to compare similar things, you cannot benchmark two different OS and apps because you get and OS and app test not an processor test.

I think the point that previous posters were making is that in controlled environments some features might outperform others. And yes, the scientific way of testing things is isolating variables and changing one thing each time.

But, for the final user, it doesn't matter if each technology alone is better than another. If you have a background 64-core processor, driven by another 8-core... What matters is how fast something loads, performs, and how long batteries last.

Of course, this might be outplaced in a thread discussing NN branch prediction.

Why measure and therefore optimize for anything but how the product is being used?

I didn't say it's not valid, or that it shouldn't have been done. I'm just saying "NN still doesn't help them beat iPhone" is not a relevant comment regarding this test. To go with car analogy it's close to an article about an slightly improved design of one part of fuel injection in new ABC and someone commenting "look, an XYZ from last year still can go faster without it".

The test is a valid observation, but is irrelevant to this article, or to the technology.

Why measure something that doesn't show up in a standard user's test? Here are some examples: lower power usage, lower latency of small operations (rather than throughput of large actions), smaller design (saving chip space), etc. And finally - if this is a good technology, Apple can use it too and get even faster.

That is a great analogy. That test in the video didn't really mean anything towards the overall speed of the device. It's interesting to see, but I how this test would change over n trials.

You should do that, but not only that. If you just test the way the product is being used, you have a million possible reasons why it could be slow. If you just want to know whether slapping a neural network on the chip is a good idea or not, it's best to test that in such a way as to exclude as many confounding factors as possible.

In general: because how a piece of hardware (or set of hardware) is used today isn't necessarily how it's going to be used tomorrow. Faster hardware allows for use-cases that weren't possible prior.

However, when we're talking about phones or consumer devices, how it's used today is the only thing that's relevant. It will be obsolete in just a couple of years.

I think the parent point is still valid. Look at Pokemon Go. Average user wouldn't care about AR performance, all that high gps accuracy or gfx performance in unity. Suddenly, there are millions of players who are choosing their hardware based on that. If you are planning for today's usage you you may be lagging.

> Everything else is pointless.

Better understanding individual components lets engineers build better overall systems. Millions of choices and decisions go into something like a phone, most of which can't have their result easily measured by just looking at the end result. But if enough of them are good, you end up with the iPhone.

If enough of them are bad, you end up with mass recalls, and a class action lawsuit because your phones catch fire when charged overnight because your engineers didn't think to test that specific use case, didn't test their capacitors, didn't test their QA process, didn't unit test their battery management code, or otherwise failed to indulge their "fetish" (read: job.)

Proper hardware tests might be pointless to end users, but that doesn't make them pointless.

> Err? It seems to me that the only tests that make any sense are those that count for end users. Everything else is pointless.

If you are specifically comparing two specific phones against each other then yes this is the only sort of test that matters, and if you are comparing manufacture's flagship models then that is what you are doing.

From every other point of view, either comparing performance of individual components or comparing like-for-like performance of devices with very similar spec it is less valid though: the screen resolution is a huge variable for a game so it isn't a "fair" test to compare performance of two devices like that. You could equally state that the higher resolution produces a better result because of the resolution difference and you would similarly be called to task by for not comparing like for like. Maybe the a device with the same hardware other than the better screen would outperform the iDevice on the test, or at least not appear to underperform by as much, this test can't tell us that.

This is difficult of course because few phone models are practically identical and few vary by just one factor, and can make a developers life more difficult when trying to support a varied market like Android based devices.

If you're an engineer who's working on branch predictors, or a developer working on compiler optimisations, then such a test isn't exactly pointless...

This os a test for an entire phone, including the OS. The article focuses on the chip alone.