Apple has made claims that the iPhone OS 3.0 yields significant performance gains on the 3G model, and that the new 3G S can accomplish the same tasks up to twice as fast as its predecessors. Anecdotally, the new 3G S definitely "feels" faster under certain conditions. But how do Apple's devices and OS versions really compare to one another? And perhaps of even greater interest, how does the latest hardware from Cupertino compare to smart phones recently released from other vendors?
One of the challenges in conducting an evaluation of software performance across devices that utilize different operating systems lies in accounting for the fundamental differences in the various OSs. While the iPhone 3G and 3G S could potentially run the same app on the same Objective-C-based operating system (making direct comparisons relatively straightforward), Android apps are Java-based, and the Palm Pre runs the entirely new webOS. Given these divergent OS implementations, is there anything that comes close to a standard unit of measure for judging performance of this growing breed of "superphones?"
Finding Common Ground
Medialets ran the SunSpider test suite in the following environments:
- Safari 4.0.1 on a 2.0 GHz Intel Core 2 Duo White MacBook.
- The MacBook results were used as a baseline for relative comparisons.
- Mobile Safari on the iPhone 3G with iPhone OS v2.2.1
- Mobile Safari on the iPhone 3G with iPhone OS v3.0
- Mobile Safari on the iPhone 3g s with iPhone OS v3.0
- The "Browser" app on the T-Mobile G1 with Android OS v1.5 (Cupcake)
- The "Web" app on the Palm Pre with Web OS v1.0.2
Each device was fully restored and rebooted immediately before running the test suite. Every attempt was made to assure that no atypical background tasks were executing while the tests were running. The SunSpider tests automatically run five times sequentially and the mean average from all five tests are reported. Network speed and latency have no effect on the results of the test.
Do any of these numbers really indicate which phone might be the best choice for a given individual? Absolutely not. At Medialets we use all of these devices, and love each one for many reasons. The fact that these tests can even be performed across this many device/OS combinations is a testament to how far mobile technology has come in such a relatively short time. We are looking forward to seeing an even greater variety of advanced mobile devices and OS revisions enter the market and we'll keep you posted as we test more devices in our lab.
Rana Sobhany is co-founder and VP of Marketing at New York-based Medialets, a premium advertising network and analytics provider for mobile platforms such as iPhone, Android and Palm. In this role, she leads the company's strategic marketing, affiliate, and community programs, as well as product marketing. Rana specializes in strategic brand management for media and technology companies and is passionate about building communities, both on and offline, using social media and viral marketing. Rana co-founded Aux Interactive, an interactive marketing agency focused on results-driven social media and new platform technologies.