Medialets

Apple has made claims that the iPhone OS 3.0 yields significant performance gains on the 3G model, and that the new 3GS can accomplish the same tasks up to twice as fast as its predecessors. Anecdotally, the new 3GS definitely “feels” faster under certain conditions. But how do Apple’s devices and OS versions really compare to one another? And perhaps of even greater interest, how does the latest hardware from Cupertino compare to smart phones recently released from other vendors?

“Objavectaweb-C” OS?

One of the key challenges in conducting an objective evaluation of software performance across devices that utilize different operating systems lies in accounting for the fundamental differences in the various OSs. While the iPhone 3G and 3GS could potentially run the same app on the same Objective-C-based operating system (making direct comparisons relatively straightforward), Android apps are Java-based, and the Palm Pre runs the entirely new Web OS. Given these divergent OS implementations, is there anything that come close to a standard unit of measure for judging performance of this growing breed of “superphones?”

Finding Common Ground

The common thread between these three OS’s is JavaScript execution in WebKit—the open source project that, in varying degrees, powers web browsing technology for these three disparate operating systems. With the exception of certain browser plugins (e.g., Flash), web rendering technology installed on today’s premiere mobile devices makes almost all—and in some cases even more—features of their ubiquitous desktop web browser counterparts available. Therefore, given the global commonality of JavaScript and WebKit-based web browsers, it becomes possible to compare the performance of these “pocket computers that make phone calls” to the performance of desktop machines.

The Yardstick

The WebKit Open Source Project provides a JavaScript test Suite dubbed SunSpider. According to the description on the SunSpider home page, “this benchmark tests the core JavaScript language only, not the DOM or other browser APIs. It is designed to compare different versions of the same browser, and different browsers to each other.” We at Medialets have found it to be one of the best attempts to measure real world JavaScript performance in a balanced and statistically sound way.

Medialets ran the SunSpider test suite in the following environments:

1. Safari 4.0.1 on a 2.0 GHz Intel Core 2 Duo White MacBook.
   The MacBook results were used as a baseline for relative comparisons.
2. Mobile Safari on the iPhone 3G with iPhone OS v2.2.1
3. Mobile Safari on the iPhone 3G with iPhone OS v3.0
4. Mobile Safari on the iPhone 3GS with iPhone OS v3.0
5. The “Browser” app on the T-Mobile G1 with Android OS v1.5 (Cupcake)
6. The “Web” app on the Palm Pre with Web OS v1.0.2

Each device was fully restored and rebooted immediately before running the test suite. Every attempt was made to assure that no atypical background tasks were executing while the tests were running. The SunSpider tests automatically run five times sequentially and the mean average from all five tests are reported. Network speed and latency have no effect on the results of the test.

Results

The results of the iPhone-based tests alone are rather astonishing and seem to indicate that many of Apple’s claims about the performance gains of their 3.0 OS and the iPhone 3GS may hold some water. Using OS 3.0 on the same iPhone 3G yields nearly 3X the JavaScript performance in Mobile Safari vs. using iPhone OS 2.2.1. The iPhone 3GS ups the ante by another factor of 3, bringing JavaScript performance on the iPhone 3GS to just 12X that of a full-powered desktop machine that has well over four times the raw processing muscle alone. The T-Mobile G1 running the “Cupcake” version of the Android OS completed the test suite in about 91 seconds. This makes it about a third faster than the iPhone 3G running Apple’s previous OS (2.2.1). The Palm Pre came storming out of the gate with speeds that closely rival the iPhone 3G running Apple’s latest iPhone OS.

Do any of these numbers really indicate which phone might be the best choice for a given individual? Absolutely not. At Medialets we use all of these devices, and love each one for many reasons. The fact that these tests can even be performed across this many device/OS combinations is a testament to how far mobile technology has come in such a relatively short time. We are looking forward to seeing an even greater variety of advanced mobile devices and OS revisions enter the market and we’ll keep you posted as we test more devices in our lab. Subscribe to our feed, leave a comment below, or reach out to us directly if you have any questions.

Just 18 days after the launch of the Palm Pre, their App Catalog has hit its first major milestone, 1 million downloads. On launch day, Palm sold 50k devices and their App Catalog experienced 100k downloads – that’s an average of 2 apps downloaded per device and 5,500 downloads per app (with a low of 600 and a high of 20k). 18 days later and they’ve tripled the user base to 150k units, averaging 6+ apps downloaded per device and 33.3k downloads per app (with a low of 2.4k and a high of 114k).

A Perspective: Apples To Palms

The iTunes App Store is the most successful app store in the mobile market today, so it makes perfect sense that analysts and critics immediately compare the two. Unfortunately, there really is no common ground for the two app stores based on the numerous factors involved. So instead of a comparison of the two, we’ll just use the precedent the App Store has set to put things in perspective.

The App Store hit the 1 million download point about 17 days earlier than the App Catalog, housing more than 16x the amount of apps and was accessible by more than 26x the number of devices than the App Catalog. Looks pretty bad for the Pre, right? Well, that’s until you take a look at the downloads themselves.

Still comparing at the 1 million mark, the average Palm Pre user had downloaded 26x the number of apps that iPhone users had, and the average app in the App Catalog experienced 16x the number of downloads that apps in the App Store had experienced. Funny how those numbers look so familiar.

So congratulations Palm! The Pre is an amazing device with lots of potential and we’re looking forward to what a public SDK will offer both consumers and developers alike. And just like my father told me on my millionth download “Learn from the mistakes of those who came before you.”

On Saturday, June 6th, the Palm Pre launched nationwide after roughly two years of development. Two years isn’t a whole lot of time when you think about the fact that they not only created a new device, but a new operating system and app store at the same time. The device has a sharp design (literally and figuratively), a stunning user interface, and a lot of potential to grow into a competitive platform in the smartphone market.

On May 29th, 2009, 9 days before the device officially launched nationwide, the Palm Pre App Catalog went live with 4 apps (Classic, Sudoku, Today Show, and WHERE). By launch day (6/6/09), this number grew to 18 apps total and then jumped to 30 at the end of the first week (6/12/09), and has remained unchanged since. Compared to the other app stores we’ve seen so far, this number is a mere fraction on what we’ve experienced at launch, but there are a few factors that paint a picture as to why this isn’t an issue.

Palm has been very selective about who they have offered their development SDK to prior to it’s expected public release later this summer. This decision stems from the fact that originally, Palm had no plans to launch their App Catalog on day one. Somewhere along the line, they changed their mind and went forward with a beta version of the App Catalog, which many would agree is much better than launching without one. Those previously developing for PalmOS were approached directly by Palm with the WebOS SDK. This offered Palm more of a launch filter than any other app store has witnessed.  Also noteworthy is that aside from Palm’s App Catalog, the only other app store to have a simultaneous launch alongside the first device running its operating system is the Android Market, which launched with just over 60 apps in late October, 2008.

Digging deeper into Palm’s App Catalog actually provides quite a bit of information. The first thing to note is that the entire store and all of the apps in the store are in beta except for one app (Classic by MotionApps). The App Catalog clearly lists this with a banner over the top right corner of the screen and so far, only the app Classic has had a version number of 1 and higher, the rest have been variations of 0.9 or lower.

The most distinctive piece of information we see is that the App Catalog lists actual downloads, which no other market currently does (see the Chart below).  Apple’s App Store listed downloads only for a couple of hours post-launch before they were made unavailable.  Android Market provides “buckets” of download ranges which, at the low end are helpful, but at the upper end vary widely (e.g. 50K -  250K, >250K).

For developers, the greatest appeal of working with WebOS has been the promise of a platform that is simple and easy to develop for. With the limited time the select developers have had access to the SDK, the fact that the App Catalog houses two developers (out of 28 total) that each currently offer two apps definitely gives that impression. What appears to be the issue at this point, and one of the major hold ups behind releasing a public SDK, is the App Catalog itself.

Currently in the App Catalog, when a developer updates an app, their release date changes to the date they released the update along with removing all traces of the original date. This allows for any developer to release an update and reposition themselves at the top of the Most Recent category and the top of whatever other categories they belong to when sorting by date. After all the gaming we’ve seen take place in the App Store, this issue, coupled with a lack of payment system in the store itself, are two of the major reasons why the App Catalog isn’t ready to handle the volume of submissions that a public SDK would bring. While releasing the SDK to the public and barring submissions/approval to the store may sound like a smart alternative to some, Palm has clearly thought things through and decided to go with their current selective approach.

The excitement for the Palm Pre has only grown since we first heard word of it. Now, nearly two weeks after launch with the hype winding down, you have to wonder, did Palm miss their window of opportunity with their App Catalog, or do they have something up their sleeve?