Medialets

Apple has made claims that the iPhone OS 3.0 yields significant performance gains on the 3G model, and that the new 3GS can accomplish the same tasks up to twice as fast as its predecessors. Anecdotally, the new 3GS definitely “feels” faster under certain conditions. But how do Apple’s devices and OS versions really compare to one another? And perhaps of even greater interest, how does the latest hardware from Cupertino compare to smart phones recently released from other vendors?

“Objavectaweb-C” OS?

One of the key challenges in conducting an objective evaluation of software performance across devices that utilize different operating systems lies in accounting for the fundamental differences in the various OSs. While the iPhone 3G and 3GS could potentially run the same app on the same Objective-C-based operating system (making direct comparisons relatively straightforward), Android apps are Java-based, and the Palm Pre runs the entirely new Web OS. Given these divergent OS implementations, is there anything that come close to a standard unit of measure for judging performance of this growing breed of “superphones?”

Finding Common Ground

The common thread between these three OS’s is JavaScript execution in WebKit—the open source project that, in varying degrees, powers web browsing technology for these three disparate operating systems. With the exception of certain browser plugins (e.g., Flash), web rendering technology installed on today’s premiere mobile devices makes almost all—and in some cases even more—features of their ubiquitous desktop web browser counterparts available. Therefore, given the global commonality of JavaScript and WebKit-based web browsers, it becomes possible to compare the performance of these “pocket computers that make phone calls” to the performance of desktop machines.

The Yardstick

The WebKit Open Source Project provides a JavaScript test Suite dubbed SunSpider. According to the description on the SunSpider home page, “this benchmark tests the core JavaScript language only, not the DOM or other browser APIs. It is designed to compare different versions of the same browser, and different browsers to each other.” We at Medialets have found it to be one of the best attempts to measure real world JavaScript performance in a balanced and statistically sound way.

Medialets ran the SunSpider test suite in the following environments:

1. Safari 4.0.1 on a 2.0 GHz Intel Core 2 Duo White MacBook.
   The MacBook results were used as a baseline for relative comparisons.
2. Mobile Safari on the iPhone 3G with iPhone OS v2.2.1
3. Mobile Safari on the iPhone 3G with iPhone OS v3.0
4. Mobile Safari on the iPhone 3GS with iPhone OS v3.0
5. The “Browser” app on the T-Mobile G1 with Android OS v1.5 (Cupcake)
6. The “Web” app on the Palm Pre with Web OS v1.0.2

Each device was fully restored and rebooted immediately before running the test suite. Every attempt was made to assure that no atypical background tasks were executing while the tests were running. The SunSpider tests automatically run five times sequentially and the mean average from all five tests are reported. Network speed and latency have no effect on the results of the test.

Results

The results of the iPhone-based tests alone are rather astonishing and seem to indicate that many of Apple’s claims about the performance gains of their 3.0 OS and the iPhone 3GS may hold some water. Using OS 3.0 on the same iPhone 3G yields nearly 3X the JavaScript performance in Mobile Safari vs. using iPhone OS 2.2.1. The iPhone 3GS ups the ante by another factor of 3, bringing JavaScript performance on the iPhone 3GS to just 12X that of a full-powered desktop machine that has well over four times the raw processing muscle alone. The T-Mobile G1 running the “Cupcake” version of the Android OS completed the test suite in about 91 seconds. This makes it about a third faster than the iPhone 3G running Apple’s previous OS (2.2.1). The Palm Pre came storming out of the gate with speeds that closely rival the iPhone 3G running Apple’s latest iPhone OS.

Do any of these numbers really indicate which phone might be the best choice for a given individual? Absolutely not. At Medialets we use all of these devices, and love each one for many reasons. The fact that these tests can even be performed across this many device/OS combinations is a testament to how far mobile technology has come in such a relatively short time. We are looking forward to seeing an even greater variety of advanced mobile devices and OS revisions enter the market and we’ll keep you posted as we test more devices in our lab. Subscribe to our feed, leave a comment below, or reach out to us directly if you have any questions.

Here we are a little over a week since the launch of the first Android-powered phone, the G1. As of Monday, Android Market has been open to developers to distribute their applications as they wish. We’ve made some more observations on how the Android Market is evolving, and we’re eager to share them with you. 

Highlights

(1) 167 Apps have been downloaded between between 667,000 and 2.9 Million times.

(2) Downloads are being driven by 41 apps, which account for between 73% and 83% of all possible downloads.

(3) Two applications, Pac-Man by Namco and The Weather channel generated the most downloads (50,000 – 250,000).

(4) Although download ranges can be very large, growth on the low-end was still nearly 80% during the first week.

(5) 5 Categories account for 61% of all apps.

(6) The number of apps has nearly tripled since launch, led by the Games, Tools and Productivity categories.

Applications By Category

Android Market applications totaled 167 as of 10/29.  The top 5 Categories with the most apps accounted for 61% of all titles: Tools, Games, Lifestyle, Multimedia, and Productivity.

Title growth has nearly tripled since launch, as the chart below shows.  Three categories – Tools, Games and Productivity led the growth, representing just over 50% of the 105 apps added since last week.

Downloads

Android Market provides download data in ranges or “bins.”  While it’s better than having no information at all (e.g. Apple’s App Store) it gets a bit unwieldy within the top 2 bins, 10,000 – 50,000, and 50,000 – 250,000, where the top end is 5x larger than the bottom end, and the variance can be up to 200,000.  In the chart above, we’ve looked at the trend of total downloads from both ends of the ranges given, and growth seems pretty healthy on both ends.  Apps downloaded from the end of the Market’s first day until 10/29 grew at a rate anywhere between 80% on the low end to potentially 97% on the upside.

The table below gives detail on the distribution of downloads, which are driven by 41 apps in the two highest bins. Taking the low/high ranges into consideration, these 41 apps comprise anywhere between 73% and 83% of all possible downloads

Within these 41 apps, two of them, Pac-Man by Namco and The Weather Channel generated the highest number of downloads (50,000 – 250,000).  The remaining 39 are in the 10,000 – 50,000 range – we displayed the “Top 5″ in the table below, which were determined by  the most recent average of user ratings.  

We’ll keep you posted as data continues to roll in.  In the meantime, we’d love to hear your comments and ideas as our ecosystem continues to evolve.  Reach out to us at connect at medialets dot com and add us on Twitter at @medialets.

Google’s Android Market has been officially live for 24 hours. Here are some early observations and comparisons with the iPhone App Store’s first 24 hours.

There are myriad similarities between iPhone and Android users:

• They like to play games, shop, and know what music they are listening to,
• They are curious about the weather, and
• They generally share the same interests as iPhone users 

During the first 24 hours of Android Market, 62 apps were available to consumers, all free.  This is less than 10% of the number of apps we saw at the launch of Apple’s App Store. Although Apple allowed both free and paid applications to be distributed when the App Store launched, paid downloads for Android will not be available until Q1 2009.

Observation #1 – The average application has 7,800+ downloads.

Android Market is providing some detail on downloads per application –  in contrast to Apple’s embargo of this information after the first 15 hours of launching their App Store.  Rather than displaying exact figures, Android phones show download ranges for a given application, with the smallest range we observed being 100-500 and the largest 10,000-50,000. Given those ranges, roughly 206,000 to 770,000 downloads occurred within the first 24 hours of launch. The weighted average of midpoints is 7,850 downloads/app, just north of the middle of the 5,000 – 10,000 range.

Observation #2  - Nine apps made it to the 10,000 – 50,000 downloads range.

If we use ratings and number of reviews to differentiate, unlike the iPhone platform, games are not in the top three.  Of the nine apps in this range, only three are games.  ShopSavvy is at the top of the list factoring in ratings and number of reviews, followed by The Weather Channel, and Shazam, an app that helps people identify a song they are listening to.  During the early hours of the iPhone App Store, while Apple was still publishing download data, only two apps broke the 10,000 download mark – Remote and AIM.  Remote, the leading app, was downloaded ~16,000 times. Although the ranges for the top Android apps are similar, it is still too early  to assert with confidence that a trajectory similar to Apple’s App Store is occurring.  Other factors over time need to be considered, including the total number of apps in the market.

Observation #3 – 24 hours into the launch, it appears that either Android users are generally interested in the same types of application functionality as iPhone users, or possibly, that Android developers are generally interested in creating the same types of apps as iPhone developers.

We compared our observations of the iPhone App Store 24 hours after launch with Android Market, and found that once we normalized the names of categories between the two platforms, the categories have similar distributions of applications. We had to make some assumptions and groupings to make our best apples-to-androids comparison and noted those in the table below the chart.

As always, we love hearing from anyone who is interested in learning more about, or sharing their experiences about this new platform. Feel free to contact us at connect at www.medialets.com or follow us on twitter at @medialets.