Note: Article title is a tounge-in-cheek reference to a recent ARSTechnica article. See below for the real story. It’s actually all about the browser engine and platform, not the OS.
As I begin a series of tests on the Aava prototype phone today, the first thing I wanted to get out of the way was a Sunspider javascript test. It’s CPU-intensive and forms an important part of the chain of events that take place to get a web application displayed on your browser. Yes, it’s one of many variables but it’s a good indicator of CPU performance.
I’ve been recording SunSpider tests for a long time now (feel free to contribute to that list) and have been very impressed by the way that both hardware and software improvements have brought the figures down. In the ARM-based world, the best-in-class devices are producing SunSpider results in less than 10 seconds (iPad, iPhone 4) with some new devices even reaching down to sub 6 seconds [See title reference.] In the world of Intel, netbooks are producing Sunspider results of about 2 seconds (using the latest Chrome build.)
Click to enlarge.
Above is the result for the Aava phone I’m testing at the moment. 4215ms. It runs on Intel’s Moorestown platform at 1.5Ghz and uses the same CPU as you find in netbooks. Running Sunspider on the latest Firefox build on the netbook and the Firefox Mobile build on the MeeGo-based Aava phone you can see the difference is marginal. 4.2 seconds for a prototype phone is quite impressive. The figures confirm the CPU speed and also confirm that the sunspider score is likely to drop to around 2 seconds with the latest Chrome or Android browser build. Try Sunspider on your phone or PC browser to see what score you get. For reference, I use a ultra mobile PC as my desktop PC. It runs a 1.6Ghz Atom CPU and returns a score of about 4.1s. True desktop PCs will come in at below 0.5s
We really are talking PC-power here but there are two things we must not forget. 1) Multi-core ARM-based devices are round the corner too. A dual-core 1.2Ghz Snapdragon platform is likely to reach right down to the 2-3s range. 2) We must also remember that if you were to run this test continuously on these high-power platforms, you are likely to have a dead phone in just a few hours!
Wait. 3D Morph took 149.9ms to run on that? You know the score I got on my crap PC? ** 26140ms ** And there is NO typo in that figure. This test suite seemed to keep repeating itself and I bailed after watching it churn through things at least 3x — with lots of script errors — and *ten minutes*.
How have to tested the OS ‘s javascript performance ?
Chippy, Sunscript Javascript benchmark doesn’t use multiple cores, therefore Dual Core versions won’t help speed it up.
Thanks David – Are you sure? I tested a Tegra2 on Android 1.6 and the results were sub 10-second.
The only way to multithread javascript in a browser is to use the new WebWorkers API. Otherwise everything is serial.
If a Tegra2 ran sub 10 seconds that’s just a testament to how much ARM has improved single threaded performance on the A9.
If you want to be sure watch task manager during a sunspider run with nothing else going on in the background. Only one core will be busy at a time. Since the Window’s scheduler is so poor at keeping processes in cache and on the same core the browser process will switch cores and appear to keep multiple cores busy but it’s only running one javascript thread. If you try sunspider in linux and run top you will see the browser process never takes more than 100% of the CPU. If it was multithreaded it would take > 100%.
From my reply on UMPCPortal:
Javascript Sunspider results:
Core i5 661(3.33GHz)/X25-M SSD Win 7 64-bit:
IE8: ~4500ms
Safari 4: ~630ms
Firefox: ~830ms
Viliv S5 Atom 1.33GHz HT WinXP 32-bit
A/C Safari 4: ~2990ms
Battery: ~3050ms
The results for Core i7 965EE running at 3.20GHz with a superior memory subsystem gets 840ms(Firefox). Meaning, at least for the modern desktop, it doesn’t stress the computer much at all.
As David points out, javascript is not multithreaded so an extra core doesn’t make any sort of difference. However a 1.5 GHz snapdragon running android 2.2, if it scales perfectly, would take about 2.9 seconds.
What I’m interested in is how much power it takes to do these sunspider runs. I know this is still an early prototype so the numbers are likely to drop over the next few months but what is powertop saying about power consumption on the phone?
I’m afraid I won’t have time to get into any command line work. I’ts taken 24 hours to do the photos, video and familiarisation work and it has to be sent back today. Sorry.
You need to compare with Android 2.2 on ARM Cortex A9 at up to 2ghz. As announced, those will be out before the Moorestown products. Cheaper, better, longer etc (as usual).
Vaporware vs. Vaporware, surely you can do better than that.
Did you see this bug on your Aava? http://bugs.meego.com/show_bug.cgi?id=3730 With that fixed I believe the systerm can deliver even better numbers :)
Took 130 seconds on a humble Nokia 5230 running S60v5 OS and Opera Mobile 10 browser. It is running a 434MHz ARM11 processor and 128MB RAM. The processor and the OS is the same as Nokia 5800 or X6 etc so results on those phones should be similar.
Tested Firefox Mobile 1.1 on a Nokia N800 which I believe runs a 400MHz ARM11 CPU and running Maemo Diablo OS. Result 34.6 seconds.
hmm, maybe i should do the same with the opera mini alpha they have available.
end result, 47seconds.
Nokia N900, ARM Cortex 600MHz, Maemo Fremantle 1.2:
Opera Mobile 10.1: 8.8 sec
Firefox Fennec 3.6.5: 16.9 sec
MicroB: 31.4 sec
Macuco (webkit): 13.9 sec
Dell, Intel Xeon Quad 2,6GHz, Debian Lenny:
Opera 10.6: 0.388 s
Firefox 3.0.6: 3.1 s
Moorestown devices will be heavily software dependent for battery life because of the OSPM. It’s probably not wise to test on a device with prototype software.
The meego kernel already includes extensive power management support for moorsetown and the aava in particular so it wouldn’t be horribly unfair. Certainly production devices would be no worse.
idle power consumption might have a long way to drop yet so testing how long it lasts in the pocket would almost certainly be unfair but load consumption ought to be right around what shipping devices will experience.
Check this out(thanks to Roger): http://bugs.meego.com/show_bug.cgi?id=3730
That’s a substantial bug that will impact battery life. I know with flash video my Viliv S5 can last only 3 hours, but playing WC3+downloading over wifi can do 5.
That is a pretty nasty bug but it would only affect idle power and performance, not load power draw.
Getting a low idle draw is a combination of technical tricks and making sure software behaves itself so idle can be achieved as long as possible. Load consumption purely reflects the quality of the design and process used to make the chip.
With moorestown devices just over the horizon and development kits already shipping this is almost certainly very close to if not already final silicon. I agree intel has lots to do to improve power usage on moorestown but this work relates to idle consumption and how to spend as much time there as possible. At this point they are pretty stuck with the load consumption of the chip.
I’m not so interested in a bettery run-down test as even seemingly intensive things like video or flash playback will leave the cpu idle quite a bit and we’ve already established that’s a bit unfair. What I really wanted to know is if you spike the CPU for a few minutes how many watts does powertop report were used. Chippy has already said he doesn’t have time to investigate any command line tools so I’ll have to wait.
Flash video runs with 70-80% CPU usage on my S5 regardless of flash version OR driver version. If you think that can be called leaving “idle for quite a bit”, ok sure.
one iPad
http://twitpic.com/23p0ew/full
10881.88 ms
6.0.461.0 (51913) Built running on Debian 64bits unstable
model name : Intel(R) Core(TM)2 Duo CPU E6550 @ 2.33GHz
cpu MHz : 2327.966
cache size : 4096 KB
MemTotal: 8067124 kB
Chromium 6.0.461.0 (51913) Built running on Debian 64bits unstable
Total: 431.6ms +/- 5.7%
http://1.bugabundo.net/dybae
Firefox 3.7~a6~hg20100629r46385
Total: 965.8ms +/- 7.1%
http://1.bugabundo.net/bging
QRCode for mobile devices: http://qrcode.kaywa.com/img.php?s=8&d=http%3A%2F%2Fwww2.webkit.org%2Fperf%2Fsunspider-0.9%2Fsunspider-driver.html
HTC Desire with Android 2.2
Total: 5584.4ms +/- 4.1%
http://1.bugabundo.net/xojnf
HTC Magic with Android 2.1 (Cyanongemod 5.0.8) did not finish the test
GMsmart Boston Android 1.6
Total: 38436.8ms +/- 1.3%
http://1.bugabundo.net/krcwu
HTC Magic with Android 2.2 (CM 6.0.0)
Stock Browser
Total: 85299.2ms +/- 12.4%
iPhone 4: 10264.2ms +/- 2.0%
Desktop: Core2Quad Q9300 a 2.5Ghz (windows?)
Firefox: 829.0ms +/- 1.2%
Chrome: 334.8ms +/- 4.5%
AMD 5050e 2.6GHz
Ubuntu 10.04 32bits
FF: 1332.4ms +/- 1.5% :S
Chromium: 438.8ms +/- 5.0% :)
Chrome: 461.0ms +/- 3.3% :)
Midori: 589.6ms +/- 1.1% :)
Opera: 442.8ms +/- 1.1% :)
Windows via VBbox VM
FF: 1456.0ms +/- 3.3%
IE8: 7245.2ms +/- 1.1% XD
Core 2 Duo @ 2.33 with Opera 10.6: 358.2ms
Ubuntu Linux 10.04 (amd64), Chromium 6.0.459.0, AMD Athlon X3 425e @x4 @2835MHz: 316.2ms +/- 4.1%
^
http://www2.webkit.org/perf/sunspider-0.9/sunspider-results.html?%7B%223d-cube%22:%5B34,18,19,19,19%5D,%223d-morph%22:%5B19,14,23,15,15%5D,%223d-raytrace%22:%5B13,14,14,16,13%5D,%22access-binary-trees%22:%5B3,1,2,2,2%5D,%22access-fannkuch%22:%5B14,14,13,15,15%5D,%22access-nbody%22:%5B18,19,14,21,16%5D,%22access-nsieve%22:%5B5,5,4,4,6%5D,%22bitops-3bit-bits-in-byte%22:%5B3,2,3,2,6%5D,%22bitops-bits-in-byte%22:%5B6,6,6,7,6%5D,%22bitops-bitwise-and%22:%5B9,12,10,9,9%5D,%22bitops-nsieve-bits%22:%5B7,8,7,7,8%5D,%22controlflow-recursive%22:%5B3,2,2,2,2%5D,%22crypto-aes%22:%5B8,9,8,8,9%5D,%22crypto-md5%22:%5B4,4,5,4,4%5D,%22crypto-sha1%22:%5B3,3,3,4,3%5D,%22date-format-tofte%22:%5B18,14,13,16,14%5D,%22date-format-xparb%22:%5B27,23,23,31,25%5D,%22math-cordic%22:%5B13,11,12,13,12%5D,%22math-partial-sums%22:%5B16,18,16,16,24%5D,%22math-spectral-norm%22:%5B9,8,11,8,9%5D,%22regexp-dna%22:%5B11,11,11,10,10%5D,%22string-base64%22:%5B9,11,9,10,8%5D,%22string-fasta%22:%5B16,15,16,16,16%5D,%22string-tagcloud%22:%5B23,23,23,23,22%5D,%22string-unpack-code%22:%5B25,25,25,25,25%5D,%22string-validate-input%22:%5B16,15,17,15,19%5D%7D
TOUNGE
IN
CHEEK
/sigh
what the hell … man MEEBO rocks
thats pretty cooool … cant stop admiring
Meego development
Chromium: 438.8ms +/- 5.0% its like a jet on this check out
chromium . it works like super fast
295ms + – 5 %
64 bit win 7. Core 2 duo 3.0ghz 8 gb ram
running on Chrome 5.03
i’ve the samsung jet s8000 and have trouble synchronising with outlook 2003. Electronic mail Sync icon on new pc studio 1.2.0 ID8 and 1.3.0 IJ1 are disabled and hence are not able to synchronise with my outlook 2003, computer operating method is windows xp sp3. have to have support on why the e-mail sync icon is disabled and why sync does not operate.
I liked reading this post so much I wished it was longer. Very engaging writing style!
Samsung wave on BADA OS 1.0 , 1 Ghz snapdragon processor ->
A 3rd Generation iPod touch -> 13507.9 ms
By the way i have a doubt, shouldn’t the benchmark results vary when the same device is tested on two different internet connections? Please bear with me as I am not a developer like you all, I am just a student who is interested in all this. So I am just here to learn.
Samsung wave -> It says page too large, content cannot be displayed.
I left it incomplete in my previous comment by mistake.
Both are using the stock browsers.. i.e safari and bada os browser