State of mobile web development, part 3/3: the mobile industry’s failings

Recently Mike Rowehl, a mobile developer with relatively little knowledge of the web world, confessed to being baffled by the attitude of web developers interested in mobile.

This is the last part of my reply. In parts one and two we talked about what web developers are doing wrong; now we’re going to talk about the errors of the mobile world.

Web developers, web developers, web developers

Company after company declares its eternal and undying allegiance to the web. W3C Widgets, pioneered by Opera and Vodafone, are now more-or-less supported by Samsung, Nokia, and RIM. Palm staked its entire existence as a company on webOS.

Problem is, companies that choose for the web don’t bother to follow up with outreach to web developers. I was fascinated by webOS when it was announced in early 2009, but I was unable to establish a connection to Palm and in the end decided not to bother with it. Similar stories abound.

The first company that succeeds in really communicating with web developers will get its product accepted by an already-existing, large and tightly-knit development ecosystem that hasn’t taken sides yet in the mobile platform wars. As a result, its web-based platform will be filled to overflowing with cool applications.

But mobile companies just don’t seem to be able to do it. (Based on personal observations I’d say Vodafone, Palm, and RIM are willing; but they’re still in the process of getting their act together.)

Absent any change the prize will eventually go to Google or Microsoft (or possibly Opera) because these companies have already established communications with (desktop) web developers. (Apple could do it, too, but it’s not interested since it already has a platform filled to overflowing with cool applications.)

In other words, if the mobile companies don’t adapt to web developers they’ll be usurped by the old “fixed web” companies — at least, as far as web technologies are concerned.

Fiercely independent

I think I have a vague inkling of what the fundamental problem is. The web development ecosystem of blogs and conferences, books and barcamps, was created by web developers themselves by the sweat of their brows. Especially during the early formative years we got little support from large companies.

That changed, when (in no particular order) Adobe, Yahoo!, Microsoft, Google, Opera and some other companies were accepted by the web developers and were listened to seriously.

Why exactly these companies? Because they took the time and trouble to listen to web developers, meet them on their own ground, and make serious efforts toward standards compliance.

Still, web developers have learned to be fiercely independent and to distrust large companies, and especially marketing nonsense. They’re quite willing to listen to serious arguments, but those arguments have to be made in their style, at their conferences and blogs, and have to address issues that web developers deem important.

However, mobile companies are just telling their marketing people to establish contact, and web developers are allergic to marketeers. They want to be in touch with engineers and technicians. As a result there’s no meaningful communication.

Contests

The highest revealed truth about communicating with web developers is a kind of contest, where there’s an inordinate amount of money to be won with writing a widget, an HTML5 game, or whatever technology the organising company wants to push.

This doesn’t work. Sure, some web developers are swayed and participate in order to get some money. But, even if they win, will they ever make a mobile web product again, when there’s no specific incentive to do so?

As far as long-term results go the mobile companies could just as well have flushed their money down the toilet. If they’d spent a fraction of it on actually communicating with web developers on web developers’ terms, they would have some results to show now.

SDKs

Another point that just has to change is SDKs. Right now the revealed truth is that anyone who pushes anything into the mobile web space must release an SDK, too.

They’re just not necessary. If you want to make a website, you need a text editor, a graphics program, a web server, and some browsers to test in. That’s it, and many of the best web developers pride themselves on needing nothing more.

Besides, there’s far too many SDKs. If I were to take them seriously, I’d have to install the Android, BlackBerry, Samsung, Vodafone, Microsoft, and Nokia ones, plus probably a few more I can’t remember right now. Not only will that make an unholy mess of my computer, it’s also far too much cruft if I just want to make a website or mobile web app.

The only thing that these SDKs hold that’s actually interesting to web developers is the various signing procedures for widgets; especially for the BlackBerry and Samsung bada platforms. I wonder if web developers are going to download them just for this one feature.

Sure, if you have a paying client you’ll do it, but I’m thinking of web developers that don’t have particular mobile clients but want to try their hand at these exciting new technologies. Will they download Eclipse plus a series of plugins for three, four, six, eight platforms? I don’t think so.

It would be very useful if signing procedures could be done online. Upload your widget and request a sign-off, and a few days later you’d get it back signed and ready to run. Unfortunately all mobile companies are still in walled-garden mode, so this won’t happen any time soon.

Concluding: mobile companies are fundamentally unable to communicate with web developers.

Browser detects

Twelve years ago this was the highest reveled truth about making websites. Nearly everybody did it.

And everybody was wrong. Not “there’s something to say for it but sometimes you don’t need it” wrong, but just plain “you have no clue what you’re doing” wrong.

Do you know why nearly all browsers start their UA string with Mozilla/? It’s because of browser detects. Not current browser detects, but those of the early nineties. Back then clueless people used the string Mozilla/ to determine whether a browser was modern (i.e. Netscape) or not.

When IE hit the market it could do most of what Netscape could. However, in order to end up on the right side of the browser detects, Microsoft, too, had to start its identification string with Mozilla/. All browsers that were released later followed suit.

Just say no

I was one of the first web developers to speak out against this practice, and I will continue to do so now that I’ve moved to mobile. I aim to prove that it’s simply not necessary, besides being wrong on a fundamental level.

As far as I’m concernd people who use it aren’t aware of current thinking in web development (and by “current” I mean anything that happened in the last twelve years).

What I’ve noticed far too often is that people who don’t know too much about the intricacies of web development choose this solution, mostly because it seems an easy way out of a tricky situation.

It was only a few months ago that I talked to someone from the mobile world who advocated device detection “because all Java programmers do it.” That’s the perfect summary of all that’s wrong with this technique.

That said, I must admit that even within the web development community some are arguing for a form of detection. It would not be based on the browser string, but rather on a list of capabilities created by some nifty JavaScript checks, and it would not serve completely different websites, but only those parts of the script that are necessary for the device, leaving out useless code branches meant for other devices.

To me, the crucial question is whether the server-side device detection takes place by browser string or otherwise. If by browser string, it’s just plain wrong. Period.

Conclusion

In this series we’ve seen that mobile web developers are making plenty of mistakes, but that that’s mostly due to an iPhone-centeredness that’s partially (but not entirely) caused by demand, coupled with a crippling lack of tools that are available on desktop but still have to be ported to mobile.

Simultaneously, communication by the big mobile companies that don’t play a role in the desktop web, is shockingly bad.

As far as I’m concerned that explains why there’s such a discrepancy between what mobile web developers should do, what they actually do, and what mobile companies have to offer.

Incidentally, if you’re interested in these matters, why not join the Mobile Web?

Comments

1 Posted by Leonardo on 22 September 2010 | Permalink

So do you think that libraries such as WURFL and DeviceAtlas are the wrong approach for the problem?

2 Posted by ppk on 22 September 2010 | Permalink

For the moment I'm saying they're using the wrong approach, but I'm open to arguments to the contrary.

Their data is very useful, but the conclusion that one *must* use a browser/device detect in order to use that data is wrong.

I think.

3 Posted by Terren on 22 September 2010 | Permalink

You say browser detection is the wrong approach and has been for years, but then you say progressive enhancement isn't here yet (though a few pioneers are making it happen). So the alternative to doing it the wrong way is to be a pioneer? Sorry, but I have to get things done.

I'm happy to let the pioneers pave the way, and then I will adopt whatever best practices they come up with. Until then, it is far easier (for us) to detect the device and serve a lowest-common-denominator mobile version or a full blown desktop version.

Down the road, of course, we'd love to do progressive enhancement, I'm certainly not wedded to our current strategy. But we only have so many resources and device detection works (a la WURFL), even if it has the flaws you mention.

4 Posted by jl on 22 September 2010 | Permalink

I've tried tera-wurfl library and it was messy code and offered poor detection (I've added my own test suite and easily found dozen UAs that weren't matched). I had to add several fallback regexes to make it usable.

And I'm afraid we'll end up with every single mobile browser having iPhone in their UA string, because they get served Flash or HTML2.0 shit otherwise.

5 Posted by Mike H. on 22 September 2010 | Permalink

Could Javscript ever be an optimal detection method? I worry that it gets executed too far down the scope of the request. Sees to me that detection should happen as high up the chain as possible.

6 Posted by Ronan Cremin on 22 September 2010 | Permalink

"1. It’s incomplete. There’s just no way we’re going to be able to incorporate all devices in this detect. And even if we miraculously did, the next week a new device would hit the market and we’d be incomplete again."

This is not a problem in reality. Devices hit the market relatively slowly in practice and people inside the industry have typically tested and profiled devices long before they reach the hands of consumers, and even longer before they are used to access public websites. Solutions like DeviceAtlas and WURFL ensure that you don't need to care about this -- you just make sure that your device database is fresh. (DeviceAtlas releases daily updates)

7 Posted by Ronan Cremin on 22 September 2010 | Permalink

"2. It’s unreliable. You’re always going to miss something or make a mistake in your detection."

Again, not a problem in reality. Yes, there will always be devices that you don't have or fail to recognize but both DeviceAtlas and WURFL will correctly detect the vast majority of devices in use.

8 Posted by Ronan Cremin on 22 September 2010 | Permalink

"3. It’s an arms race. If device detection really catches on, browsers will start to spoof their user agent strings to end up on the right side of the detects. Web developers will retaliate by even more finely-grained (and even less complete and reliable) detects, which will cause the browser vendors to improve their spoofing ..."

First off, note that device detection has very much caught on already, unless you think that sites Google, Facebook, Microsoft, BBC, LonelyPlanet, AccuWeather and countless others are anomalous.

Secondly, in 10+ years of mobile browsers there isn't really any evidence of spoofing happening yet, with the sole exception of Opera who take a different philosophical stance on the problem (their opinion is that mobile-friendly websites are not necessary with their transcoder & rendering technology). The Opera problem is easily worked around (they thankfully send the original UA string as well as their faked one). In fact we've seen the opposite happening -- Nokia briefly toyed with desktop-like user agent strings in a couple of early Webkit browsers but quickly changed back. The vast majority of other mobile browsers have quite distinct UA strings that are easily told apart with near 100% reliability.

9 Posted by Ronan Cremin on 22 September 2010 | Permalink

"4. It’s cheating. If you make any mistake at all (and that’s inevitable, really), you’re either denying advanced functionality to a browser tht can handle it, or sending advanced functionality to browsers that can’t handle it. In either case you cheat your end users."

You're assuming that it isn't possible to do this well. It is possible, it's not that hard, and many people do it. The issue of mistakes is minimal. How is an attempt to serve the best experience to all devices cheating? Surely this is better than a one-size-fits-all approach that necessarily dumbs down everything to the level of the least capable device?

10 Posted by Ronan Cremin on 22 September 2010 | Permalink

"5. It’s inflexible. Sites that were created for any device will be much more flexible in all kinds of situations because the web developer planned for the unknown. Conversely, if you rigidly sort devices into groups and then code for those groups, this rigidity is going to make your site unable to react to the unforeseen."

Device adaptation is only less flexible if you do it badly i.e. with no fallback to a less sophisticated rendering if you don't recognize the device. Rigidity is a bad implementation.

Ultimately the device detection debate comes down to customer experience: if you want to deliver your customers the best experience regardless of what device they're holding, you really have no choice. Yes, it's harder to do this but for some sites it's well worth it.

If your mobile audience is important to you, do it. If it's not, don't.

(Sorry about the number of comments - it's hard to response to a 10Kb article in a 1250 byte comment)

11 Posted by Ronan Cremin on 22 September 2010 | Permalink

@Mike H.

Not really. There are a couple of issues. 1) The main issue is that JS is not understood by quite a few devices that are in use and, 2) by the time you fetch any JS linked in the page you've already served up quite a bit of HTML and the device may fetch linked resources before executing any JS, which defeats much of the purpose of device adaptation.

12 Posted by ppk on 22 September 2010 | Permalink

A script tag blocks downloading and parsing of the rest of the page. Therefore, if you place a detection in a script tag right at the start of your page, the rest of the page will wait for the script to be executed.

That said, from a performance perspective a server-side detect is indeed better. But it's just not reliable enough.

13 Posted by Daniel Hunt on 22 September 2010 | Permalink

@ppk: "...from a performance perspective a server-side detect is indeed better. But it's just not reliable enough."

Can I ask how or why you've come to this conclusion?
There are plenty of use-cases for device detection (which may or may not also include device capabilities), and with the introduction of more capable devices with standards aware (and compliant) browsers in my mind it becomes even more important.

Dumping 20 or 30 lines of 'magic' JS device detection (or JS support detection) to an iPhone may not break things, but doing the same thing to a 4 year old S40 could do, literally, anything. It might work, it might crash, it might fail gracefully. Who knows?

14 Posted by Daniel Hunt on 22 September 2010 | Permalink

I should also add that dumping:
//magic JS support detection goes here

... isn't a particularly good idea. What works for a S60, may not work for a BlackBerry, and may not work for a S40 ... and so on

15 Posted by Scott Jehl on 22 September 2010 | Permalink

In regards to doubts about the presence of UA spoofing on mobile:

I've got several devices here in the jQuery Mobile "testing lab" which the let you spoof your UA in the same menu you'd use to disable popups!

Blackberry Storm for instance: Options > Browser Configuration (Browser Identification choices are Blackberry, Firefox, Internet Explorer).

I'd imagine the others aren't hard to find.

+1 for feature-detection driven PE!

16 Posted by Esa on 22 September 2010 | Permalink

You did not mention connection speed.

That is one of the crucial parameters that I'm using.

Thank's for a great series of articles.

17 Posted by Bob Boson on 22 September 2010 | Permalink

Feature detection beats browser string testing, no question.

But lots of mobile devices are seriously buggy. Should I come up with a complex JS test for each one, which would be executed on every client that's visiting? Or should I just issue a couple of patches to specific phone versions?

Until mobile browsers reach the maturity of desktop browsers, the browser string will probably be the best way for bug detection on mobile.

18 Posted by Ronan Cremin on 22 September 2010 | Permalink

@Scott Jehl

From memory, there are less than 10 devices that let you change the UserAgent string from a menu option.

There are >6,000 devices that don't.

Even on the devices that do let you spoof the UA, I can't imagine how any normal user would find this option, or dare to change it.

19 Posted by Steve Souders on 22 September 2010 | Permalink

@Mike H point about getting detection more upstream is spot on. This is a requirement for progressive enhancement and document flushing. I encourage the use of WURFL and Device Atlas. I wrote Browserscope's device detection and it works really well.

20 Posted by gossi on 22 September 2010 | Permalink

I've just started my first mobile web development experience. I am a student and can hardly afford my own mobile contract but I got a pre plus with it. So, that's the device I can test with and my desktop browsers, I cannot afford more.

I can install various browsers on my pc err mac to test my websites but I cannot do this for mobile yet, that's a really big issue.
What would be really helpful would be a VM of the other mobile browsers. I'm not saying a VM of the whole OS with all its features, don't kill me with overblown stuff. I am just interessted in the browsers.
Would be really great if Mobile OS developers could release VMs (for Virtual Box?) that run their browser (or other browsers on their system, like Opera Mini). They all need some kind of option to simulate touch events with the mouse.
Maybe they can all decide on an API, that they can release their browsers for a special Mobile-Browser-Box with the touch-simulate option. We as webdevelopers would definitely benefit of that.

@ppk you have good connections to these vendors? If you like the idea feel free to use it and demonstrate to them :)

21 Posted by lazukars on 23 September 2010 | Permalink

There are plenty of articles out there telling developers what not to do, but a lot of us want to know what to do. There are great tools out there that can help. I just wish someone came up with a a bulletproof guide putting it all together. Hey, maybe I should get started :)

22 Posted by Roland van Ipenburg on 23 September 2010 | Permalink

I'd rather do server-side device detection in the bitmaps I serve to various mobile devices than rely on some JavaScript feature detection to be available in every HTML page that might include those images. Mobile devices can be pretty bad at dithering a 24-bit image to their screen palette, so serving images pre-dithered to a screen palette gives better results.

23 Posted by Daniel Hunt on 23 September 2010 | Permalink

Without wanting to pimp it too much, this is related to what we're talking about:
@lazukars There are quite a few articles and blog posts on http://mobiforge.com that can be used to guide you and other mobile developers

24 Posted by James Pearce on 23 September 2010 | Permalink

How will client-side feature detection turn a airline's brochure-ware web site into a mobile electronic check-in service?

Progressive enhancement assumes that all users want the same thing, but with jiggled layout.

Wrong.

Design for the user and her context - not the screen size and Javascript capabilities of the lump of metal, plastic and glass she's holding.

25 Posted by Damien P. on 23 September 2010 | Permalink

I'm upset about mobile os companies because they do use their own language and tools. And as a contrast with computer browsers there are only 3 or 4.

We (the web developpers) can't make a website ready for so much phones, it's time-consuming !

26 Posted by Ian Homer on 24 September 2010 | Permalink

We have the opportunity to use the best of both worlds. Progressive enhancement as a baseline HTML which we would like to deliver, but with server side processing (with device detection / javascript capability querying) to:

* handle cases where devices don't comply with the progressive enhancement techniques and tweak the markup before delivery
* evaluate media queries prior to delivery, to minimise content delivered
* optimisation / selection of appropriate media - images, video, flash to deliver
* aggregation of page objects (e.g. CSS, javascript) prior to delivery to reduce number of page objects loaded by browser

in this way we can get excellent device coverage today and transition towards ideal best practices in the future.

27 Posted by Ian Homer on 24 September 2010 | Permalink

... and I second James' comment. Not all functions / features will be delivered to all the devices in the same way. Consider how the introduction of browser access to camera and contacts will impact the design of the web app. Potentially javascriptable, but using javascript alone to apply these design variations might be tough.

28 Posted by Ronan Cremin on 24 September 2010 | Permalink

+1 to @JamesPearce This needs to be more than just visual jiggling & hence is really hard to do within the browser.

29 Posted by Alejandro Woywood on 28 September 2010 | Permalink

I vote for server detection. We handle 2 operator mobile portals with a lot of pages. For us mantainability is 1st priority, we simply cannot afford to provide the best-best-best experience to all devices.
Therefore we segment all devices in a number of groups (wml, XHTML-128, XHTML-240, XHTML-240 touch, HTML5, HTML5 touch, ...) and create a different presentation for each group in each page.
Furthermore, that's the best way to work with normal designers and mobile-savyy designer (each works with a group).

Progressive adaptation needs too much human power.

30 Posted by J Haag on 29 September 2010 | Permalink

I look at it this way. If you start with progressive enhancement, you have one set of source HTML files to maintain. You can build the stylesheets and features around this content core. If you start with a web app framework (e.g. Sencha Touch, iWebkit, etc.) then you have to develop two or more versions of your HTML content to support different mobile browsers that can't handle JQuery and CSS3. Then you are forced to deliver muliple versions using device detection such as http://detectmobilebrowsers.mobi for your older devices or browsers. This is not progressive enhancement, but at least your are trying to support as many users as possible. The point PPK didn't state, but inherently made is that in the long run you are going to spend more time maintaining multiple versions of your content if you go the web app framework + device/browser detection route rather than spending more upfront time planning and designing for progressive enhancement.