All posts by RichardB

2016 browser usage round-up – some encouraging signs

Roughly once a year, we blog about browser usage, based on visitors to our applications.

This year, I’ve taken logs since April 2015, run them through some one-off analysis code and plotted results using matplotlib. These stats cover all our servers worldwide. I’ve broken them down by continent and I’ve also split out results for pages that only admin users use. Since our customers are primarily government organisations, the numbers for admin pages mostly correspond to what government employees are using.

For those who only have time to look at one chart

This is the one chart that makes me most happy.

Legacy vs modern browsers for all apps

The red represents people using bad web browsers that make me unhappy: IE7 and IE8. The yellow portion of this chart represents people using somewhat nicer web browsers  – IE9 and IE10. The green portion of this chart represents people using high-quality, modern web browsers that are pleasant to develop software for – such as Firefox, Safari, Chrome, Edge and IE11.

This chart makes me happy because it shows that about 80% of people visiting our sites are using high-quality, modern web browsers, and that this number portion is going up. ♥

(NB: I’ve omitted February 2016 from this (and all the other monthly charts) because we’re only partway into February, so there isn’t enough data yet to draw valid conclusions for February.)

Worldwide browser usage

The first thing I’m interested in knowing is how often our sites were visited from desktop devices vs mobile devices. This chart gives the split for January and February 2016:

Desktop vs Mobile in 2016 for all apps

We’re still mostly seeing hits from desktop browsers – which is understandable, since a) responding to a consultation is something that can take a while and b) a good number of participants will be at in a workplace/at a desk when they respond. It’s exciting to see people are using our sites from mobile devices too, though.

The next thing I’m interested in knowing is: which browsers are our visitors on desktops using?

I’m glad to see usage of both Internet Explorer 7 and 8 shrinking over the course of the last nine months. Given that Internet Explorer 11 and Microsoft Edge (the two latest browsers from Microsoft) are growing by about the same amount as IE7 and IE8 are shrinking, it looks like people are upgrading all the way to the newest available browsers, which is great news.

On the bottom of the chart, you can see that Google Chrome is responsible for a large quantity of hits all by itself, so it looks like Google’s advertising for its browsers has been paying off.

The next thing I’d like to know is what browsers the mobile visitors are using:

This isn’t quite as pressing as the desktop browsers breakdown, because there isn’t such a disparity in technical quality between the different widely-used mobile web browsers. Apple’s Safari, Google Chrome, mobile Internet Explorer 11 and Android’s built-in browser are all technologically excellent and currently very easy to write good software for.

It’s interesting to see that we receive more hits from Apple’s Safari on iOS devices (iPhones, iPads and iPods) than from Android devices running Google Chrome or the Android Browser, despite the fact that Android devices are sold very widely at low price points. There isn’t any clear trend of market share change between the two.

What browsers are people in government using?

This chart shows what desktop browsers are being used in the admin sides of our products. (I’m only showing desktop browsers here because the people who use our apps as admins very nearly never use mobile browsers to do so.)

You can see that the variance in this chart is much higher than all the previous charts. The number of requests sampled for measuring admin usage is a small portion of the apps’ total usage, so there are accordingly fewer total hits.

While the chart is fairly chaotic at all points, I’m quite happy to see that there’s clear evidence here that many admin users have been upgraded away from IE7, 8 and 9 and on to IE11. I’m also very glad to see that the portion of admin users using IE10 has shrunk rather than grown in the last couple of months, since this indicates that peoples’ computers are being updated all the way to the very latest version of Internet Explorer available, rather than just bumped up to the next one.

Seeing Chrome and Firefox usage go up is also pleasant. It’s hard to claim a clear trend here because there is a lot of noise in this data, but it does look like a growing portion of admin users have had their IT department either allow them to use or provide them with one of these two Evergreen Browsers.

What browsers are people using in the United Kingdom?

In the United Kingdom, we see a slightly higher proportion of mobile device visits than elsewhere:

This chart shows the desktop browser breakdown in the United Kingdom:

Again, I’m glad to see IE7 and IE8 shrinking and that people are upgrading from them to Internet Explorer 11 and Microsoft Edge. This is pretty much identical to our worldwide chart.

Here’s the chart for mobile browser share in the UK:

This, too, is almost identical to the worldwide chart.

What browsers are people using in Australia?

We see less mobile uptake in Australia than in the UK, but more than in New Zealand.

Here’s a breakdown of the desktop browsers in Australia:

We were receiving a relatively large amount of traffic from a single location where people were still using Internet Explorer 6, which has slowly dwindled. There’s a distinct trend in the last couple of months for IE7 through IE10 to go away in favour of IE11, which is a good sign. I’m not quite sure why the “Other” line suddenly jumps up at the end…

Here’s a breakdown of the mobile browsers we see in Australia:

It looks like almost everyone in Australia who bought a smartphone or tablet went with an iPhone or iPad, and the fact that that market share grew in December is interesting. Perhaps a large number of people in Australia got shiny new Apple hardware for Christmas this year!?

What browsers are people using in New Zealand?

The vast majority of people who visit our sites in New Zealand are doing so using desktop browsers:

I won’t dive into those mobile users because there aren’t enough of them to draw good conclusions, but here’s the breakdown of the desktop usage:

This one is intriguing. It looks like a large number of the visitors to our sites in New Zealand started using IE10 in October (or we gained a large amount of traffic from a single group of people using IE10 in October) and then they were immediately upgraded, by December, to IE11 and other current-generation web browsers.

It looks like these numbers include a lot of people using corporate- or government-managed machines with active, diligent operations teams keeping them up to date.

Small print

These statistics are compiled from visits to all three of our products: Budget Simulator, Citizen Space and Dialogue.

For comparable browser usage statistics from previous years, see our posts from 2015, 2014 and 2013.

2015 Browser Support Roundup: 3 reasons you need to transition off IE7 and IE8 ASAP

Roughly once a year, we produce a roundup of browser usage based on Apache logs from our web applications: Citizen Space, Dialogue App and Budget Simulator. I recently spent a morning running (a modified version of) an open-source log analysis program called Visitors against our log files and now I have a few charts to share.

We especially wanted to see the numbers of people still using Internet Explorer (IE) 7 and Internet Explorer 8. Why? Because everyone should be hoping for the day when the number of people using IE7 and 8 is precisely ‘zero’.

Now, that might sound harsh. Maybe you really like your old version of IE. Or, more likely, certain systems you work on are dependent on it, or the upheaval of switching everyone’s computers over to a different browser is just too much to contemplate. But I’m afraid that sticking with old IE is a false economy and a bad move. If you’re an IE7 or 8 user, you really need to be gearing up to transition to a newer browser ASAP. Here’s why:

1. No more support

Microsoft have announced they will be dropping support for all but the latest-available versions of Internet Explorer for each version of Windows on the 12th of January 2016.

On Windows Vista, this means that only IE9 will be supported. On Windows 7, 8 and 8.1, only IE11 will be supported. There are no other supported desktop versions of Windows, Windows XP having been discontinued some time ago.

This leaves IE7 and IE8 as unsupported software. On the 13th of January 2016, Microsoft tech support will not help you out with issues on Internet Explorer 7 or Internet Explorer 8. Microsoft want people to use a modern IE; they ran a ‘kill IE6’ campaign and 7 & 8 now look to be headed in a similar direction.

2. Serious security vulnerabilities

Using unsupported browsers like this will expose you (and potentially citizens on whose behalf you’re working) to unnecessary risk of having sensitive information leaked or stolen.

If the next big, scary, security bug like Heartbleed or FREAK to be unveiled after the 13th of January 2016 turns out to affect Internet Explorer 7 or Internet Explorer 8, no fix or patch will be forthcoming.

Your choices will come down to: running a web browser that may permit your computer to be broken into or connections to be spied on at any time; ceasing to use a web browser at all; or switching to a different web browser in a hurry with no warning or time to test that the applications you need to use still work with the new browser.

If your organisation is currently deploying IE7 or IE8 to end users, you need to already be working on migrating to a web browser for which security support will exist on the 13th of January 2016 so that you can deploy before the official cut-off deadline.

3. Outdated browsers cost you (and everyone else) money

The harsh reality is that these browsers are deprecated, dreadfully faulty and have very poor support for modern web standards. They each include an enormous quantity of bugs that must be worked around by every developer trying to target them. And there’s no getting around the fact that this costs money.

The ecosystem of open-source and commercial libraries that most contemporary web applications rely on has been moving away from these antiquated browsers for some time now. For instance, Internet Explorer 9 is the oldest version of Internet Explorer which the jQuery project’s main 2.x branch now supports. This makes writing applications that support obsolete browsers more expensive by the day. Programmers who have to target obsolete browsers often can not reuse software components that their peers and middleware vendors publish. Not being able to reuse off-the-shelf software means that programmers must re-build, from scratch and at great expense, functionality that already exists elsewhere.

The ecosystem of commercial application vendors is moving away from antiquated browsers, too. Organisations that are stuck on obsolete Internet Explorer versions are cutting themselves off from the ability to use best-of-breed web applications. Commercial vendors have been deciding that it is not financially viable to provide support for the dinosaurs of the web. For instance, Google Apps dropped support for IE8 in 2012 and then dropped support for IE9 in 2013. Even Microsoft’s own Office 365 supports only versions 9 and later of Internet Explorer.

Old IE helps nobody

As I said, it really is in everyone’s interest to see an end of IE7 & 8 usage. Switching away from these old browsers will save you money, save you security nightmares and save you the frustration of unanswered requests for help.

Are we there yet?

So…to the statistics. Will we find that we’ve already made it to the promised land of a world without IE7 & 8?

How many people are using obsolete browsers in the UK?

When we look at all visits to our customers’ sites in the UK, we’re seeing a pleasing uptake of the so-called evergreen browsers such as Chrome and Firefox. “Evergreen” web browsers are those which always keep themselves up to date without bothering users. This is fantastic news for us and any organisation that makes web applications since we only need to worry about the latest stable version of each evergreen browser, rather than having to develop against all of the historical versions at once.

Browsers used by all site visitors in the United Kingdom

The fact that the two most widely-used versions of Internet Explorer are version 11 and version 9 is also good news. IE9 is the latest version of Internet Explorer that Microsoft has made available for Windows Vista, and IE11 is the latest version available for all newer versions of Windows. The fact that these versions are widely used by the public indicates that many users of Internet Explorer are getting their browser upgraded by Windows Update.

However, it seems (for the moment, at least) that the public sector has still got a way to go to catch up to the general public on this front.

How many people are using obsolete browsers in the UK government?

Here I’ve filtered things down to just visitors to management pages in our applications. These management pages are used by our customers in the UK, who are mostly staff in the UK public sector. Of course, the situation isn’t precisely comparable – here, browser usage will mostly be determined by IT departments who have to decide how to transition hundreds of users at a time, rather than individuals who can simply click ‘update now’.

Browsers used by government users in the United Kingdom

These numbers are…not great. There’s a little good news here: the second single most-used browser is IE9, which is okay. Not the greatest browser ever, but okay. There’s also a little bit of relatively-early IE11 adoption. That’s good to see.

But mostly, this is a troubling picture. About a quarter of our UK government users are still using Internet Explorer 8. And 3.1% of our UK government users are still on Internet Explorer 7.

While 3.1% doesn’t sound like much, it’s enough that developers like us still have to write workarounds for IE7’s numerous bugs and deficiencies into our software on a regular basis, which means it’s costing these clients money. And, more than that, this chart represents a lot of people who’ve not got a whole lot of time to get switched over to a supported, up-to-date browser.

How many people are using obsolete browsers in Australia?

Evergreen browsers are not so common in this case. People using Internet Explorer seem to mostly be using the newer versions, but not so often the versions that I would have expected Windows Update to give them. (Not sure why this would be – possibly a consequence of Australia’s high bandwidth prices?)

Browsers used by all site visitors in Australia

There’s a very surprising spike of IE6 users on our sites from Australia. When I’ve dug into this, it turns out to be all coming from a single institution and going to a single site, so it’s not really representative of the population of Australia at large. I would have expected these to have stopped when we (and pretty much every other organisation running services on the internet) switched off SSLv3 support because of the POODLE vulnerability, but presumably this group has switched on TLS 1.0 support manually as a workaround.

Browsers the government use in Australia

Happily, the situation amongst government customers in Australia is less worrying than in the UK. Internet Explorer 8 is a small minority here and Internet Explorer 7 is gone entirely. The browsers in use tend to be far more modern on the whole. (It looks like IT departments haven’t started to pick up IE11 just yet.)

Browsers used by all site visitors in Australia

This is pretty much exactly the picture that I would expect to see from a population of IT departments that are determinedly working their way through an update schedule, being conservative because of the need to take the time to test everything before moving to on to the next version.

The 1.1% of Internet Explorer 6 users is due to the same one holdout. I’m not particularly worried about it: since they’re still using IE6 in the year 2015, I expect they’re probably already quite used to the fact that some things on the internet just mysteriously don’t work.

Conclusion

For everyone’s sake, we’re intending to drop Internet Explorer 7 to level 3 in our browser support policy as soon as we possibly can. To this end, we plan to start working with our customers to help them move off IE7 as soon as possible.

With Microsoft ending their support for it, IE8 needs to go the same way, too. Because 25% of our UK customer base currently use IE8, that’s a not-insignificant challenge. But it does need to be overcome at some point soon.

Also, we may start dishing out cookies/champagne/pictures of cats* to any IT department that pushes all the way up to IE11. The newer versions of Internet Explorer cause significantly fewer compatibility problems and run much faster than their coal-fired cousins, to boot. The lesser-spotted win-win!

*probably not really. Maybe the latter. Maybe.

48GB of DDR3 memory, in individual 8GB sticks.

I’ve just added more RAM to our testing machines

Here at Delib, we use a little XenServer 6 cluster for continuous integration. We have programs that act like users of each of our online applications and put them through their paces, which we write using a framework called Selenium. We use these automated tests to demonstrate that the entire software stack under each of our applications all works together, before we roll out changes to any individual part of it. These are full end-to-end tests, demonstrating all of the layers from our application code through to the operating system that they are running on.

The XenServer cluster is a small group of physical computers that work together to host a larger group of virtual computers. This makes running our tests on different versions of operating systems much easier than if we had installed them directly onto physical computers, because we can do things like clone virtual machines or roll them back to a specific point in the past with just a few clicks (or commands).

Virtualisation is an enormous help for testing how our software interacts with its operating systems, but it has one very small downside: you end up needing a lot of RAM to run a large number of virtual machines. Each physical computer needs to have as much RAM in it as all of the virtual machines that it is hosting, added together. We’re not big fans of spending our lives manually shuffling things around to fit within limited resources when they’re cheap, so the first alteration that we are making to our QA cluster here is to double the amount of memory in it.

48GB of DDR3 memory, in individual 8GB sticks.
48GB of DDR3 memory, in individual 8GB sticks.

This was a stack of sticks of plastic and silicon to the immediate left of my desk, when it had just arrived this morning from Crucial. I’ve just installed them all into the physical computers, and we should be up and running and making use of this soon. This upgrade should, as soon as we’ve reconfigured everything to take advantage of the extra memory, make the time that it takes to complete one of our test runs at least a little shorter, which makes me happier.

Crowds keep your sysadmins awake

We host a number of clients’ websites for them on our own servers. It’s convenient and it saves a lot of deployment headaches, since we know our machines like the backs of our hands.

Yesterday was one of those days.
Server load. Flat until yesterday, when a big spike appears. Another spike appears today.

This chart shows the load average for one of our webservers. It’s – roughly – a measure of how busy the machine is at any given time. As you can see, one of the sites being hosted by that server suddenly started getting hammered around lunchtime yesterday. All very exciting.

If that weren’t excitement enough, the load surfaced a concurrency issue in the site, which had lain dormant until that point. It wasn’t causing data corruption (thankfully – professional ethics would require that I commit seppuku right now if it had), but it was slowing the site down and causing exceptions to get returned in full sight of innocent users.

Cue an immediate dive into the sources to find out what was wrong. By seven o’clock we’d tracked down the issue and designed and tested a viable fix, which both sped the site up and made it possible to spread the load to a second server. Too late for the initial flood, unfortunately, but in plenty of time to help the second wave of visitors which started up this morning.

The final numbers aren’t in yet, but it appears that we’re seeing even more responses today than we did yesterday – and the servers are holding up far more smoothly, with much better response times. It’s stressful when the sky falls in on your head all at once. You feel more alive afterwards, though, for having survived it.

Incidentally, the above graph was generated by Munin, a truly beautiful piece of software. It’s like a heart monitor for servers.