Hello, we’re looking for an experienced web developer to join us in Bristol, UK.
Author: Jess Henderson
Over the past couple of months we’ve been focusing our development efforts on improving our hosting and associated product environment via an appropriately titled ‘production infrastructure sprint’.
Although this doesn’t sound as exciting as adding features to our products, it’s a vital part of Delib’s service to our customers, as it helps to ensure that we continue to meet our uptime and performance commitments. Here’s a little overview of what we’ve been up to.
What we’ve been doing
Up until recently we hosted all our customer instances on large multi-tenancy servers. ‘Multi-tenancy’ means that several Delib customer sites run side-by-side on the same machines, although all their data is stored in separate databases. These servers live in secure data centres, physically located in the same territory as the customers they serve. The data centres are responsible for providing Internet connectivity for the production servers.
Over the past few weeks, we’ve been moving customers slowly and carefully in batches from our current hosting providers to new providers who can better meet our service and uptime requirements.
Why we’re changing our hosting infrastructure
The reasons for migrating to new hosting providers are threefold:
1. Improvements in availability
In the UK, we are moving all our hosting to Rackspace, the market leader in cloud hosting, which offers a 100% uptime guarantee. Since our uptime is necessarily bounded by that of our upstream providers, it’s important to use the best that we can get. We are researching the best providers in other territories, to ensure that we continue to meet and exceed our commitments for all our customers.
We use a server monitoring service that notifies our account managers and developers by text message whenever a customer’s instance is unavailable for any reason (even if it’s in the middle of the night) so we’re all keen to ensure that these improvements pay off as soon as possible!
2. More hosting options for customers
After migration, every Citizen Space and Budget Simulator instance will live on its own virtual machine. This allows us to offer different hosting packages for different usage patterns: we can now tailor the system specification (RAM, disk space, number of processors) to the requirements of the customer. Furthermore, large spikes in one customer’s traffic can no longer adversely affect the response times of other customers’ sites.
Dialogue App instances will continue to run on a multi-tenancy setup by default. However, customers with heavy usage requirements (eg large, heavily-publicised national dialogues), will have the option to host their Dialogue App instance on its own machine.
3. Consistent configurations and automation
As our number of customers grows, our developers have been spending more and more time engaged in administrative tasks such as rolling out new instances and upgrading existing customers. While this is vital to the business and to our customers, developers would much prefer to spend their time developing new features and fixing bugs in the products.
At the same time as moving customers to the new hosting infrastructure, we’ve been improving our suite of developer tools so that more of the day-to-day tasks can be done without developer intervention.
For our customers, this means that planned maintenance should soon be able to take place, as far as possible, outside working hours. It also means that developers will have more time to spend on improving our products, resulting in a better user experience for our customers and end users.
Find out more
If you are interested in finding out more about the improvements we are making please feel free to get in touch with either Louise or Rowena.
As in previous years, I took the last month’s logs from our web servers, and ran them through an open source analysis package called Visitors. In contrast to previous years, these stats now cover all our servers worldwide – not just in the UK.
Since visits to our Australian servers now account for a notable proportion of our apps’ traffic, this year I have also included a separate breakdown just for our Australian stats.
Worldwide browser usage
I’ve generated two different reports: one for our apps’ management pages (i.e. pages that can only be accessed by logged-in admin users), and one that includes public-facing pages as well. Here are the figures for our admin users:
It’s a relief to see that IE6 has not made a reappearance in the past year, and also that IE7 usage has dropped from 15.3% in April 2013 to less than 10% a year later.
We currently provide Level 2 support for IE7, which means that all functionality and navigation must work, and all content must be readable in IE7. However, we’d much prefer to spend our time developing new features that benefit everyone, rather than fixing bugs that only appear in this eight-year-old browser. Over the next few months we hope to encourage our customers to move to more modern browsers so that we can drop admin support for IE7.
The second chart shows visits to all Citizen Space and Dialogue App pages, including visits from members of the public:
As you would expect, this shows a much wider range of web browsers, including a few visits from IE6. By the way, I’ve excluded visits from crawlers, bots, RSS feed readers and other things that aren’t conventional human-controlled web browsers.
It’s interesting to compare these stats with last year’s numbers. In particular, the most popular browser is now Google Chrome, which has overtaken IE8 – last year’s frontrunner. However, usage of IE8 still remains far higher amongst our users than you’d predict based on global figures from StatCounter.
Browser usage in Australia
Although the majority of traffic to Delib’s apps is served from our UK servers, traffic to our Australian servers now constitutes a notable proportion of visitors (8.5%, based on last month’s stats).
IE8 is still the most popular browser amongst our Australian admin users, but its lead is far less marked than in the UK.
When we consider all pages, not just admin pages, it’s interesting to see that the most popular browser used to visit our Australian sites is Firefox. This is in contrast to StatCounter’s figures for Australian Browser usage, which pegs Chrome usage at more than double that of Firefox. The numbers here are fairly small (16K visits a month) so this could have been skewed by one particularly Firefox-heavy demographic of survey respondents, for example.
* Visits to our newest app, Budget Simulator, are not included here, as a lot of the administration is done by Delib account managers, and our choice of browser would skew the statistics quite a lot. Also, the visitor profile of Budget Simulator leans quite heavily towards mobile users. I feel another blog post coming on!
** For the purposes of these statistics, a ‘visit’ comprises all the requests from a given IP address and useragent on a given day.
Maybe I’m just naturally nosey, but I’m fascinated by other companies’ support staff. When I’m talking to my bank, my phone company or whatever, I always wonder what it’s like for the person on the other end. What is their desk like? Do they like their job? How many other people do they work with? So I thought some of you might like to hear five secrets from support@delib.
1. We’re not a vast, outsourced call centre
If you are an existing Delib customer, you will know that you have your own account manager. You can phone up and ask for them by name, and they will know who you are, which organisation you work for, and that you’ve just got back from holiday in France.
But did you know that the personal service continues when you email firstname.lastname@example.org with your query? The first person who sees your query is likely to be your account manager. Often, they will be able to answer your question very quickly. If not, they’ll usually drop you a line to say that they’ve passed the question on to a developer who can look into it in more detail. The useful thing about emailing the support address is that it pops up in our support ticket system and I (the support developer) can see all the info you’ve provided, and the account manager’s response, straight away.
I might never speak to you on the phone, but I almost certainly recognise your name too, and know which organisation you work for. I might even know you went to France last week, if I overheard your account manager chatting to you about it. You see, in the UK office we all work within earshot (and potentially nerf-shot) of each other. When we answer your question, we sign our names, so hopefully you’ll get to know us too.
At any time, there will be one or maybe two developers answering support queries, along with one or two testers to check our work. That means it’s very likely that your entire query will be answered by the same developer. If you ask a couple of questions in the same week, you will probably get the same developer each time.
2. All Delib developers take turns on support duty
We think it’s vital that the people who develop Delib’s apps are the same people who support them. This insight works two ways: As one of the people who designed, coded and deployed the feature you’re having trouble with, I am the best placed to help you use it, or to track down the bug (sorry!) and work out how to fix it.
Conversely, having talked to you – a real user of the system – I have a much better insight into how you use it every day in your job. This really helps us when prioritising new features and understanding how we can improve the usability of our apps. In some companies, developers see their users as an anonymous “them”. When we plan the work for each release, we always like to use real names. “Bob from Westonshire Council had trouble using feature X last week. Is there anything we can do to make it easier to use?”
At the moment, we have a fortnightly rotation and hand over the support baton on a Thursday morning, so if you ask a question on Wednesday you may get answers from two different developers. Don’t worry though, we always have a handover chat with the person taking over, so they know which issues are open and how far we’ve got with resolving them.
3. We love our Knowledge Base
If you’re not sure how to use one of the features of our apps, the first place to look is in our Knowledge Base. If you don’t already know about it, I recommend you bookmark it. It’s a huge, searchable repository of user guides, FAQs, tips and tricks. If you’re using Citizen Space and are logged into your site as an admin, you can also get to the Knowledge Base via the ‘support’ link at the top of every page.
However, we don’t only recommend the Knowledge Base to our customers – we use it for all our internal documentation as well (obviously you have to be a Delib employee to see that stuff). One of the things that support developers do when we’re not answering customer queries is to keep the documentation up to date, and write handy how-to guides for our future selves, so that the next time a particular question comes up, we can answer it even more quickly.
4. We like it when people say ‘thank you’
Sometimes when I contact other companies’ customer support and they solve my problem, I wonder, “Should I drop them a line to say ‘thanks’, or is that just using even more of the support person’s valuable time?” Well I still don’t know about other companies, but here at Delib, we always appreciate a quick “Thanks, that did the trick.”
Of course it makes us smile to know you’re happy, but also it means we know we can close the support ticket without offending you, or worrying that we’ve left any of your questions unanswered.
5. This is what my desk looks like
- This flag means I am on support duty (we cheesily refer to the role as ‘Support Superstar’). The flag is attached to the side of my monitor with velcro. This morning I will give the flag to Alan, who will take over from me. Support Superstar doesn’t only do customer support; they’re also the go-to developer for any sales or operational questions that need a technical answer, so this flag makes it obvious who to ask.
- Kanban board. This is one of the ways we visualise the tasks we have in progress. It helps keep our turnaround times low on things like new customer rollouts.
- Consultants sit here.
- Account managers sit here (see, I told you they were nearby).
- Samuel the spider plant. He is a bit under the weather at the moment but I am expecting a full recovery.
- This blog post (paradoxically)
- Other developers, testers and sysadmin sit just there.
So, I’m signing off from support duty for now, but if you write to email@example.com in a few weeks’ time, there’s every chance that I might be in touch.
In what seems to have become a roughly annual tradition, I’ve just done a survey of the browser usage amongst our UK Citizen Space users. As in previous years, I took the last month’s logs from our UK-based web servers, and ran them through an open source analysis package called Visitors.
I’ve generated two different reports: one for Citizen Space’s management pages (i.e. pages that can only be accessed by logged-in admin users), and one that includes public-facing pages as well. Here are the figures for our admin users:
This first chart shows something wonderful: last month, nobody used Internet Explorer 6 to administer their UK Citizen Space site.
For those who don’t know, Microsoft’s venerable Internet Explorer version 6 (AKA IE6) was released back in 2001, and is notorious for rendering web pages very differently from modern, standards-compliant browsers. To support IE6, web developers have to spend a lot of time writing workarounds to make web pages display correctly in all browsers. Of course, this increases the cost of product development without benefitting those users who don’t use IE6. Last year Delib decided to stop actively supporting IE6 in our apps, and these latest figures clearly vindicate that decision.
The second chart shows visits to all Citizen Space pages, including visits from members of the public:
As you would expect, this shows a much wider range of web browsers, including a few visits from IE6. The ‘Unknown’ useragents are mostly made up of crawlers, bots, RSS feed readers and other things that aren’t conventional human-controlled web browsers.
It is interesting to compare our statistics with current worldwide browser usage. At the time of writing, Google Chrome and Firefox are the most widely used browsers, followed by IE9 and then IE8. In contrast, our stats show that IE8 is by far the most common browser used to access Citizen Space.
My theory is that most of the activity (both administration and participation) on our Citizen Space sites takes place during the working day, where people are less likely to have a choice about the software they use, and are more likely to be stuck with a slightly out-of-date standard-issue browser. Incidentally, Edd at GDS made some interesting observations about this same phenomenon last month.
* For the purposes of these statistics, a ‘visit’ comprises all the requests from a given IP address and useragent on a given day.
We’ve always been keen to encourage Citizen Space to play nicely with other friendly applications on the web. For example, you can:
- Link your consultation records to awesome sites such as EventBrite, so that consultations you’re running elsewhere on the web (or even offline) are still listed and searchable through Citizen Space.
- Embed rich media such as maps and videos from third-party services.
- Couple Citizen Space with third-party mailing tools, like MailChimp, to stay in touch with people who want to be kept updated.
We love these features and so do our clients, and they’re definitely not going away. However, we really want to provide more flexible ways to open up the data held within Citizen Space, and that’s why we’re so excited about our new API. “API” is short for “Application Programming Interface”. This is a set of documented procedures that third-party systems can use to communicate with an application.
API integration with Citizen Space requires more technical know-how than the features described above, so it’s something for your web or IT team to get their teeth into. The important thing is that the API doesn’t care what platform or language you use for your existing infrastructure. Citizen Space is written in Python and uses the Zope application framework, but other organisations may use WordPress, .net or Drupal for their websites, which are written in other languages such as PHP. Using the API, your tech team can use the languages and platforms that are familiar to them, but still access and manipulate the data held in Citizen Space.
Version 1 of Citizen Space’s API (included with April’s release of Citizen Space) is all about getting public consultation details out of the system so that you can include them in your own site. We’re providing ways to embed search forms, search results, consultation listings and consultation overview pages into your site.
But that’s only the first step. Version 1 of the API provides all of its output in simple, cleanly formatted HTML. This is quick and easy to work with, but we’d like to build on it by providing other output formats such as JSON and XML. We also want to publish more plugins for other systems, code snippets and example projects, because even we haven’t investigated all the exciting opportunities that this API provides.
Looking further ahead, we’re going to think about how the API could be extended to include data from Citizen Space’s admin areas (with suitable authentication of course). For example, would you like to check your organisation’s intranet dashboard and see how many responses your QuickConsult consultations have had? We also want to consider how the API could be used to get information into Citizen Space.
Is there information in Citizen Space that you’d particularly like to access from elsewhere? Are there applications you’d like Citizen Space to play more nicely with? Are you using our API already? We’d love to hear from you.
Almost a year ago, we published some interesting browser statistics based on the logs from one of our Central Government Citizen Space servers. We ran the logs through a piece of open source analysis software called Visitors, and this gave us an anonymous breakdown of all visits to Citizen Space, showing the browsers and versions that were used. We looked at the statistics for all pages, and compared them with the stats for pages only accessible to admin users. The results were insightful but rather scary: Internet Explorer 6 accounted for more than 1 in 3 visits by our Central Government admin users.
For those who don’t know, Internet Explorer version 6 (lovingly known as IE6) is a web browser that Microsoft released over a decade ago. Because it renders web pages differently (in some cases dramatically differently) from more modern browsers, web developers spend a great deal of time creating workarounds so that IE6 users can still access our websites. Of course, this increases the cost of product development without necessarily offering any benefit to the majority of web users who don’t use IE6. Coupled with the fact that IE6 now only receives limited support from Microsoft, almost everyone is in agreement that this ancient and decrepit browser must be phased out.
This morning, we had a comment on last year’s post from a reader who was interested in how Citizen Space’s browser stats had changed. Thank you Perry – you reminded me that I’d been meaning to re-do this analysis soon. So here are the graphs comparing the numbers 11 months ago with where we are today:
Central Government Citizen Space – all users (admin and public)
These statistics roughly follow the browser trends of the general internet population*, with IE8 and 9 increasing in popularity while the older IE versions decrease as expected. Pleasingly, IE6 usage has roughly halved since last May.
Firefox, Chrome and Safari have gained more of a stronghold in the past year, although interestingly, Internet Explorer as a whole has retained a far larger share of Citizen Space users than worldwide browser usage statistics* would predict.
Central Government Citizen Space instances – admin pages only
When looking at the statistics for our admin users, the most exciting thing is that usage of IE6 has crashed by 90% – from 35% down to 2.4% of visits. This is a great relief to us, and shows the huge effort that has taken place in government IT departments to upgrade users away from this insecure, ill-supported browser.
It’s worth noting that overall, usage of Internet Explorer among our Central Government users is more than 90%, compared to 34% worldwide*.
The interesting question is what levels of support to provide for different browser capabilities. We currently provide Level 2 support for IE6, which means that all content must be readable and navigable, but differences in styling and layout may exist. This works OK for our products at the moment, but as web users come to expect a richer and more fluid experience, the likes of IE6 are going to lag further and further behind. How small does the percentage of IE6 users need to be before we can stop supporting it at all?
As always, I’d love to hear your views.
*Worldwide browser statistics from statcounter.com.
I’ve recently spent some time playing with the idea of associating Citizen Space consultations with a geographic location.
We already do this to some extent. Consultations can be associated with one or more local wards or areas, so that visitors to Citizen Space can enter their postcode and see a list of consultations related to the area they live in. This is great for helping people find out what’s going on near them, but I’ve been itching to take this geographic information further. In particular:
1) It would be nice to show this information visually, for example on a map. This is particularly useful if a policy relates to a specific object (eg a building, road junction or monument), or an area that doesn’t correspond to pre-defined ward boundaries (eg a bus route, catchment area or park)
2) We love Open Data. If we’re storing data about a consultation, it’s always nice to make it available in a standard format so that other websites and applications can make use of it. Currently we’re not sharing any geographic information with the rest of the world.
So how could this work? A sneak peak of Citizen Space pre-release features:
I’ve added some extra fields so that you can enter longitude and latitude coordinates when setting up a consultation. Alternatively, if you want to specify a shape (such as the footprint of a building), a line (such as a bus route), or an outline (such as the boundaries of a catchment area), you can upload a KML file to your consultation. If you have a GIS team or supplier, they should be able to provide this data in the right format.
When visitors view the consultation record, they’ll see an interactive map marked with the information you specified:
Map showing a single point
Map showing the outline of an area
While it’s instructive to show a map of the consultation area to Citizen Space visitors, this location data becomes even more interesting if we let third-party users and developers get their hands on it. In my prototype code, I’ve included three ways of sharing the data:
Method 1: KML
If you look at those screenshots you’ll see that there’s a link to a KML feed underneath the map. If you uploaded your own KML file, it will link straight to that file. If you entered coordinates, it will generate a new KML file just containing the point you specified.
KML is a very simple way to share mapping data with other online applications. It can be done using sites such Google Maps and Microsoft Virtual Earth without any programming knowledge. Lets say you’re consulting about a proposed cycle path, and have uploaded a KML file plotting the complete route of the path. A local parents’ group might use the KML data to overlay your route over their existing map of schools, parks and youth centres, to show how child safety could be improved by the construction of the path.
Method 2: GeoRSS
Citizen Space currently provides RSS feeds that let users subscribe to all the latest consultations, or consultations that meet certain search criteria. These feeds are also used by third-party sites to embed up-to-date consultation information, or to aggregate consultations from multiple sources into a single list.
If we have geographic coordinates associated with a consultation, it’s very easy to publish this information as part of the RSS feed, using the emerging GeoRSS standard. Apps that understand and are interested in the location data can use it in much the same way as the KML data above. Apps and users who are not interested in location data won’t see any difference to their feeds.
Method 3: RDFa
It turns out the RDFa specification includes the ability to link your document to a place, using the “based_near” tag. If you’ve entered latitude and longitude information when setting up your consultation, this extra bit of RDFa will be published along with the rest of the consultation record. Sadly I haven’t found many examples of applications that currently use the based_near feature of RDFa, but here are a couple of ideas I’ve been thinking about:
- A visual version of the Citizen Space aggregator, that can display consultations from many different sources on the same map.
- A mobile app that alerts you when you’re near an object or place that’s being consulted on.
This new code isn’t in Citizen Space at the time of publishing this post, but will likely be included in a future release.
If anyone else has ideas for making use of location-based consultation data, please drop me a line. The more good ideas we get, the more likely it is that this feature will make it into a future Citizen Space release.
I’d also be keen to hear of formats we could use to share the data other than KML, GeoRSS or RDFa. Has anyone used GeoJSON for example?
We spend a lot of development time ensuring that our software is usable across all the commonly used web browsers. A disproportionate amount of this time is spent ensuring compliance with Microsoft’s archaic Internet Explorer 6 (IE6), which is 10 years old this year.
From time to time, we wonder whether we could provide better value for money by dropping support for IE6 and making more use of the facilities provided by newer browsers; a decision that has already been taken by the likes of YouTube, Facebook and Google.
Steph Gray recently blogged about whether Alphagov should have dropped IE6 support. Steph critiqued this decision, pointing out that a lot of civil servants still use IE6. We thought it might be useful to share the breakdown of browser usage by our civil service clients.
What do the numbers show?
Here is a chart of browser usage on the admin pages of our Citizen Space app, for the servers used by our Central Government clients:
And here is a graph of browser usage across all Citizen Space pages (admin and public-facing) on the same servers:
Here you can see that IE6 is used by more than a third of our Citizen Space administrators, but only about a tenth of the total visitors. At the moment, there is clearly a need to continue supporting IE6 for our clients, but it does seem a shame when this investment could be put towards improving the user experience of the site’s end users.
I could write more words about these differences, but here’s another chart that tells the story pretty clearly:
We’ll be keeping an eye on these figures to see how they change with time, and we’d also be interested to know how they compare with data from other sites aimed at government clients around the world. Does anyone else have any data they’d like to share?
The science bit:
- Data was taken from May 2011’s Apache access logs from one of our Citizen Space servers. Data is anonymised.
- We parsed the logs using the open-source Visitors software (which we modified to include the most recent versions of IE). The software can be downloaded from http://www.hping.org/visitors/doc.html.
- Statistics are based on visits rather than pageviews, where a visit is all requests for a given useragent and IP address on one day.
- We excluded any visits from our own IP address and from our server monitoring services.
- We did not exclude crawlers and other bots, which probably account for the majority of the ‘unknown’ useragents in the second chart.