Category Archives: Code & Tech Stuff

Clever stuff we’re developing: code, prototypes, tech demos.

We’re hiring! Developer role in Bristol, UK

Hello, we’re looking for a Developer to join Delib HQ in Bristol.  We think we have a pretty good environment in which to write software. We have a big airy studio in a listed building in the city centre, and we’re a small enough company that everyone knows everyone else. It’s not perfect; there are never enough hours in the day, but we care, we say thanks, and we go out for lunches and drink together after work and I reckon that counts for a lot.

Typically we work well with people who’ve got a Computer Science degree and have been coding since at least their early teens. YMMV. We prefer people who can communicate with humans as well as computers.

We need to get some web app, support and operations stuff done. All developers do a bit of everything:

  • For frontend development we generally use XHTML, Less/CSS and JQuery/Javascript. We have to support a wide range of browsers including mobiles and tablets.
  • We generally use Python for backend development. Generally Python doesn’t suck. We work with python frameworks including Pyramid, Zope and Plone. You don’t need to have used these – or even have used Python – but experience with a web framework would be good.
  • Most of our app data is stored in object databases (ZODB) but from time to time we also use various flavours of SQL.
  • We have lots of devops things to do, including deployment automation for servers around the world. We use Ansible for this, along with a bunch of our own scripts. Again, you don’t need to have used Ansible, but it would be best if you’re not (too) scared of SSHing into linux servers, grepping logs and tweaking apache configs.
  • All our application code, automation scripts and configuration are version controlled using git, as is most of our test data. We all need to be able to modify, build and run each other’s code, so these days we’re pretty hot on documenting things too.
  • All developers take rotating fortnightly shifts as Developer on Support, which means we help our customers and account managers with technical issues via our online ticketing system, help sales people with quoting and tendering, and are generally available to answer questions without being excessively grumpy. This is actually really important – it means that developers get to see how the stuff we’ve built really affects our customers’ lives, and customers love getting a reply directly from the person who can fix their problem.
  • Unfortunately being on Support does also mean being on call. But calls/texts outside office hours are infrequent and if you do get called you get paid for it. Oh and don’t panic – you don’t get calls directly from customers.

These days we’re pretty good at using agile development processes like Scrum and Kanban. We also have grown-up things like continuous integration and Aeron chairs (or sofas to work on if that’s more your style). Bring your own laptop or we can supply one – you’ll get a decent quality Macbook Pro.

Hours and Salary

Could be a full-time, part-time or freelance scenario (but we’re not up for paying extortionate day rates to freelancers I’m afraid). Currently all Delib’s developers are part-time, with the option for scale-up days each month. We find that this arrangement suits our work/life/childcare/hangover requirements perfectly.

We’re offering £30-40k pro-rata depending on experience.

Contact details

Sound interesting? Send a cover letter and your CV to Jayne@delib.net. We don’t place too much faith in CVs, the covering letter is really what we look at. If we like the look of yours we’ll get you in for a standard hiring interview.

We follow the HMG Baseline Personnel Security Standard and you will therefore need to satisfy basic eligibility criteria/certain conditions of employment (e.g. nationality rules/right to work); and provide appropriate documentation to verify ID, nationality, employment and/or academic history, criminal record (unspent convictions only).

No applications will be accepted via recruitment companies.

Cheers,

Andy (Director) and Jess (Developer)

Digital Hero – Fran Bennett

FranBIt’s that time again; another installment of Digital Heroes, a quick chat with people doing interesting digital work, in and around government. This time I want to introduce you to someone grappling with the challenges of big, <sometimes open> data, a subject we haven’t covered yet. I’ve got to admit I struggle with the whole open data thing, after attending one too many conferences where it’s been discussed to death. It’s not that I don’t believe in its potential power, it’s just that I sometimes question why we’re still talking about how one can publish it, the problems of common file formats etc, etc, etc, and instead moved the conversation onto – ‘what useful things shall we do with it’? The prevailing answer seems to centre on, in my opinion, a slightly naive idea that somehow people will build apps and business models from it, which I don’t think is going to happen any time soon, outside of the odd isolated example.

It was therefore with interest that I learned about the work Fran and her co-founder Bruce Durling, have been doing, with their company Mastodon C, processing and using the data for public gain. The project which has deservedly got the most amount of press, is one that analysed vast amounts of prescription data, looking at disparities in prescriptions of licensed and non-licensed cardiovascular medication, which identified how the NHS could saves 100’s of millions of pounds – worth getting out of bed for.

Let’s see what Fran has to say for herself.

1.  What’s your name and where are you from?
My name’s Francine Bennett. I grew up in Norfolk and on the Welsh borders, and I now live in central hipsterville, Dalston in London.

2.  What do you do for a living?
I run Mastodon C. We’re a tech company (startup? I’m not sure how long we keep that label…) which builds and manages custom cloud-based big data systems, using open source technology. We work in particular with government, health, and energy data. It’s a lot of fun.

3.  Favourite band and/ or artist?
AARGH REALLY HARD QUESTION. I’m going to go with Tom Waits, but also Super Furry Animals.

4.  Android or iPhone?
Definitely Android. I prefer being able to tinker with things.

5.  PC or Mac?
Mac (even given comment above). I love my Macbook Air, it comes everywhere with me.

6.  Creature of habit or maverick thinker?
I’m a creature of habit, as a way to stay sane. We work with cutting-edge technologies, and are often trying to solve hard or new problems in innovative ways, plus there’s the general unpredictability of running a startup, so routine and habit are really important to keeping things rolling.

7.  Your house is on fire, what do you save?
Just the people in it. The stuff can burn if it needs to.

8.  Biscuits – dunk or leave unsullied?
Dunk, quickly. There is nothing more sad than a disintegrated biscuit, but the risk is worth the reward.

9.  Best project you’ve worked on at Google/ now/ whenever and why?
The best project is always the next one! I’m really excited right now about the new Embed system which is heading for launch (http://www.getembed.com) – this is a system we run for Energy Savings Trust, using Cassandra technology, which collects and analyses hundreds of millions of datapoints at high frequencies, and lets them understand building and energy efficiency performance in a way that should make a real difference to UK-wide housing decisions. I’m excited because we’ve managed to turbocharge the technology to deal with huge quantities and speeds of data without falling over, but also because the end result is an important one.

10.  Where do you hope Mastodon C will be in 10 years in terms of wider
digital democracy? Opportunities and pitfalls.
We are doing a lot of prototype work at the moment on Future Cities – building systems that make sense of city data, to help city leaders make operational and strategic decisions better. I think there’s a huge additional opportunity in this area to use this data and these
visualisations to explain how and why policies are made, and to help people participate in that democracy.

Doing that is going to be tough, though – the technology is there already, but there’s a big culture change required in order to have those conversations in an open way.

11.  Best gov site you’ve seen and why? Other than GOV.UK.
I just this week came across g0v.tw, run by a civic group which remixes Taiwanese government websites to make them more transparent and more useful. They’ve explained more at http://g0v.asia/tw/. I think their approach is really constructive and impressive.

So there you have it, another fascinating installment of Digital Heroes; what a journey we’ve been on. If you want to talk about harnessing the power of all that data you may or may not have, you can holler at Fran here. Failing that, she does the Twitter thing as well as anyone else.

Until next time.

2014 Browser Usage roundup: Will IE7 soon go the way of IE6?

Roughly once a year since 2011 I’ve been doing a roundup of browser usage, based on visitors to two of our apps: Citizen Space and Dialogue App*.

As in previous years, I took the last month’s logs from our web servers, and ran them through an open source analysis package called Visitors.  In contrast to previous years, these stats now cover all our servers worldwide – not just in the UK.

Since visits to our Australian servers now account for a notable proportion of our apps’ traffic, this year I have also included a separate breakdown just for our Australian stats.

Worldwide browser usage

I’ve generated two different reports: one for our apps’ management pages (i.e. pages that can only be accessed by logged-in admin users), and one that includes public-facing pages as well. Here are the figures for our admin users:

March 2014 admin visits by browser: IE8: 41.4%; Chrome: 16.5%; IE9: 13.4%; Firefox: 12.4%; IE7: 9.5%; Safari: 3.7%; IE10: 2.1%; IE11: 1.0% Visits to admin pages by browser, March 2014**

It’s a relief to see that IE6 has not made a reappearance in the past year, and also that IE7 usage has dropped from 15.3% in April 2013 to less than 10% a year later.

We currently provide Level 2 support for IE7, which means that all functionality and navigation must work, and all content must be readable in IE7.  However, we’d much prefer to spend our time developing new features that benefit everyone, rather than fixing bugs that only appear in this eight-year-old browser.  Over the next few months we hope to encourage our customers to move to more modern browsers so that we can drop admin support for IE7.

The second chart shows visits to all Citizen Space and Dialogue App pages, including visits from members of the public:

March 2014 visits by browser: Chrome: 21.1%; IE8: 17.4%; Firefox: 14.8%; Safari: 14.0%; IE11: 5.1%; IE7: 4.8%; IE9: 4.3%; Opera: 4.2%; IE10: 4.0%; IE6: 3.9%; Other Mozilla-based: 3.6% Visits to all pages by browser, March 2014**

As you would expect, this shows a much wider range of web browsers, including a few visits from IE6. By the way, I’ve excluded visits from crawlers, bots, RSS feed readers and other things that aren’t conventional human-controlled web browsers.

It’s interesting to compare these stats with last year’s numbers.  In particular, the most popular browser is now Google Chrome, which has overtaken IE8 – last year’s frontrunner.  However, usage of IE8 still remains far higher amongst our users than you’d predict based on global figures from StatCounter.

Browser usage in Australia

Although the majority of traffic to Delib’s apps is served from our UK servers, traffic to our Australian servers now constitutes a notable proportion of visitors (8.5%, based on last month’s stats).

March 2014 australian admin visits by browser: IE8: 28.7%; IE9: 25.9%; Chrome: 23.5%; IE10: 8.0%; Firefox: 7.6%; Safari: 4.8%; IE11: 1.2%; IE7: 0.4% Visits to admin pages on Australian Servers by browser, March 2014**

IE8 is still the most popular browser amongst our Australian admin users, but its lead is far less marked than in the UK.

March 2014 australian visits by browser: Firefox: 20.2%; Chrome: 18.6%; Safari: 14.9%; IE7: 11.0%; IE9: 7.0%; IE6: 6.4%; IE11: 5.8%; IE10: 4.6%; IE7: 4.5%; Opera: 2.3%; Other Mozilla-based: 2.0%; IE Unknown Version: 1.2% Visits to all pages on Australian servers by browser, March 2014**

When we consider all pages, not just admin pages, it’s interesting to see that the most popular browser used to visit our Australian sites is Firefox.  This is in contrast to StatCounter’s figures for Australian Browser usage, which pegs Chrome usage at more than double that of Firefox.  The numbers here are fairly small (16K visits a month) so this could have been skewed by one particularly Firefox-heavy demographic of survey respondents, for example.

Notes

* Visits to our newest app, Budget Simulator, are not included here, as a lot of the administration is done by Delib account managers, and our choice of browser would skew the statistics quite a lot.  Also, the visitor profile of Budget Simulator leans quite heavily towards mobile users.  I feel another blog post coming on!

** For the purposes of these statistics, a ‘visit’ comprises all the requests from a given IP address and useragent on a given day.

48GB of DDR3 memory, in individual 8GB sticks.

I’ve just added more RAM to our testing machines

Here at Delib, we use a little XenServer 6 cluster for continuous integration. We have programs that act like users of each of our online applications and put them through their paces, which we write using a framework called Selenium. We use these automated tests to demonstrate that the entire software stack under each of our applications all works together, before we roll out changes to any individual part of it. These are full end-to-end tests, demonstrating all of the layers from our application code through to the operating system that they are running on.

The XenServer cluster is a small group of physical computers that work together to host a larger group of virtual computers. This makes running our tests on different versions of operating systems much easier than if we had installed them directly onto physical computers, because we can do things like clone virtual machines or roll them back to a specific point in the past with just a few clicks (or commands).

Virtualisation is an enormous help for testing how our software interacts with its operating systems, but it has one very small downside: you end up needing a lot of RAM to run a large number of virtual machines. Each physical computer needs to have as much RAM in it as all of the virtual machines that it is hosting, added together. We’re not big fans of spending our lives manually shuffling things around to fit within limited resources when they’re cheap, so the first alteration that we are making to our QA cluster here is to double the amount of memory in it.

48GB of DDR3 memory, in individual 8GB sticks.
48GB of DDR3 memory, in individual 8GB sticks.

This was a stack of sticks of plastic and silicon to the immediate left of my desk, when it had just arrived this morning from Crucial. I’ve just installed them all into the physical computers, and we should be up and running and making use of this soon. This upgrade should, as soon as we’ve reconfigured everything to take advantage of the extra memory, make the time that it takes to complete one of our test runs at least a little shorter, which makes me happier.

Cookies, Delib and the EU e-Privacy Directive

“The Cookie Law” has been extensively covered elsewhere. If you’re not familiar with it, the website of the Information Commissioner (ICO) is the best place to start.

The e-Privacy Directive is broadly intended to prevent abusive, invasive and malicious behaviour by website operators, where information is gathered about individual website users without their informed consent. This is a worthwhile goal. As a side-effect, the law also prevents website operators from using practices which are widespread, and undertaken with no malicious or abusive intent.

Compliance with this directive is a process, and ICO guidance has been clear on what steps need to be taken by website operators. Here’s what we’ve been doing about it.

Recognition that is both a technical and legal process

We have:

  • – Taken legal advice about this.
  • – Read the law and the ICO guidance for ourselves.
  • – Looked at the technical implications.
  • – Made a plan for compliance.

 

How we’re tackling compliance

Steps we’ve taken or are taking include:

  • – Audit for cookie use of the apps we build and operate on behalf of our clients.
  • – Audit for cookie use of our own Delib-branded websites.
  • – Automation of auditing, to ensure we remain compliant over time (this is particularly important when 3rd party services can be embedded in to sites).
  • – Making it clear how cookies are used in our apps via their privacy and cookies statements.
  • – Technical work which helps our clients comply when embedding content from 3rd party services which may set cookies (for the end user we wrap an explicit ‘accept this embed’ choice around embedded content).
  • – Removing services on our own sites that set 3rd party cookies. We have already removed Google Analytics from a large number of our pages, and we’re working on removing further services that set cookies. For complete compliance, we are also planning to add an explicit cookies notice, similar to the solution implemented by the BBC.

We’ve chosen to see compliance with this directive as something that needs to be achieved and maintained, not simply a nuisance that might go away if ignored (as some website operators hope).

This work does use up limited time and money which could alternatively have been applied to features which help our clients, and to competing with other similar businesses around the world. However there’s a good case for users being informed about the use of cookies and similar local storage. Our practices are not abusive and malicious, but the prevention of those that are is important.

Good news on IE6: government usage plummets in 2012

Almost a year ago, we published some interesting browser statistics based on the logs from one of our Central Government Citizen Space servers. We ran the logs through a piece of open source analysis software called Visitors, and this gave us an anonymous breakdown of all visits to Citizen Space, showing the browsers and versions that were used. We looked at the statistics for all pages, and compared them with the stats for pages only accessible to admin users. The results were insightful but rather scary: Internet Explorer 6 accounted for more than 1 in 3 visits by our Central Government admin users.

For those who don’t know, Internet Explorer version 6 (lovingly known as IE6) is a web browser that Microsoft released over a decade ago. Because it renders web pages differently (in some cases dramatically differently) from more modern browsers, web developers spend a great deal of time creating workarounds so that IE6 users can still access our websites. Of course, this increases the cost of product development without necessarily offering any benefit to the majority of web users who don’t use IE6. Coupled with the fact that IE6 now only receives limited support from Microsoft, almost everyone is in agreement that this ancient and decrepit browser must be phased out.

This morning, we had a comment on last year’s post from a reader who was interested in how Citizen Space’s browser stats had changed. Thank you Perry – you reminded me that I’d been meaning to re-do this analysis soon. So here are the graphs comparing the numbers 11 months ago with where we are today:

Central Government Citizen Space – all users (admin and public)

All users by browser, May 2011
Breakdown of visits by browser: IE8: 23.0%, Firefox: 15.6%, IE9: 14.9%, IE7: 14.9%, Chrome: 13.2%, Safari: 10.8%, IE6: 5.9%
All users by browser, April 2012

These statistics roughly follow the browser trends of the general internet population*, with IE8 and 9 increasing in popularity while the older IE versions decrease as expected. Pleasingly, IE6 usage has roughly halved since last May.

Firefox, Chrome and Safari have gained more of a stronghold in the past year, although interestingly, Internet Explorer as a whole has retained a far larger share of Citizen Space users than worldwide browser usage statistics* would predict.

Central Government Citizen Space instances – admin pages only

Admin users by browser, May 2011
Breakdown of visits by browser: IE8: 60.3%, IE7: 29.3%, Chrome: 3.8%, IE6: 2.4%, Firefox: 1.8%, IE9: 1.3%, Safari: 1.1%
Admin users by browser, April 2012


When looking at the statistics for our admin users, the most exciting thing is that usage of IE6 has crashed by 90% – from 35% down to 2.4% of visits. This is a great relief to us, and shows the huge effort that has taken place in government IT departments to upgrade users away from this insecure, ill-supported browser.

It’s worth noting that overall, usage of Internet Explorer among our Central Government users is more than 90%, compared to 34% worldwide*.

What next?

The interesting question is what levels of support to provide for different browser capabilities. We currently provide Level 2 support for IE6, which means that all content must be readable and navigable, but differences in styling and layout may exist. This works OK for our products at the moment, but as web users come to expect a richer and more fluid experience, the likes of IE6 are going to lag further and further behind. How small does the percentage of IE6 users need to be before we can stop supporting it at all?

To other web developers: when do you stop supporting ancient browsers and those with limited functionality? How much can you rely on the presence of client-side technologies like Javascript, cookies, HTML5, CSS3..?

To government IT managers (thank you for phasing out IE6 btw!) what level of support do you expect for older browsers? Do security constraints dictate that you disable features like Javascript or cookies?

As always, I’d love to hear your views.


*Worldwide browser statistics from statcounter.com.

Awesome Citizen Space version 1.6.2 new features now out!

Citizen Space 1.6.2 has just been released with some really awesome new features. This is part of our commitment to keeping Citizen Space constantly improving and evolving with the ever-changing times.

Both current and future clients can now benefit from a range of new features, including these two great additions worth explaining in detail:

1) Generated graphical PDF reports

All Quick Consult consultations now include an extra link on the consultation dashboard to create a summary report in PDF format. Citizen Space administrators have the option to create a report which can be used to both track open consultations’ progress or provide a quick and easy to use summary report of closed consultations’ outcomes.

We are really excited about this new feature and have already been chatting to our current clients about some of the potential benefits and use cases. The analysis of results and subsequent consultation feedback loop back to the public can now be much quicker and easier. For example reports can be generated quickly for a meeting with stakeholders and policy makers to review/assess progress.

  • For questions where respondents can select at most one answer, such as radio buttons or a drop down, a pie chart is displayed:

  • For questions where respondents can select more than one answer, such as checkboxes, a bar chart is displayed:

2) Mailing list sign up for Quick Consult respondents

Respondents can now have the option to opt-in to a mailing list once they have completed a response. The email list can then be exported and used to keep respondents informed on consultation outcomes and results. The email address opt-in feature can be enabled on a per-consultation basis to ensure that it is only used on relevant consultations.

The text above the email opt-in option can be easily edited by the administrator to ensure that respondents will know how their email address will be used.

It is possible to view the number of email signups at any stage of the consultation on the dashboard without needing to download the list.

We also included continuous improvements across the app that many of our users will no doubt appreciate; such as a nice bright ‘Jump to a page’ bar on the ‘view response’ page:

For more information on Citizen Space or to request a demonstration please contact one of our Team on 0845 638 1848.

#IndyRef

On January 25th Alex Salmon announced that The Scottish Government would be consulting on the measures that will influence the proposed independence referendum of 2014. The consultation paper seeks opinion on a variety of measures includes voting age, days to vote and even the nature of the ballot paper itself. This is arguably the most contentious and important consultation of the year, influencing as it does, the potential breakup of the Union. Recognising the importance of the consultation, and the number of potential respondents The Scottish Government chose to use Delib’s Citizen Space, owing to, amongst other things, it’s unparalleled record of running the UK’s largest online consultations. Naturally, I therefore thought it would be worth a quick blog post to see how they’ve utilised the system.

The first thing you notice upon landing on the ‘Your Scotland, Your Referendum’ page is the clarity of the information presented. A participant can choose to look at the supporting documentation in addition to clear succinct headline information. The latter point is often overlooked, especially by organisations new to digital tools, who feel the need to promote transparency by bombarding a participant with a discouraging amount of information.

Once you click through into the online consultation which uses the ‘Quick Consult’ module within Citizen Space, you notice the way in which the offline survey has been mirrored online. Often internet tools can struggle with things such as multiple answer components, or indeed the ability to upload documents to support an answer; not so here.

And finally…
It’s worth noting that the consultation seeks views from all UK residents, and not just the Scots, and I therefore implore you to take part. You’ve got until the 11th of May which, in combination with the ‘Save your response to return later…’ feature, gives you no excuse not to.

Thames Tunnel videos syndicated from our site

As I touched upon in my previous launch post we made the decision to host all of Thames Tunnel’s consultation materials on resources that allowed users to easily share, embed and generally syndicate them. One of these systems is the well known (and pretty, popular alternative to YouTube) Vimeo.

Whilst navigating the popular Londonist site on a recent excursion to the city I was pleased to see that they had taken one of the excellent explanatory videos from the site and embedded it on the site. I think this is a really great example of how the consultation can be explained quickly in one article using resources that were easily accessible.

Well done Londonist, and well done Vimeo.

Delib and Thames Water go live with phase 2 of Thames Tunnel consultation site

Today at 00:01 the Delib team deployed an updated site for Thames Water’s latest phase of their Thames Tunnel project. The update coincides with the launch of the second phase of the consultation which is running using our Quick Consult.

The site itself is built on the (much loved by us and others) Word Press platform and makes use of a number of excellent plug-ins to further enrich the presentation of the huge amount of information that is available on the project. Other technologies and services used include Scribd, Vimeo, Google Maps and some rather impressive cross-browser compliant Javascript libraries.

Screenshot of a site detail page from the consultation website
The site now contains a separate section for each of the 25 proposed construction sites providing maps, documents and news.

A lot of hard work has gone into getting this site live from teams at London Tideway Tunnels and Thames Water alongside some inspiring technical work from my colleagues here at Delib. We are very proud to be a part of the project and look forward to our app providing a platform for engaging the public gathering their opinions.