We just attempted to make the PublicWhip code compatible with the PHP “Fig standards” in particular PSR-1-basic-coding-standard and PSR-2-coding-style-guide by utilising PHP Coding Standards Fixer. However, during testing it was quickly shown that due to the age of the PublicWhip code, it appears nearly impossible to automate the bringing of version 1 code up to standard (we even tried a reduced set of changes: in particular disabling the psr0 ["Classes must be in a path that matches their namespace"] and include ["Include and file path should be divided with single space"] fixes, but the site still errored).

Yesterday we rolled out a few more SQL injection fixes (we’ve moved the vast majority – if not all – of the public “used” SQL statements from mysql_* connection strings to parametrised PDO connections) and a number of XSS (Cross-Site-Scripting) vulnerability fixes to make the PublicWhip code even more secure. We’ve also tweaked import routines ever so slightly (to try and use “indexes” on table joins) and to provide a bit logging as to what is happening during the import. We also updated the code to move away from PHP short tags (such as “

We also made some database optimisation changes in MySQL which should see some page load speed improvements and also enabled Apache suExec to help increase security that little bit more.

We've just finished pushing all the code changes to our new BitBucket repository at https://bitbucket.org/publicwhip/publicwhip-v1/ – the old GitHub repository will be maintained until at least April this year, but we urge people to move over to the BitBucket Git repository as soon as possible.

There were 81,205 email addresses included in the exploited data – of those, 598 were exact duplicates. We therefore tried sending to 79,809 different addresses (we didn’t filter the list any further at this point as we just wanted people to be notified as quickly as possible).

After the emails were sent, we did analyse the data and found:
* 12 email addresses did not actually have a valid domain name structure
* 2,906 email addresses were sent to domains which cannot receive email (such as no longer being registered)
* 14,427 email addresses were actually duplicates (such as gmail.com and googlemail.com, e.x.a.mple@gmail.com and example@gmail.com and test+123@example.com and test@example.com)
* 31,582 email addresses were on “common domains” (defined as gmail.com, aol.com, aol.co.uk, hotmail.com, hotmail.co.uk, yahoo.com, yahoo.co.uk, outlook.com, aol.co.uk, btinternet.com, fastmail.fm, wanadoo.fr, bbc.co.uk, mac.com, ntlworld.com, tiscali.co.ul, mailinator.com and msn.com)
* 19,467 email addresses were classed as “likely spammers” (the domain was listed on http://www.stopforumspam.com/spamdomainsandips but wasn’t in the list of “common domains”)
* 5,029 “other” email addresses looked valid, had good DNS entries and were not listed on the common domains and “likely spammers” list
* 12 of those email accounts accounts were over quota and have not yet received the notification.

We are therefore counting the number of exposed “data users” as “common domains”+”others”:-”over quota”: 35,599.

The top 20 most common domain names used (and their classifications) were:
1. gmail.com (31,159 emails) [Common domain]
2. mail.ru (5,605 emails) [Spammer domain]
3. aol.com (5,491 emails) [Common domain]
4. yahoo.co.uk (2,893 emails) [Common domain]
5. hotmail.com (1,792 emails) [Common domain]
6. nextfash.com (1,288 emails) [Invalid domain]
7. vipmail.net (1,159 emails) [Spammer domain]
8. o2.pl (1,095 emails) [Spammer domain]
9. msn.com (1,056 emails) [Common domain]
10. yahoo.com (1,016 emails) [Common domain]
11. dabjam.com (913 emails) [Spammer domain]
12. yandex.ru (842 emails) [Spammer domain]
13. go2.pl (672 emails) [Spammer domain]
14. tlen.pl (645 emails) [Spammer domain]
15. prokonto.pl (596 emails) [Spammer domain]
16. web.de (594 emails) [Spammer domain]
17. vaver.info (576 emails) [Invalid domain]
18. rambler.ru (496 emails) [Spammer domain]
19. socmail.net (424 emails) [Spammer domain]
20. gawab.com (414 emails) [Spammer domain]

Impact to you
Anybody that has signed up to the PublicWhip website (forum or membership) will have had their plain text email address exposed, their PublicWhip username and an md5 hash of their PublicWhip password exposed. Whilst the passwords were “hashed”, they were not salted (due to interaction with other code) and hence should be declared as “in the wild”.

You are advised to change any passwords which are the same as the PublicWhip one.

You will no longer be able to log in to PublicWhip as the accounts have been wiped from the server.

No financial information was exposed.

We are sorry for this inconvenience and any problems caused.

Background
This exploit took advantage of an SQL hole in the code (basically, the inherited PublicWhip code did not properly “escape” data sent from the user: we’ve now been through and changed the vast majority of these to “PHP PDO parametrised queries” – we’re still trying to catch any remaining queries). This allowed the third party access to the MySQL database.

Impact to PublicWhip
What we’ve done:

• We’ve amended quite a lot of the core code of PublicWhip to prevent this from happening in the future and have setup a number of alerts to try and catch code that we’ve missed.
• We’re going to be checking for the next two weeks or so to ensure we’ve caught all changes before making the new codebase available
• Disabled the “account section” of the site as there are a number of exploitable holes that we have not currently fixed
• Changed all the passwords stored on the live database to a random 32 character string
• Changed all the email addresses stored on the live database to a random 32 character string
• Changed all the user names stored on the live database to a random 32 character string

What we are going to do:

• Over the course of the next few days, we will be emailing all 79,810 email addresses held on the PublicWhip database with details of blog post and asking them to review their passwords
• Report this loss of data to the Information Commissioner’s office (it is unclear whether we need to do this, but we’re going to be doing it anyway)

FAQ
Q. How quickly did you find out about this?
A. We found out about it at around 7.30am Friday via a comment on this blog (about a web page which had been live for less than 2 hours).

Q. Have you wiped all the user data?
A. We know which divisions have been edited etc by which user id, but the live site does not have a list of usernames, email addresses or passwords that those user ids relate to. We have got off-line backups (held in an encrypted partition on our office NAS which cannot be accessed via the internet) which holds these relations: however, we have not yet decided whether just to let those backups expire.

Q. How old was the code affected?
A. The code that was exploited was nearly 10 years old and was written before PHP had various security measures (such as DBI, PDO) in-built.

Q. Does this affect any other sites hosted or maintained by Bairwell?
A. No. The PublicWhip site has always been kept separate from the rest of the Bairwell hosted sites due to a combination of resources needed and suspected security holes.

Q. Were you are of the potential of this security breach before hand?
A. No. We were aware of some basic HTML injection holes (which may or may not still exist) which just meant carefully crafted links would display set text to the user – but these did not impact the actual data storage system and did not pose a security risk. We did not know that the SQL injection issue was possible.

Q. What will you be doing in the future?
A. Once we’ve raised some more money to support development of PublicWhip v2 (please feel free to donate), we will rebuild the whole site from scratch using best practices and higher security (such as using bCrypt based password hashing). We are, actually, of a mind to not have any “on-site” login facilities at all, but allow logins managed by third parties (Facebook, Google, OpenId etc).

If you’ve got any questions about this, please do not hesitate to contact us:

* Removed logging of inbound requests
Previously all page requests were logged locally – however, we were alerted to a problem when people had search for names containing apostrophes in names (such as Baroness O’Cathain). This has been fixed (and the appropriate page displays) instead of people receiving a “Something’s gone wrong” message (thanks R.Vanderbeck for alerting us to this issue)

* Powered by CloudFlare
We’ve started using CloudFlare as a CDN front-end solution for the main PublicWhip site: this should lead to faster page loads for everybody, along with reducing the load on our server (CloudFlare has saved 4Gb of bandwidth in the last 7 days: from a total of 20.6Gb). It also includes some anti-hacker/anti-web spammer technologies which should stop some issues: however, it may also flag up some “false positives” – please let us know if you encounter any issues.

* Rebuild started
We are in the “ground breaking” stages of PublicWhip v2 – where we are rebuilding the codebase from scratch and making it entirely self supportive (instead of having to rely on Hansard being parsed by third parties). There’s not much code yet, but we’re piecing things together. Looking at our “paid workload” (i.e. the stuff that pays our rent/wages), we’re going to be a bit busy next month (February), but should be able to continue development in March.

* Switch back to Google Adsense
PublicWhip is now back using Google Adsense to try and raise funding to pay for development. To give you an idea of the revenue, in the last 30 days, Adsense has earnt enough money to pay for one fifth of it’s monthly hosting bill. So it’s definitely not making money for us: but at least it’s not quite so much of a drain on our limited resources.

We’re now in a much better state to get PublicWhip rebuilt and we should be able to start work before the end of January. The plan is to first rewrite the import routines, then the display system and then the search routines. We are not planning on reintroducing “community features” (such as a forum) at the moment in time – we’re planning on PublicWhip “just being about the facts” (we’ve also been advised by our insurance company that keeping things “just from an official source such as Hansard” will reduce any liability for us).

PublicWhip is still a “labour of love”: it doesn’t bring in enough to cover its hosting costs (so it is already being run at a loss for us) and we hoping that any funds we raise from advertising will enable us to “spin off” PublicWhip to its own “non-profit” company or set it up as a charity: but it’ll need enough money to cover it’s own costs at that point. As with all “labours of love”, if it is no longer “fun” to do (i.e. people are negative about it), we may consider whether it is worth our time: so please be positive and helpful!

This has now been fixed.

However, we still need to raise funds to pay for the overhaul of Public Whip, so we’ve switch to using Pledgie and we’re asking for your help and your donations!

We’re hoping to raise £15,000 to redevelop Public Whip to include the following new features:
* Better security with coding standards code
* Faster site
* Better designed/easier to use
* “Tagging” of divisions to increase findability of related issues
* API for you to easily integrate Public Whip data into your own website
* Internationalisation (hopefully including MEP and local councillor support)
* Faster updates (taking feeds directly from Hansard with the ability to “live update” on important issues)
* Better search facilities
* Twitter updates

amongst other things. If you would like to make a donation, you can:

• Donate via Pledgie.
  .
  Payments are accepted by Paypal.
  Donations via this method will cost us 6.4%+20p in total. “PAYPAL*PLEDGIE” will appear on your credit card statement.
• Donate via Paypal directly.
  
  
  
  
  
  

  All major credit/debit cards accepted.
  Donations via this method will cost us 3.4%+20p in total. “PAYPAL*BAIRWELL” will appear on your credit card statement

• Send us a cheque or postal order. These should be made payable to “Bairwell Ltd – Publicwhip fundraising” and sent to “Publicwhip, c/o Bairwell Ltd, 43 Haymakers Lane, Ashford, Kent, TN23 4GL”. Donations via this method will cost 75p

If you would like your donation to be anonymous, please let us know. If you would like to donate via some other method (credit/debit cards over the phone) please email team [at] publicwhip.org.uk . If you would like to pledge some time in helping re-write PublicWhip (which is MySQL/PHP based and will be using the Zend Framework), please again let us know – we’re hoping to be able to be able to do the rewrite in October/November time…

PublicWhip is currently hosted on a Memset Miniserver VM4000 which costs us £24.95 per month. We are willing to write this hosting cost off. We’re also writing off the cost of the New Relic monitoring system, our time spent responding to queries, kicking the old code when it breaks and other associated gubbins.

We charge for PHP development and WordPress / Perch development between £40-£60 per hour dependent on the complexity of the work.

We estimate that a total rewrite of PublicWhip will take 480 man hours – even at our lowest possible “extremely easy work for charity” pricing, it’s still looking at £9,600 (baring in mind that when we’re working on Public Whip, we aren’t working on projects which are making us money). At the current revenue rate we’re getting from Google Adwords (baring in mind, it is still very early days) – it’ll take 6.5 years to get that money, but we don’t really want to increase the number of adverts.

We’d much prefer a “PublicWhip is supported by [your brand here]” style thing: one sponsorship message per page (that’s 46,877 times viewed per month across 10,454 unique visitors: April to date) and no other advertisements. So please contact us at team [at] publicwhip.org.uk for sponsorship details!

For the last 6 months or so, Bairwell Ltd has been hosting the PublicWhip website totally free of charge (even though we’ve got expenses associated with the hosting) and generally maintaining the site/answering queries. However, we have got big ideas for the site: but we’ll need to rewrite it from scratch to fix these issues – this will take time (a few months) – and we do, unfortunately, need to be able to eat during this period! Hence the advertisements to try and raise funds.

We have tried to raise funds in alternative ways (spending in excess of £500 to do so: at a conservative estimate, PublicWhip has cost us over £3,000 in the last 6 months!), but we were unable to find backers

We don’t like running adverts and would prefer not to – so if you want to sponsor the site (nothing too political please: the site needs to keep its neutrality), please please please get in contact with us at team [at] publicwhip.org.uk .

If you’ve got any thoughts on these changes, please let us know in the comments!