Yesterday we rolled out a few more SQL injection fixes (we’ve moved the vast majority – if not all – of the public “used” SQL statements from mysql_* connection strings to parametrised PDO connections) and a number of XSS (Cross-Site-Scripting) vulnerability fixes to make the PublicWhip code even more secure. We’ve also tweaked import routines ever so slightly (to try and use “indexes” on table joins) and to provide a bit logging as to what is happening during the import. We also updated the code to move away from PHP short tags (such as “ echo..." and "= $variable" to make it a bit more usable (PHP now comes as "default" with short tags disabled) - we've also updated the mysql_escape_string statements to use mysql_real_escape_string as the first one has been depreciated (the latter is also being depreciated, but can still be used - hopefully this will buy us some time whilst we recode the site).
We also made some database optimisation changes in MySQL which should see some page load speed improvements and also enabled Apache suExec to help increase security that little bit more.
We've just finished pushing all the code changes to our new BitBucket repository at https://bitbucket.org/publicwhip/publicwhip-v1/ – the old GitHub repository will be maintained until at least April this year, but we urge people to move over to the BitBucket Git repository as soon as possible.
We’ve completed sending out notifications to all affected users of Friday’s exploit. Email notifications started going out at 19:31 on the 25th of January at a rate of 1 per second and finished at 18:32 on the 26th of January.
There were 81,205 email addresses included in the exploited data – of those, 598 were exact duplicates. We therefore tried sending to 79,809 different addresses (we didn’t filter the list any further at this point as we just wanted people to be notified as quickly as possible).
After the emails were sent, we did analyse the data and found:
* 12 email addresses did not actually have a valid domain name structure
* 2,906 email addresses were sent to domains which cannot receive email (such as no longer being registered)
* 14,427 email addresses were actually duplicates (such as gmail.com and googlemail.com, firstname.lastname@example.org and email@example.com and firstname.lastname@example.org and email@example.com)
* 31,582 email addresses were on “common domains” (defined as gmail.com, aol.com, aol.co.uk, hotmail.com, hotmail.co.uk, yahoo.com, yahoo.co.uk, outlook.com, aol.co.uk, btinternet.com, fastmail.fm, wanadoo.fr, bbc.co.uk, mac.com, ntlworld.com, tiscali.co.ul, mailinator.com and msn.com)
* 19,467 email addresses were classed as “likely spammers” (the domain was listed on http://www.stopforumspam.com/spamdomainsandips but wasn’t in the list of “common domains”)
* 5,029 “other” email addresses looked valid, had good DNS entries and were not listed on the common domains and “likely spammers” list
* 12 of those email accounts accounts were over quota and have not yet received the notification.
We are therefore counting the number of exposed “data users” as “common domains”+”others”:-“over quota”: 35,599. Continue reading →
We are saddened to announce that on the 24th of January 2013 between 04:02 and 04:56 GMT, that the main PublicWhip site was exploited. We became aware of this around 07:30 GMT on the 25th.
Impact to you
Anybody that has signed up to the PublicWhip website (forum or membership) will have had their plain text email address exposed, their PublicWhip username and an md5 hash of their PublicWhip password exposed. Whilst the passwords were “hashed”, they were not salted (due to interaction with other code) and hence should be declared as “in the wild”.
You are advised to change any passwords which are the same as the PublicWhip one.
You will no longer be able to log in to PublicWhip as the accounts have been wiped from the server.
No financial information was exposed.
We are sorry for this inconvenience and any problems caused.
This exploit took advantage of an SQL hole in the code (basically, the inherited PublicWhip code did not properly “escape” data sent from the user: we’ve now been through and changed the vast majority of these to “PHP PDO parametrised queries” – we’re still trying to catch any remaining queries). This allowed the third party access to the MySQL database.
Impact to PublicWhip
What we’ve done:
We’ve amended quite a lot of the core code of PublicWhip to prevent this from happening in the future and have setup a number of alerts to try and catch code that we’ve missed.
We’re going to be checking for the next two weeks or so to ensure we’ve caught all changes before making the new codebase available
Disabled the “account section” of the site as there are a number of exploitable holes that we have not currently fixed
Changed all the passwords stored on the live database to a random 32 character string
Changed all the email addresses stored on the live database to a random 32 character string
Changed all the user names stored on the live database to a random 32 character string
What we are going to do:
Over the course of the next few days, we will be emailing all 79,810 email addresses held on the PublicWhip database with details of blog post and asking them to review their passwords
Report this loss of data to the Information Commissioner’s office (it is unclear whether we need to do this, but we’re going to be doing it anyway)
Recently, we’ve made the following changes to PublicWhip:
* Removed logging of inbound requests
Previously all page requests were logged locally – however, we were alerted to a problem when people had search for names containing apostrophes in names (such as Baroness O’Cathain). This has been fixed (and the appropriate page displays) instead of people receiving a “Something’s gone wrong” message (thanks R.Vanderbeck for alerting us to this issue)
* Powered by CloudFlare
We’ve started using CloudFlare as a CDN front-end solution for the main PublicWhip site: this should lead to faster page loads for everybody, along with reducing the load on our server (CloudFlare has saved 4Gb of bandwidth in the last 7 days: from a total of 20.6Gb). It also includes some anti-hacker/anti-web spammer technologies which should stop some issues: however, it may also flag up some “false positives” – please let us know if you encounter any issues.
* Rebuild started
We are in the “ground breaking” stages of PublicWhip v2 – where we are rebuilding the codebase from scratch and making it entirely self supportive (instead of having to rely on Hansard being parsed by third parties). There’s not much code yet, but we’re piecing things together. Looking at our “paid workload” (i.e. the stuff that pays our rent/wages), we’re going to be a bit busy next month (February), but should be able to continue development in March.
* Switch back to Google Adsense
PublicWhip is now back using Google Adsense to try and raise funding to pay for development. To give you an idea of the revenue, in the last 30 days, Adsense has earnt enough money to pay for one fifth of it’s monthly hosting bill. So it’s definitely not making money for us: but at least it’s not quite so much of a drain on our limited resources.
We’ve switched PublicWhip back to Google Adsense advertising for a short term just to see if it’ll perform better than before.
We’re now in a much better state to get PublicWhip rebuilt and we should be able to start work before the end of January. The plan is to first rewrite the import routines, then the display system and then the search routines. We are not planning on reintroducing “community features” (such as a forum) at the moment in time – we’re planning on PublicWhip “just being about the facts” (we’ve also been advised by our insurance company that keeping things “just from an official source such as Hansard” will reduce any liability for us).
PublicWhip is still a “labour of love”: it doesn’t bring in enough to cover its hosting costs (so it is already being run at a loss for us) and we hoping that any funds we raise from advertising will enable us to “spin off” PublicWhip to its own “non-profit” company or set it up as a charity: but it’ll need enough money to cover it’s own costs at that point. As with all “labours of love”, if it is no longer “fun” to do (i.e. people are negative about it), we may consider whether it is worth our time: so please be positive and helpful!
We’ve just resolved a problem with the http://www.publicwhip.org.uk/project/data.php data feed for the votematrix-2010.txt file. It appears the incorrect data for the 2010 parliament was stored (2010-05-18 instead of 2010-05-06) causing only the 7 MPs voted into the Commons since that data to be included.
Based on the feedback we’ve received, we’ve decided to drop the Google Adsense from the Public Whip website. Of the 52,142 advert views we had in just over a month, the income from the site was just £19.32 [these funds have been transferred to the new donation scheme].
However, we still need to raise funds to pay for the overhaul of Public Whip, so we’ve switch to using Pledgie and we’re asking for your help and your donations!
We’re hoping to raise £15,000 to redevelop Public Whip to include the following new features:
* Better security with coding standards code
* Faster site
* Better designed/easier to use
* “Tagging” of divisions to increase findability of related issues
* API for you to easily integrate Public Whip data into your own website
* Internationalisation (hopefully including MEP and local councillor support)
* Faster updates (taking feeds directly from Hansard with the ability to “live update” on important issues)
* Better search facilities
* Twitter updates
amongst other things. If you would like to make a donation, you can:
Donate via Pledgie. . Payments are accepted by Paypal. Donations via this method will cost us 6.4%+20p in total. “PAYPAL*PLEDGIE” will appear on your credit card statement.
All major credit/debit cards accepted. Donations via this method will cost us 3.4%+20p in total. “PAYPAL*BAIRWELL” will appear on your credit card statement
Send us a cheque or postal order. These should be made payable to “Bairwell Ltd – Publicwhip fundraising” and sent to “Publicwhip, c/o Bairwell Ltd, 43 Haymakers Lane, Ashford, Kent, TN23 4GL”. Donations via this method will cost 75p
If you would like your donation to be anonymous, please let us know. If you would like to donate via some other method (credit/debit cards over the phone) please email team [at] publicwhip.org.uk . If you would like to pledge some time in helping re-write PublicWhip (which is MySQL/PHP based and will be using the Zend Framework), please again let us know – we’re hoping to be able to be able to do the rewrite in October/November time…
PublicWhip is currently hosted on a Memset Miniserver VM4000 which costs us £24.95 per month. We are willing to write this hosting cost off. We’re also writing off the cost of the New Relic monitoring system, our time spent responding to queries, kicking the old code when it breaks and other associated gubbins.
We estimate that a total rewrite of PublicWhip will take 480 man hours – even at our lowest possible “extremely easy work for charity” pricing, it’s still looking at £9,600 (baring in mind that when we’re working on Public Whip, we aren’t working on projects which are making us money). At the current revenue rate we’re getting from Google Adwords (baring in mind, it is still very early days) – it’ll take 6.5 years to get that money, but we don’t really want to increase the number of adverts.
We’d much prefer a “PublicWhip is supported by [your brand here]” style thing: one sponsorship message per page (that’s 46,877 times viewed per month across 10,454 unique visitors: April to date) and no other advertisements. So please contact us at team [at] publicwhip.org.uk for sponsorship details!
As you can see, we’re currently trailing advertisements on the Public Whip website. These advertisements, currently powered by Google Adsense, will be kept to a minimum and are just intended to try and fund the development and hosting of Public Whip.
For the last 6 months or so, Bairwell Ltd has been hosting the PublicWhip website totally free of charge (even though we’ve got expenses associated with the hosting) and generally maintaining the site/answering queries. However, we have got big ideas for the site: but we’ll need to rewrite it from scratch to fix these issues – this will take time (a few months) – and we do, unfortunately, need to be able to eat during this period! Hence the advertisements to try and raise funds.
We have tried to raise funds in alternative ways (spending in excess of £500 to do so: at a conservative estimate, PublicWhip has cost us over £3,000 in the last 6 months!), but we were unable to find backers
We don’t like running adverts and would prefer not to – so if you want to sponsor the site (nothing too political please: the site needs to keep its neutrality), please please please get in contact with us at team [at] publicwhip.org.uk .
If you’ve got any thoughts on these changes, please let us know in the comments!
Due to numerous inherited flaws in the codebase we’re seeing rather a lot of spam, both in the forums and in the email of registered users. As such this weekend we’ll be applying a number of patches, and stop-gap measures to tighten things up.
Our primary concern is the reports we’ve received from people who have begun to receive spam email to addresses that are only utilised on PublicWhip. We despise spammers and can categorically state that we have never, and will never, share or sell the personal data that our users entrust with us. If you have received spam email to an address that has only been used on PublicWhip, would you please let us know the address details (email firstname.lastname@example.org) to help us identify where the data leak is coming from and to plug it.
One of the main culprits is a very out of date installation of PHPBB and whilst we look at alternatives, and gather input as to the usefulness of the forum (beyond acting as a changelog for divisions and policies), we’ll be taking the forum offline.
We’re still getting to grips with a system and a codebase that is relatively new to us, but we’re determined to make the site the best that it can be, and data security and data integrity will always be at the heart of what we do.