This is another techy post – sorry guys!

Yesterday we rolled out a few more SQL injection fixes (we’ve moved the vast majority – if not all – of the public “used” SQL statements from mysql_* connection strings to parametrised PDO connections) and a number of XSS (Cross-Site-Scripting) vulnerability fixes to make the PublicWhip code even more secure. We’ve also tweaked import routines ever so slightly (to try and use “indexes” on table joins) and to provide a bit logging as to what is happening during the import. We also updated the code to move away from PHP short tags (such as “

We also made some database optimisation changes in MySQL which should see some page load speed improvements and also enabled Apache suExec to help increase security that little bit more.

We've just finished pushing all the code changes to our new BitBucket repository at https://bitbucket.org/publicwhip/publicwhip-v1/ – the old GitHub repository will be maintained until at least April this year, but we urge people to move over to the BitBucket Git repository as soon as possible.

We’ve completed sending out notifications to all affected users of Friday’s exploit. Email notifications started going out at 19:31 on the 25th of January at a rate of 1 per second and finished at 18:32 on the 26th of January.

There were 81,205 email addresses included in the exploited data – of those, 598 were exact duplicates. We therefore tried sending to 79,809 different addresses (we didn’t filter the list any further at this point as we just wanted people to be notified as quickly as possible).

After the emails were sent, we did analyse the data and found:
* 12 email addresses did not actually have a valid domain name structure
* 2,906 email addresses were sent to domains which cannot receive email (such as no longer being registered)
* 14,427 email addresses were actually duplicates (such as gmail.com and googlemail.com, e.x.a.mple@gmail.com and example@gmail.com and test+123@example.com and test@example.com)
* 31,582 email addresses were on “common domains” (defined as gmail.com, aol.com, aol.co.uk, hotmail.com, hotmail.co.uk, yahoo.com, yahoo.co.uk, outlook.com, aol.co.uk, btinternet.com, fastmail.fm, wanadoo.fr, bbc.co.uk, mac.com, ntlworld.com, tiscali.co.ul, mailinator.com and msn.com)
* 19,467 email addresses were classed as “likely spammers” (the domain was listed on http://www.stopforumspam.com/spamdomainsandips but wasn’t in the list of “common domains”)
* 5,029 “other” email addresses looked valid, had good DNS entries and were not listed on the common domains and “likely spammers” list
* 12 of those email accounts accounts were over quota and have not yet received the notification.

We are therefore counting the number of exposed “data users” as “common domains”+”others”:-”over quota”: 35,599.
Continue reading →

We are saddened to announce that on the 24th of January 2013 between 04:02 and 04:56 GMT, that the main PublicWhip site was exploited. We became aware of this around 07:30 GMT on the 25th.

Impact to you
Anybody that has signed up to the PublicWhip website (forum or membership) will have had their plain text email address exposed, their PublicWhip username and an md5 hash of their PublicWhip password exposed. Whilst the passwords were “hashed”, they were not salted (due to interaction with other code) and hence should be declared as “in the wild”.

You are advised to change any passwords which are the same as the PublicWhip one.

You will no longer be able to log in to PublicWhip as the accounts have been wiped from the server.

No financial information was exposed.

We are sorry for this inconvenience and any problems caused.

Background
This exploit took advantage of an SQL hole in the code (basically, the inherited PublicWhip code did not properly “escape” data sent from the user: we’ve now been through and changed the vast majority of these to “PHP PDO parametrised queries” – we’re still trying to catch any remaining queries). This allowed the third party access to the MySQL database.

Impact to PublicWhip
What we’ve done:

• We’ve amended quite a lot of the core code of PublicWhip to prevent this from happening in the future and have setup a number of alerts to try and catch code that we’ve missed.
• We’re going to be checking for the next two weeks or so to ensure we’ve caught all changes before making the new codebase available
• Disabled the “account section” of the site as there are a number of exploitable holes that we have not currently fixed
• Changed all the passwords stored on the live database to a random 32 character string
• Changed all the email addresses stored on the live database to a random 32 character string
• Changed all the user names stored on the live database to a random 32 character string

What we are going to do:

• Over the course of the next few days, we will be emailing all 79,810 email addresses held on the PublicWhip database with details of blog post and asking them to review their passwords
• Report this loss of data to the Information Commissioner’s office (it is unclear whether we need to do this, but we’re going to be doing it anyway)

Continue reading →

Recently, we’ve made the following changes to PublicWhip:

* Removed logging of inbound requests
Previously all page requests were logged locally – however, we were alerted to a problem when people had search for names containing apostrophes in names (such as Baroness O’Cathain). This has been fixed (and the appropriate page displays) instead of people receiving a “Something’s gone wrong” message (thanks R.Vanderbeck for alerting us to this issue)

* Powered by CloudFlare
We’ve started using CloudFlare as a CDN front-end solution for the main PublicWhip site: this should lead to faster page loads for everybody, along with reducing the load on our server (CloudFlare has saved 4Gb of bandwidth in the last 7 days: from a total of 20.6Gb). It also includes some anti-hacker/anti-web spammer technologies which should stop some issues: however, it may also flag up some “false positives” – please let us know if you encounter any issues.

* Rebuild started
We are in the “ground breaking” stages of PublicWhip v2 – where we are rebuilding the codebase from scratch and making it entirely self supportive (instead of having to rely on Hansard being parsed by third parties). There’s not much code yet, but we’re piecing things together. Looking at our “paid workload” (i.e. the stuff that pays our rent/wages), we’re going to be a bit busy next month (February), but should be able to continue development in March.

* Switch back to Google Adsense
PublicWhip is now back using Google Adsense to try and raise funding to pay for development. To give you an idea of the revenue, in the last 30 days, Adsense has earnt enough money to pay for one fifth of it’s monthly hosting bill. So it’s definitely not making money for us: but at least it’s not quite so much of a drain on our limited resources.

We’ve switched PublicWhip back to Google Adsense advertising for a short term just to see if it’ll perform better than before.

We’re now in a much better state to get PublicWhip rebuilt and we should be able to start work before the end of January. The plan is to first rewrite the import routines, then the display system and then the search routines. We are not planning on reintroducing “community features” (such as a forum) at the moment in time – we’re planning on PublicWhip “just being about the facts” (we’ve also been advised by our insurance company that keeping things “just from an official source such as Hansard” will reduce any liability for us).

PublicWhip is still a “labour of love”: it doesn’t bring in enough to cover its hosting costs (so it is already being run at a loss for us) and we hoping that any funds we raise from advertising will enable us to “spin off” PublicWhip to its own “non-profit” company or set it up as a charity: but it’ll need enough money to cover it’s own costs at that point. As with all “labours of love”, if it is no longer “fun” to do (i.e. people are negative about it), we may consider whether it is worth our time: so please be positive and helpful!

We’ve just resolved a problem with the http://www.publicwhip.org.uk/project/data.php data feed for the votematrix-2010.txt file. It appears the incorrect data for the 2010 parliament was stored (2010-05-18 instead of 2010-05-06) causing only the 7 MPs voted into the Commons since that data to be included.

This has now been fixed.

Yesterday, I mentioned why PublicWhip is currently running advertisements – today, I provide figures.

PublicWhip is currently hosted on a Memset Miniserver VM4000 which costs us £24.95 per month. We are willing to write this hosting cost off. We’re also writing off the cost of the New Relic monitoring system, our time spent responding to queries, kicking the old code when it breaks and other associated gubbins.

We charge for PHP development and WordPress / Perch development between £40-£60 per hour dependent on the complexity of the work.

We estimate that a total rewrite of PublicWhip will take 480 man hours – even at our lowest possible “extremely easy work for charity” pricing, it’s still looking at £9,600 (baring in mind that when we’re working on Public Whip, we aren’t working on projects which are making us money). At the current revenue rate we’re getting from Google Adwords (baring in mind, it is still very early days) – it’ll take 6.5 years to get that money, but we don’t really want to increase the number of adverts.

We’d much prefer a “PublicWhip is supported by [your brand here]” style thing: one sponsorship message per page (that’s 46,877 times viewed per month across 10,454 unique visitors: April to date) and no other advertisements. So please contact us at team [at] publicwhip.org.uk for sponsorship details!

As you can see, we’re currently trailing advertisements on the Public Whip website. These advertisements, currently powered by Google Adsense, will be kept to a minimum and are just intended to try and fund the development and hosting of Public Whip.

For the last 6 months or so, Bairwell Ltd has been hosting the PublicWhip website totally free of charge (even though we’ve got expenses associated with the hosting) and generally maintaining the site/answering queries. However, we have got big ideas for the site: but we’ll need to rewrite it from scratch to fix these issues – this will take time (a few months) – and we do, unfortunately, need to be able to eat during this period! Hence the advertisements to try and raise funds.

We have tried to raise funds in alternative ways (spending in excess of £500 to do so: at a conservative estimate, PublicWhip has cost us over £3,000 in the last 6 months!), but we were unable to find backers

We don’t like running adverts and would prefer not to – so if you want to sponsor the site (nothing too political please: the site needs to keep its neutrality), please please please get in contact with us at team [at] publicwhip.org.uk .

If you’ve got any thoughts on these changes, please let us know in the comments!

Due to numerous inherited flaws in the codebase we’re seeing rather a lot of spam, both in the forums and in the email of registered users. As such this weekend we’ll be applying a number of patches, and stop-gap measures to tighten things up.

Our primary concern is the reports we’ve received from people who have begun to receive spam email to addresses that are only utilised on PublicWhip. We despise spammers and can categorically state that we have never, and will never, share or sell the personal data that our users entrust with us. If you have received spam email to an address that has only been used on PublicWhip, would you please let us know the address details (email team@publicwhip.org.uk) to help us identify where the data leak is coming from and to plug it.

One of the main culprits is a very out of date installation of PHPBB and whilst we look at alternatives, and gather input as to the usefulness of the forum (beyond acting as a changelog for divisions and policies), we’ll be taking the forum offline.

We’re still getting to grips with a system and a codebase that is relatively new to us, but we’re determined to make the site the best that it can be, and data security and data integrity will always be at the heart of what we do.