Post: 12 September 2013
As I’m sure we all know, back in 2008 the European Union made a regulation requiring all Member States to publish details of legal persons who are recipients of subsidy payments under the Common Agricultural Policy (CAP), on a single national website.
This was widely hailed as a major victory for transparency. (I expect. I don’t have a link, but let’s assume.)
So of course a couple of years later the European Court of Justice decided that publication of the names of “natural persons” receiving CAP payments was “disproportionately” transparent. The EU made an amended regulation requiring the redaction of the names of the aforementioned natural persons.
In the UK the current state of play is that Defra publishes two years of CAP payments data on a dedicated website:
http://cap-payments.defra.gov.uk/.
Names of “natural persons” are redacted, but company names and all payment amounts are included. The data is searchable and available as a bulk download for each year. The current data covers the periods from 16 October 2010 to 15 October 2011 and 16 October 2011 to 15 October 2012.
As explained on the website, the information remains available for a period of two years from publication and is then removed. That’s a requirement in the original EU regulation.
There are metadata records on Data.gov.uk for the 2008 and 2009 datasets, but Defra has diligently added a note to each explaining why the datasets themselves are not available. So in principle, from the UK Government at least, we only have the most recent two years’ CAP payments data available on a rolling basis:
2012
2012_All_CAP_Search_Results_Data_P14.xls
2012_Intervention_Data.xls (public storage transactions)
2011
2011_All_CAP_Search_Results_Data_P14.xls
2011_Intervention_Data.xls (public storage transactions)
However!I have been poking around on the Defra website and in the National Archives repository, and found additional dataset files going back to 2007. None of these are accessible via live UK Government webpages, but they are all available for download if you have the direct links:
2010
On the live website:
2010_All_CAP_Search_Results_Data_P14.xls
Description: “The details included … (legal and anonymised natural persons data) relate to all CAP subsidy payments made to relevant beneficiaries during the financial years 16 October 2009 to 15 October 2010.”
2010_Intervention_Data.xls (public storage transactions)
Archived on 6 April 2012:
2010_CAP_Search_Results_Data.xls
Description: “The details included relate to all CAP scheme payments, including the Single Payment Scheme, made to beneficiaries who are legal persons, during the [year] … 16 October 2009 to 15 October 2010.”
2009
Archived on 6 April 2012:
2009_CAP_Search_Results_Data.xls
Description: “The details included relate to all CAP scheme payments made to relevant beneficiaries during the financial years 16 October 2008 to 15 October 2009 … The total amount received by each beneficiary is analysed between three headings: direct aids (eg the Single Payment Scheme), market schemes and rural development measures.”
CAP_BEN_INTERVENTION_COSTS1a11.XLS (public storage transactions)
2008
On the live website:
2008_All_CAP_Search_Results.xls
Description: “The details included relate to all CAP scheme payments, including the Single Payment Scheme (SPS), made to beneficiaries during the years 16 October 2007 to 15 October 2008.”
2007
Archived 9 April 2009:
All_RD_CAP_Search_Results.xls
Description: “The details included relate solely to beneficiaries of rural development schemes during the period 1 January 2007 to 15 October 2007. No details are currently shown for the Single Payment Scheme or other ‘Pillar 1’ schemes.”
… so what?
I’m not terribly familiar with CAP payments data myself, and I’ve no idea whether the continued availability of the older datasets is in contradiction to the EU regulations or the ECJ decision. As a practical matter, once useful information is published online it tends to remain in circulation somewhere or other.
There is a website, FarmSubsidy.org, that displays UK CAP payments data going back to 1999. That data is not very clearly attributed and the site doesn’t want me to download the bulk datasets.* However the sources seem to be a mix of the datasets from the Defra website and other information obtained under Freedom of Information.
My preference is always to get public data directly from the primary source where possible. Defra says on the Cap Payments website:
The UK Government remains committed to full transparency in the use of public funds, including the publication of details about all payments made under the CAP, whether to legal or to natural persons. The European Commission has stressed that the current situation is an interim arrangement. The UK is actively pressing the Commission to bring forward comprehensive proposals to achieve greater transparency in the longer term.
The older CAP payments data may have been left online as the result of an administrative oversight. However equally it may be that Defra has complied with the letter of the EU regulations by removing the links from the live website after the specified two year period, and left the datasets themselves online for the benefit of those of us with the interest to hunt them down.
Cow picture is public domain from Wikimedia. Represents the extent of my understanding of the Common Agricultural Policy.
___________________________________________________
*
Update on CAP payments datasets from FarmSubsidy.org
Jack Thurston has kindly provided some guidance on alternatives for accessing the Cap payments data underlying FarmSubsidy.org. (I gather there may be some temporary issues with the download functionality that I tried previously on the website.)
There is an API here that you can use to access the data:
https://github.com/openspending/Farm-Subsidy/blob/master/web/api/templates/documentation/search.html
and bulk datasets can be downloaded from here:
The bulk data is zipped into tar.bz2 files. If you are specifically interested in data related to Britain you will want the GB file.
The unzipped bulk data includes three files:
There is no attribution or licensing information included with the bulk data files themselves. Re-users should note that some of the underlying data was obtained under FOI and/or EIR. The FarmSubsidy.org FAQ indicates they are applying the Open Database License (ODbL) to data obtained from the website. ODbL is an open data licence according to OKFN’s Open Definition; however re-users should bear in mind that this licence includes a share-alike condition, so some types of re-use may be prohibited.