Spatial Hub Update

The Spatial Hub is an emerging resource which will provide a single point of access to quality assured Scottish local authority data in a consistent format.  The intention is to open up access to the wealth of spatial information created and maintained by local government in Scotland and do this in a way that is both sustainable and self-supporting

It is developed, operated and managed by the Spatial Information Service within the Improvement Service and is gradually becoming a reality after a period of planning and development.  We always intended to introduce it through a soft launch and it has appeared through a number of iterations to where it is today.  It is not yet developed to a point where it is fully functioning, but is moving along steadily.  It has received considerable interest and I have introduced it in a number of conference presentations.

There has been substantial support for the concepts and idea behind it, but also some misconceptions about what it will actually do when fully functioning, and what it will deliver.  The intention of this blog post is to clarify (as far as possible) what the Spatial Information Service is doing in creating the Spatial Hub, most importantly why we are doing it, and how eventually we see it working.

Making a sweeping generalisation, local government has never been particularly celebrated for its approach to managing its information and unsurprisingly this also applies to its spatial information.  It could be argued that spatial information is often managed by those from diverse backgrounds and expertise who have subject expertise e.g. land & property, environment or planning) and have drifted into GIS, without perhaps a more rigorous technical background in database design or coding.  I would include myself in this category, being originally a land surveyor.

More often than not the spatial resources available are scattered across departments and thus reflect the “cell based” organisational structure.  The result of the above is that spatial information is often buried in silos, virtually inaccessible (even internally within the organisation by others) and consequently little valued at a senior level and not used in the decision-making process.  The above “sweeping generalisation” is based on many years of working in this environment and confirmed by the extensive spatial data audit undertaken across all 32 councils in 2014 as part of the early planning stages for the Spatial Hub.  It was not uncommon for those who responded to the audit not to know what datasets their council held and full picture emerged only when pursuing more detailed information.

It would be difficult to find anyone who has ever tried to access local authority spatial information (or any other information) who does not recognise the above.  This leads to much frustration and criticism from those wishing to make use of what is extremely valuable information.  It also leads to the roles of those managing the information being devalued, which in these troubled financial times carries a hidden but very real danger. 

Scottish local government is facing a projected deficit of over £1billion per annum between projected outgoings and income over the next few years.  The recently announced budget cuts to local authority budgets are in the order of 5% and given that education and social services are ring-fenced, support services are disproportionally affected. This is of course where the “niche” spatial data managers work and these jobs are already being reduced with any vacant posts being unfilled.  Some authorities are already struggling to maintain spatial datasets which are core to their decision making process.  It has been described as setting out on a path to what I have heard described as “organisational ignorance”, at a time when the requirement is to work smarter and make services more efficient.

Inspire then sits on top of this heap of woe.  Local authorities, like all public bodies, are obliged under the EU regulation (and Scots law so Brexit won’t get us off the hook) to make their spatial data available for discovery, view and download.  Discovery alone is challenging where no one really knows what datasets the council actually owns (using the Inspire terminology – I prefer the term “custodian” to owner) and the Scottish SDI metadata records are far from complete and likely to remain so given the above lack of resources.  As far as view and download services are concerned, a few better resourced authorities are able to meet this requirement but many are struggling.  Licensing this almost entirely Ordnance Survey - derived data also presents problems, in whether to make data available under OGL or Inspire licences. Additionally, many authorities are reluctant to make their datasets available as they acknowledge their poor quality and are concerned of potential liability, but are being guided by the Scottish Government’s Open Data Strategy to do so.  These often competing influences result in the landscape of published data across local authorities varying significantly.

The above demonstrates pretty clearly that there is a significant problem which is only going to get worse unless something can be done about it.  This is where the Spatial Hub comes in.

The idea started back in 2003 under the modernising government fund where some £7.5M of funding was provided to create what has now become the One Scotland Gazetteer (OSG).  Local authorities were recognised as the most effective custodians (sorry Royal Mail) for address information through Street Naming and Numbering, Development Management, Building Standard and Council Tax.  These address registers were conflated, de-duplicated and then used to create local gazetteers called Corporate Address Gazetteers.  These conformed with BS7666 and the Scottish Gazetteer conventions and were then quality assured and aggregated to create the OSG.  Of course it wasn’t perfect and local variations crept in but it was, and remains, the authoritative and most up to date list of addresses available for Scotland.  It is most commonly accessed through OS AddressBase

An idea was formed to apply what was done for the OSG to all Scottish local authority spatial information.  The 2014 spatial audit identified that each authority had about 70 spatial datasets and given that local authorities in Scotland perform largely the same functions it was reasonable to assume that they more or less held the same datasets – even if the called them different names.  The idea was to create national (Scottish) datasets of local authority spatial information and release the significant value buried away in authority silos.  A regime of quality assurance would be put in place with the intention of driving up the quality of individual datasets.

The Spatial Hub provides a mechanism for each authority to upload its datasets, where they are processed and then made available for discovery, view and download.  This would then meet all 32 authorities’ obligations under Inspire. After discussion with the Scottish Information Commissioner we were advised to produce a formal agreement with each authority outlining both their and the Improvement Services obligations.  This has now been actioned.

At the moment access to quality assured national datasets of local authority data via the Spatial Hub is only available to One Scotland Mapping Agreement members until we finally sort out the necessary licensing arrangements.  Work is on-going in opening up access so that all can share in the value created.

The benefits are fairly limited for each authority as the data is held to support their own business processes. However, meeting Inspire has some benefit and the statutory requirement has raised the profile of spatial information to senior levels, something which has helped with engagement and participation.

 This processes which underpin the Spatial Hub are complicated as locating the actual datasets and “custodians” is challenging.  There is a need for data management plans for each dataset and these require development over time to ensure that the national layers are maintained in a consistent manner.  The Spatial Hub is cloud based and built almost entirely using open source products.

What we are doing is opening up access to good quality Scottish local authority spatial data so that all can benefit from it, rather than it festering in silos as before.  However laudable this may appear, it must be recognised that there is a cost associated with doing this and therefore it needs to be funded from somewhere.

Many councils, but certainly not all, are happy to make their local spatial data available on Open Government Licences.  This does require agreement from Ordnance Survey but several local datasets have been made available in this way.  Our market research has identified that many organisations (both public sector and commercial) struggle to access the local authority information required for their businesses.  OSMA members already make a contribution to the Spatial Hub funding as many incur considerable cost in gathering and collating information and will benefit from the Spatial Hub.  Commercial businesses who have built value added service on top of local authority spatial datasets have indicated that the Spatial Hub will save then considerable expense and that when the service becomes fully available they would be willing to pay for commercial use of the valued added datasets that will be provided.

The idea is that the national datasets will be available under an Inspire licence which provides “open” access for non-commercial purposes and that we will endeavour to recover our costs through a number of commercial subscriptions, potentially with a number of partners.

The argument which the Improvement Service Board, representing Scottish local authority interests, makes, is that this hybrid model allows all to benefit from the Spatial Hub in terms of opening up the data and improving its availability and quality.  This will be done in a way which allows the commercial sector to reduce its costs and improve profitability but at the same time support the sustainability and on-going management of the core data against a background of reducing resources and capabilities.

Several members of the open data community have commented on the proposal and it is fair to say that everyone praises the Spatial hub as an excellent way to open up access to local authority data but that they question why it is not made available, free of charge under an Open Government Licence.  The simple answer to this is that there is a cost in generating the value that the Spatial Hub adds to the raw data from councils and this has to be met from somewhere.  The Improvement Service is uniquely placed to do this on a cost effective basis, being owned by all 32 councils in Scotland which enables engagement from all and with an active community of data custodians within councils.  If the commercial world then benefits by reducing their costs, it seems only reasonable that they contribute towards a share of the costs. In order to make the data open and free of charge for all, funding would be required and in a time when every budget is being cut, this is not forthcoming.

Security level: Public

More Blog Entries

Update on IS Spatial Data Hub

Things have been ramping up with the Improvement Service’s Spatial Information Service over the...