How to evaluate the maturity of an Open Data project?

Open data are essential for the development of many strategies in Europe on how the Member States should be develop in the future. In previous articles we talked about various aspects of open data, such as the 8 principles on which they are based. Today, from OGoov we are going to explain how to measure the degree of maturity of any open data project through the UNE 178301. Let’s start.

What kind of maturity model?

As with any learning process, to achieve a degree of openness that meets the required objectives, certain levels of compliance are involved. That is why it is necessary to conduct an assessment of maturity to face the constant evolution.

While the evaluation can be performed following different methodologies, the question of public policy indicators still misses a model that is able to have widespread consensus worldwide. And much more international regulations, but we find proposals for improving policies for open data designed for generic use, as used in his studies by the European Data Portal or the Open Data Maturity Model, published by the Open Data Institute UK.

So, how to measure and evaluate open data programs? What kind of model, what tools can help us? Is it hard to do?

If our goal is to assess the degree of maturity of an open data program for a Spanish authority, we refer to the first rule of Smart Cities, UNE 178301 on Open Data, published by AENOR (Asociación Española de Normalización y Certificación) and involved in a series of documents that will allow the Spanish cities to become smart cities.

This post will summarize the steps referred to the maturity model, designed to help organizations measure their effectiveness in public and the use of open data.

UNE 178301, open data project evaluation tool

Open Data Project

The UNE 178301 tells us how to evaluate the maturity of an open data project. Being a technical standard, it is important to keep in mind that it is a document produced from the consensus, take into account that both experience and technological development, are being approved by a recognized standardization authority.

In this case, measuring the degree of maturity of a open data project depends on indicators and metrics related to sustainability criteria, quality, effectiveness and efficiency of the same initiative. Therefore, measuring the maturity of a open data project will require a move forward in the direction of meeting objectives.

And, as in any process, specific targets should be established to measure the maturity of an open data initiative. This involves setting objectives for macro and micro in a controlled manner. Always within a clear program that includes elements such as those specified in the standard, in order to facilitate the evaluating metrics. 

Metrics, domains and dimensions

The standard, meanwhile, establishes how to measure the publication of open data of a city in the form of metrics and an indicator to measure the maturity of open data, in order to facilitate their reuse in the smart cities.

The metrics are organized in forms of domains and dimensions to facilitate structuring and understanding: 

Strategic domain

Establishes the criteria to measure the capacity and implementation of the strategic policy of the agency to articulate a consistent vision of open data. The dimensions are strategy, leadership, service commitment and economic sustainability.

Examples of the aspects are evaluated in this domain, if the strategic plan for opening data is documented or not, or if you have been assigned to the functions of opening data to a policymaker.

Legal domain

Referring to the criteria for measuring the entity and the evaluation of the rules facilitating the implementation of policies and activities. The legal dimension is divided into external and internal regulations and conditions of use and licensing.

Examples of this domain would be to evaluate whether the project complies with current mandatory legislation and the recommended licenses or conditions for the use of the data to allow reuse.

Organizational domain

Establishes the criteria for measuring the ability to implement management and training activities consistent with the planning and open data strategy.

The dimensions are the organizational (to see if there is a unit responsible for opening data, team work and training, inventory and priority) and the measurement is subdivided into the compliance measurement process and measurement of usage and impact.

Technical domain

Establishes the criteria for evaluating activities to ensure protocols and mechanisms which will turn in facilitate, among others, the availability of data and activities related to the publication of the catalogue, the management of data quality and interoperability degree.

The dimensions are availability (presence in the catalogue of public information, sets documented data, categorization and search, availability and persistent and friendly URLs), access (accessibility/non-discrimination, free access systems), quality of the data (raw data, complete data, documented data, technically correct data, geo-referenced data and linked data) and update (update process, frequency of updating and expanding data sets). 

Economic and social domain

Establishes the criteria for evaluating the mechanisms linking producers of data (agencies) with re-users, sharing common structures that encourage the use of data in the production of new goods and services, the degree of involvement of the organization in the encouragement and assistance to the work of re-agents, the degree of listening and adapting to the demands and the level of dialogue established.

The dimensions are reuse (amount of data, data format, vocabularies standard data), participation and collaboration (transparency, participation and collaboration, resolving complaints and conflicts, promoting reuse and initiatives developed reuse) dimension.

Each metric has associated four levels, from 0 to 3, with results as, “non-existent”, “emerging” “existing” and “advanced”. Each level is assigned as a score, established consecutively, so that the level 0 will also score 0, level 1 will correspond a score of 1, and so on.

When reaching level 3, it is considered that the measured values ​​are expected when there is a formal open data initiative that implements best practices.

If we want to know how to calculate it directly from the regulation, you need to go to the  section 5 where open data is or to the Annex B where an illustrative example of calculation is presented.

Consider that all metrics do not have the same weight, because its impact on open data initiative is not comparable. Within each dimension, however, scores must equal to 100 percent. Be that weight, set as a percentage for each dimension, which must be associated with a score of 0 to 3, in order to obtain a value after applying the formula: Value: ((Score * Weight) / 3) * 100 that will be the score for each dimension.

Later, we will add the values ​​for each dimension (the levels achieved in each of the metrics) to obtain the total score. Once established, we will get a number that will measure the state in which the initiative is open data.

It will be a value between 0 and 1000, corresponding to 0 to 200 level 1, 201 to 400 Level 2, 401 to 600 3, 601 to 800 4 and 801 to 100 5. According to the norm,  3 is the desired  figure that marks the desired threshold for a smart city.

More on the UNE 178301

In addition to allowing measure the maturity of open data project, UNE 178301 facilitates the reuse of data generated or held by the public sector recipients of that information, including citizens or companies providing public services.

In turn, that rule establishes a list of datasets listed as priority in these initiatives, accompanied by a series of recommended vocabularies. The goal is again to be practical, facilitate documentation and implementation of open data projects.

This standard is part of the standardization strategy for smart cities developed by the Technical Standards Committee AEN / CTN 178 “smart cities”, aimed at helping their development, in which AENOR is working with the Ministry of Telecommunications and Information society (SETSI) of the Ministry of Industry, Energy and Tourism.

Leave a Reply