Lesson 2.1 What is Shared and Open Data
Learning outcomes
At the end of this lesson, learners will (be able to):
Acquire knowledge about shared and open data
Recognize the value of open data in agriculture
Define challenges of moving to an open data landscape
Understand the benefits of publishing and using open data
1. Introduction
This lesson aims to introduce you the principles of sharing data, what makes data open and the benefits of opening data. While the focus throughout the course is on shareable, structured data, this lesson makes a particular link between shared and open data on a spectrum.
In Lesson 1.1, it was explained that the availability of more data at global and farmer level helps to enhance extension, trade and financial services. These services can increase income and yield. The use of ICT makes it possible to forecast the future much better than ever or to answer seemingly complicated questions much more quickly based on data. Such questions might be: Where does our food come from? Can we manage risks in our farm and take control measures against droughts or pests? Are we able to predict problems such as floods or low yields? Can we make informed decisions on what to grow, what treatment to apply, when to plant, treat or harvest? Technologies today allow us to build services to answer these questions, but data only offers these opportunities when it is usable.
The notion of open data has been around for some years. Considerable amounts of data today are generated by the public sector, e.g. soil surveys, cultivar registrations, pesticide residues, health care, defence industries, infrastructure, public education, and telecommunications. See the categories of datasets presented in Lesson 1.1, including individual datasets accessible on public portals. In 2009, various governments, including the USA, UK and Canada, launched open government initiatives to open up their public information.
In addition to public data, for which there is a general demand for openness, private sector data is also more and more important for decision making: while it is not always feasible to make this data completely open, many of the principles of open data (access, reuse, interoperability) apply also to the sharing of private sector data even if under different access conditions (see the data spectrum in the next chapter).
Open access to research and sharing of data are vital resources for food security and nutrition, driven by farmers, researchers, extension experts, policy makers, governments, international agencies and other private-sector and civil-society stakeholders participating in ‘innovation systems’ and along value chains. Lack of institutional, national and international policies and openness of data limit the effectiveness of agricultural and nutritional data from research and innovation. Making open data and data exchange in the value chain work for agriculture requires a shared agenda to increase the supply, quality, and interoperability of data, alongside action to build capacity for the use of data by all stakeholders[1].
From mobile technology used by health workers to open data released by government ministries, data is becoming ever more valuable, as agricultural business development and global food policy decisions are being made based upon it. But the agriculture sector is also home to severe resource inequality. The largest agricultural companies make billions of dollars per year, in contrast to subsistence farmers growing just enough to feed themselves, or smallholder farmers who grow enough to sell on a year-by-year basis[2].
The scarcity of available data prevents us from identifying and learning from real progress at the global and national levels. It also hides inequalities within countries, making it more difficult for governments to know about them or for others to hold governments fully accountable[3]. National averages are not enough to see who is being left behind, as nutritional levels can vary even within households. Beyond just collecting data, it should be used actively to make better choices and inform and advocate decision-making from the household level all the way up to policy level.
2. Notion of shared and open data
Data exists on a spectrum. It can be closed, shared or open. Datasets may include sensitive information for security, personal or commercial reasons. For instance, health records may cover sensitive data which raise privacy issues. Personal health data is often considered as sensitive. Sensitive data and personal data protection are discussed widely in the next lessons. For these reasons, data can be closed or can be shared with limited persons or groups but not licensed to permit anyone to access, use and share it. The Data Spectrum in Figure 1, developed by The Open Data Institute (ODI), illustrates the degree of openness of data and helps to understand the language of data. Data can be shared within a closed or partially closed group or even publicly on the web without being identified “open data”. What makes it shareable is the structure of data and the machine readability.
To make the data open, the important thing about data is how it is licensed. For data to be considered open, it must be:
accessible, which usually means published on the web
available in a machine-readable format
with a licence that permits anyone to access, use and share it – commercially and non-commercially.
Many individuals and organisations collect a broad range of different types of data in order to perform their tasks. Government is particularly significant in this respect, both because of the quantity and centrality of the data it collects, but also because most of that government data is public data by law, and therefore could be made open and available for others to use[4].
The open data movement has been advocated strongly by governments to allow others to benefit from their data and their desire to be transparent, but research institutions and the private sector also generate data which they are willing to share as a common good[5].
Open data is data that can be freely used, reused (modified) and redistributed (shared) by anyone[6]. The Open Data Handbook emphasizes the importance of the definition of open and highlights key features about open data:
Availability and Access: The data must be available as a whole, and at no more than a reasonable reproduction cost, preferably by downloading over the internet. The data must also be available in a convenient and modifiable form. Managing data can be costly in terms of time and resources needed. An example of costing for data management can be seen at UK Data Service[7].
Reuse and Redistribution: The data must be provided under terms that permit reuse and redistribution including intermixing with other datasets.
Universal Participation: Everyone must be able to use, reuse and redistribute – there should be no discrimination against fields of endeavour or against persons or groups. For example, ‘non-commercial’ restrictions that would prevent ‘commercial’ use, or restrictions of use for certain purposes (e.g. only in education), are not allowed[8].
3. Open data principles
The Open Definition makes precise the meaning of ‘open’ with respect to knowledge, promoting a robust common in which anyone may participate, and interoperability is maximised. Knowledge is open if anyone is free to access, use, modify and share it – subject at most to measures that preserve provenance and openness[9].
Open data must comply with an open licence or a status. It must be in a public domain or under an open licence. Without a licence, the data cannot be reused.
It must be accessible and downloadable via the internet. Any additional information necessary for licence compliance must also accompany the work, such as an attribution to say that people who use the data must credit whoever is publishing it, or a share-alike requirement to say that people who mix the data with other data have to also release the results as open data.
Open data must be in a machine-readable form which is processable by a computer and where the individual elements of the work can be easily accessed and modified. It must also be in an open format which places no restrictions, monetary or otherwise, upon its use and can be fully processed with at least one free/libre/open-source software tool.
The licence used for the open data should be compatible with other open licences. It should permit free use, redistribution, creation of derivatives, and compilation of the licensed work. It must allow any part of the work to be freely used, distributed, or modified separately from any other part of the work or from any collection of works in which it was originally distributed. The licence must not discriminate against any person or group.
The Open Data Charter, which is a collaboration between over 70 governments, agrees on six principles for how governments should be publishing information. Each of them is explained below briefly. On their site, the Charter also provides detailed action items to achieve each of these principles[10].
Open by Default: Free access to and use of government data (data held by national, regional, local, and city governments, international governmental bodies, and other types of institutions in the wider public sector) brings a significant value to society and the economy, and the government data should, therefore, be open by default.
Resources, standards, and policies for the creation, use, exchange, and harmonisation of open data should be globally developed, adopted and promoted so long as citizens are confident that open data will not compromise their right to privacy.
Timely and Comprehensive: Data may require time, human and technical resources to be released and published. It is important to identify which data to prioritize for release by consulting with the data users. The data must be comprehensive, accurate, and of high quality.
Accessible and Usable: Opening up data enables stakeholders to make informed decisions. The data should be easily discoverable and accessible, and made available without any barriers.
Comparable and Interoperable: The data should be published in structured and standardised formats to support interoperability, traceability and reuse. It should also be easy to compare within and between sectors, across geographic locations, and over time in order to be the most effective and useful.
For Improved Governance and Citizen Engagement: Open data strengthens governance and provides a transparent and accountable foundation to improve decision-making and how land markets operate. It enables civic participation and better-informed engagement between governments and citizens.
For Inclusive Development and Innovation: Openness stimulates creativity and innovation. Open data by its nature offers an equitable resource for all people regardless of where they come from or who they are and provides a less digitally divided environment to access and use the data.
4. Benefits for shared and open data
The benefits of open data are diverse and range from improved efficiency of public administrations, economic growth in the private sector to wider social welfare and citizen empowerment.
Performance can be enhanced by open data and contribute to improving the efficiency of public services in health and nutrition. Greater efficiency in processes and delivery of public services can be achieved thanks to cross-sector sharing of data, which can for example provide an overview of unnecessary spending. Resources can be better targeted thanks to local-level, disaggregated data, showing which areas and populations have the greatest needs.
The economy can benefit from easier access to information, content and knowledge, in turn contributing to the development of innovative services and the creation of new business models.
Social welfare can be improved as society benefits from information that is more transparent and accessible. Open Data enhances collaboration, participation and social innovation[11].
The availability of detailed open data is essential to improve delivery of services at the local level. Such cases are mySociety [https://www.mysociety.org], the Hungarian 'right to know' portal, and Fix my Street [http://www.fiksgatami.no] in Norway. To support the emergence of new data-driven businesses and the growth of existing ones, governments need to publish key datasets. By growing economies and improving services, open data allows governments to make savings in key areas, like provision of healthcare, education and utilities. In the UK, open data helped reveal £200 millions of savings in the health service. In France, energy data is being used to drive more efficient energy generation practices[12].
GODAN’s report, ‘How can we improve agriculture, food and nutrition with open data?’, specifies three ways that open data can help solve practical problems in the agriculture and nutrition sectors.
Enabling more efficient and effective decision making. Open data enables computers to pull data from various sources and process it for us and does not rely on humans to interpret and integrate information contained in web pages. Open data underpins new products and services by presenting information from a wide range of sources that helps everyone from policy makers to smallholders find gaps in markets or fine-tune their products or services. A good example of it is in fisheries. In South Africa, the fisher community in collaboration with the University of Cape Town co-designed a suite of apps to support and improve the small-scale fisheries industry. Abalobi Fisher is one of those apps which provides valuable information about the weather and climate from open sources, plus records data about fisher practice and catch information. The hope is that it will showcase small-scale fisheries as a vital and valuable resource as well as a legitimised livelihood, not just to the local communities, but to the country as a whole. A relevant use case can be watched at Open Water – GODAN Documentary Web Series at https://youtu.be/sxY-skVv7Mk.
Fostering innovation that everyone can benefit from. As a raw material for creating tools, services, insights and applications, open data makes it inexpensive and easy to create new innovations. When data is open for all to experiment with, there is no need to invest large amounts in repeating already completed trials. When data is openly licensed, it also allows for novel combinations with other data to gain new insights. A story of Andrew from Allington, UK, who works alongside his family on their arable and dairy farm is a good example of using data to apply precision farming tactics to their land. The family has taken advantage of new tools and technology that allow them to easily view satellite data of their land, which has been opened up by the European Space Agency. His story can be watched at Open Skies – GODAN Documentary Web Series at https://youtu.be/MJ0VX7CqmSc
Driving organisational and sector change through transparency. Transparency around targets, subsidy distribution and pricing, for example, creates incentives which affect the behaviour of producers, regulators and consumers. By requiring companies, government departments and other organisations to publish key datasets – performance data, spend data or supply-chain data, for example – governments, regulators and companies can monitor, analyse and respond to trends in that sector. More importantly, publishing this data across a sector can ultimately transform how products and services are delivered. We can refer here to the same example as given above, Abolobi Fisher from Open Water, of using data about fisher practice for small-scale fisheries industry to make informed decisions.
Providing farmers with more accurate, accessible, timely information – from large agriculture groups to the individual smallholders – will help to ensure food commodity markets function well in future. Progress will be driven largely by providing better access to accurate, timely information for individual smallholder farmers, businesses and policy makers alike. Open data can and should be part of the solution. Open data promotes transparency across the sector to accelerate progress, identify areas for improvement and help create new insights[13].
Agmarknet [http://agmarknet.gov.in] in India is a good example of providing market information with more than 2700 data sources to the farmers, traders, policy makers and other stakeholders for better production and marketing decisions. The rice producer's federation of Colombia keeps data sets historically and helps small and medium growth farmers by measuring climate, yields and farming practices related to rice-growing in the country. You can watch the story of Blanca, who runs her farm outside the town of Ibagué with the assistance of early-warning systems and weather data, at Open Climate – GODAN Documentary Web Series at https://youtu.be/y9SwIpHT5A4.
5. Open data acts as change agent
Open data acts as a change agent. Implementing an open data initiative often involves cultural and institutional change. Opening data goes far beyond putting data on a website under an open licence. Applying the technology is relatively easy when compared with bringing about a cultural change, which can be much harder[14]. It requires consulting with potential data users internally within an institution as well as the external stakeholders.
The same is true for private-sector data sharing projects that may not adopt fully open data approaches but still need to change their attitude towards their data and engage with other actors for its reuse.
However, this difficulty of adopting a change does not stop the amount of data which is increasingly becoming openly available. There are still challenges related to data management, licensing, interoperability and exploitation. There is a need to evolve policies, practices and ethics around closed, shared, and open data.
The challenges involved in opening agricultural data are best addressed at the level of a particular problem in a specific field, where standards can be identified or developed, and data released as part of solving a problem. This is especially true when advocates can point to a clear theory of change. GODAN addresses this issue with care and sets out five strategic steps[16] for pursuing solution-focused open data initiatives for agriculture and nutrition:
Engage with the growing open data community, including key problem owners and experts at GODAN, to identify the challenges that open data can help solve.
Build open data strategies and projects with a focus on finding solutions to land tenure, agriculture and nutrition problems.
Develop the infrastructure, assets and capacities for open data in relevant organisations and networks.
Use open data and support users of relevant data.
Learn through ongoing evaluation, reflection and sharing to ensure we can all continue to improve our practice.
Footnotes
[1] GODAN. Statement of Purpose. November 2017. http://www.godan.info/pages/statement-purpose
[2] Ferris, Lindsay and Zara Rahman. “Responsible Data in Agriculture.” 2016. The Engine Room Library. 2017. https://library.theengineroom.org/agriculture
[3] Global Nutrition Report, 2016 http://globalnutritionreport.org/the-report-2016/
[4] Why Open Data? https://okfn.org/opendata/why-open-data
[5] Gray, J. 2015. Towards a Genealogy of Open Data. Presented at General Conference of the European Consortium for Political Research, Glasgow, UK, 3–6 September 2014. http://dx.doi.org/10.2139/ssrn.2605828
[6] Open Knowledge International. “What is Open Data?” n.d. Open Data Handbook. November 2017. http://opendatahandbook.org/guide/en/what-is-open-data
[7] UK Data Service. Costing Data Management. n.d. November 2017. https://www.ukdataservice.ac.uk/manage-data/plan/costing
[8] What is open data? http://opendatahandbook.org/guide/en/what-is-open-data/
[9] Open Definition 2.1 http://opendefinition.org/od/2.1/en[10] Open Data Charter. Principles https://opendatacharter.net/principles
[11] European Data Portal. Benefits of Open Data https://www.europeandataportal.eu/en/using-data/benefits-of-open-data
[12] Open Data in a Day http://training.theodi.org/InADay/#/id/co-01
[13] Carolan , Liz, et al. “How can we improve agriculture, food and nutrition with open data? https://www.godan.info/documents/how-can-we-improve-agriculture-food-and-nutrition-open-data
[14] European Commission. “Open data: agent of change.” n.d. European Data Portal. https://www.europeandataportal.eu/elearning/en/module3/#/id/co-01
[15] Carolan , Liz, et al. “How can we improve agriculture, food and nutrition with open data? https://www.godan.info/documents/how-can-we-improve-agriculture-food-and-nutrition-open-data
Last updated
