RM1043.8-1-Cabinet Office Data Marketplace Service Development & Integration (v2)

Description

Summary of work The Data Marketplace (DM) will be a new government service that provides a front door to discover, access and share government data in a legal, ethical and trusted way. The primary goal of the government DM is to improve both the data Acquirer’s and Supplier’s productivity by removing or reducing as much as possible the current pain points. This includes the following business objectives: 1) Enable users to easily find data available elsewhere in government. 2) Allow data sharing to be set up more efficiently, effectively, transparently and in a trusted way. 3) Support government organisations to create reusable data services that are interoperable with other government services. 4) Provide easy access to support, guidance, standards and documentation for data use and sharing. 5) Provide a holistic view of discovery and sharing of data facilitated by the Data Marketplace with metrics on the gaps and barriers to data sharing to improve the government data ecosystem. To achieve the above the following technical and process objectives need to be addressed: 1) Efficient cross-government publication and harvesting of standardised metadata from departmental data catalogues and other repositories. 2) Standardising a consistent workflow for data sharing and integrating this into existing departmental data sharing processes. 3) Providing an Identity and Access Management solution to authenticate users and to authorise access to the relevant catalogue and workflow services. In addition the following delivery objectives are within scope: 1) Resources necessary to manage the delivery and risks associated with the development of the service. 2) Informative products and knowledge transfer sessions with existing staff to ensure continual knowledge transfer throughout the life of the contract. 3) Design of future state support and maintenance required for ongoing operation of the service. 4) Business analysis and stakeholder management to define the change required by suppliers/departments of process and technologies in order that they can be onboarded to the Data Marketplace.. 5) User research to understand user needs, jobs to be done and individual pain points and inform the design and development of the Data Marketplace. Where the supplied staff will work No specific location (for example they can work remotely) Why the work is being done The Data Marketplace will be a new government service that provides a front door to discover, access and share government data in a legal, ethical and trusted way. It will initially focus on sharing between public sector organisations, but there is also exploratory work underway to understand wider industry requirements for access to public sector data and the role that the Data Marketplace could play. The business problem With the proliferation of government digital services, there has also been a proliferation of data capture (rather than re-use) and data storage (rather than from central master sources). Through research undertaken as part of the discovery phase, we have identified that these problems arise from the processes around finding data in government, assessing how useful data is, arranging to share data and setting up appropriate agreements. Namely that: Discovery of data available from other departments, what it broadly contains, and where and how to get it is difficult and time consuming. The process of arranging a data share can take too long due to the differing standards, demands, and processes of different departments. When users are assessing whether the data is fit for purpose, it can be difficult to assess it against their needs when there is insufficient metadata. Access to data may be via manual or one off export. The data sharing problems identified are faced by all organisations across government and are priorities to resolve to help us to achieve the ambitions set out in the Digital and Data Roadmap and the National Data Strategy to make better use of data in decision making and delivery of public services. . The people who will use the product or service User type: Data Acquirer Definition: DISCOVER: • The ability to search for data by keyword, category, or other criteria. • The ability to filter search results. • The ability to see the relevant details about available datasets, such as the size, format, and quality of the data, including lineage. • The ability to understand the methods available for accessing the data, such as API, CSV upload, or direct download. • The ability to understand the schema of the dataset. • The ability to provide feedback on a dataset • The ability to know what the conditions of access are (like a data sharing arrangement) • The ability to experiment with data in a sandbox/sandpit to see whether the data will meet their need SHARE: • The ability to raise a request enabling collaborative access to one or more datasets. • The ability to receive notifications about status updates for the request. DELIVER: • The ability to be directed to specifications and documentation for supplier APIs. • The ability to initiate the process for obtaining the necessary credentials to access a supplier API. • The ability to initiate the process for obtaining supplier file based data assets. User type: Data Supplier Definition: DISCOVER: • The ability to publish their metadata on the marketplace so that it can be found and used by acquirers. • The ability to choose the way they publish their data content, such as API, CSV upload, or direct. • The ability to control who can access their metadata. SHARE: • The ability to understand the acquirer's purpose for acquiring the data, ensuring a good fit. • The ability to assess the potential benefits and risks of sharing the data. • The ability to communicate with a single person on the requester side. • The ability to communicate with more than one person for those x-gov requests • The ability to understand the acquirer's plans for storing and disposing of the data, as well as the intended use of the data. • The ability to be aware of the relevant legal requirements for data sharing. DELIVER: • To support development of APIs (or other suitable interfaces) to securely provide data to acquirers. • The ability to choose the way they deliver their data assets, for example, through an API or as a CSV file. • The ability to check that their API specification conforms to government standards. • The ability to check their file based data assets conform to government standards. • The ability to extract schema information from file based data assets to include in metadata records. User type: CDDO Definition: DISCOVER: • The ability to maintain a metadata catalogue that supports federated content harvesting via an API. • The ability to manage a catalogue solution for metadata. • The ability to utilise a search solution for indexing metadata. • The ability to ingest metadata from external catalogues. SHARE: • The ability to automate the flow of tasks, documents, and information. • The ability to integrate workflows with catalogue searches or data shares. • The ability to notify relevant actors about specific events. • The ability to generate metrics about the number of data share requests created, approved, rejected, and pending, and the time taken to complete those. • The ability to have a role based access control for users. • The ability to establish SLAs indicating when requests should be completed. DELIVER: • The ability to gather metrics from suppliers around access and use of their APIs GUIDE: • The ability to provide a content management platform for self-serve access. Work done so far Working with a previous supplier, a full set of requirements have been developed as has been an MVP and extensive user research artefacts. Which phase the project is in Alpha Existing team The CDDO Data Marketplace team consists of colleagues across Data Architecture, Data Security and Data Protection and report to the Chief Data Architect. There is a Senior Product Owner, Senior User Researcher, Business Analyst, Delivery Manager, DevOps Engineer and Senior Engineer dedicated to the project with additional colleagues involved as required. There was a previous supplier assisting with work to support development of a prototype and associated User research but they have exited the project Address where the work will be done The Whitechapel Building, 10 Whitechapel High Street, London Working arrangements Work is expected to be mainly conducted remotely or at supplier's premises. The Supplier's team(s) will be expected to work with the Buyer using Agile development & delivery principles. Some travel will be required to attend workshops and meetings, usually in London at Buyers offices or other government buildings. All expenses must be claimed in accordance with the prevailing expenses policy operated by the Buyer. Security and vetting requirements Baseline Personnel Security Standard (BPSS) More information about the Security requirements: BPSS as default. Some roles may require SC Latest start date 8 January 2024 Expected contract length Contract length: 1 years 3 months 20 days Optional extension: 0 years 7 months 14 days Budget Indicative maximum: £4000000 Indicative minimum: The contract value is not specified by the buyer Contracted out service or supply of resource? Contracted out service: the off-payroll rules do not apply Terms and acronyms Term or acronym: DM Definition: Data Marketplace, the collective product being delivered by this project Term or acronym: CDDO Definition: The Central Digital and Data Office (CDDO) is part of the Cabinet Office and leads the digital, data and technology (DDaT) function for government Term or acronym: MVP Definition: Minimum Viable Proposition. This is an initial version of a product deemed suitable for wider release.