Data Management and Compliance Reporting

Description

The Department for Environment, Food and Rural Affairs (Defra) will be inviting proposals to provide the Management of ‘Air Quality’ Data and the Formal Reporting of Air Quality for the ‘Data Management and Compliance Reporting’ contract. This project will be split into two lots. Lot 1: Delivers the collection, aggregation, storage, management, and dissemination of air pollutant monitoring data. Monitored air quality data is collected through two routes a) via a self-service portal (provided by the Contractor) for non-automated data and b) automatic receipt and upload as well as being harvested from known URI’s. For the automatic data it is initially termed “provisional” until data quality checks confirm it as ratified. Modelled meteorological data as well as UV and stratospheric ozone is also acquired and stored. Aggregations and statistics are calculated, including exceedance statistics against UK statutory limits. All changes to the data are logged, and all site metadata and measurement configurations are stored. The data is then made available to various authorities, scientists and academics, and third-party data modelling suppliers, for various purposes in various forms. Pollutant data is to be made available in specialist formats for formal reporting, daily “air quality” forecast validation as well as other uses. Data is made available as R objects, XML files, RSS feeds, APIs, and to support gov.uk services being developed and the legacy UK-AIR web service. Lot 2: Delivers the formal reporting of air quality across the UK as required under the Air Quality Standards Regulations (AQSR) (2010) and the Air Quality (Miscellaneous Amendment and Revocation of Retained Direct EU Legislation) (EU Exit) Regulations (2018). E-reporting is required annually by 30th September each year in a pre-defined format that meets the requirements of the INfrastructure for SPatial InfoRmation (INSPIRE) Regulations. Rules around reporting are defined within Implementing Provision Decisions . The results will be assessed, and accessible summaries and technical reports of compliance will need to be produced. Data is made available as XML files, shapefiles and csv’s. Data is also reported to the European Monitoring and Evaluation Programme (EMEP) . Defra requires a supplier or suppliers to carry out the services detailed in both Lots for the Data Dissemination Unit for 3 years between April 2026 to March 2029 with a potential for 2 x 1 year extensions. Further information on Lots 1 and 2: Lot 1: The work to be delivered by the successful Contractor is as follows: • Host a data providers portal and create templates for data submission (for non-automatic data). • Host an annual data providers workshop and send reminders to ensure data is provided to defined timescales. • Manage automatic and non-automatic data acquisition, acquiring data either through the Data Providers Portal or harvesting from known URI’s. • Collect all site metadata and measurement configurations. • Calculate statistics and aggregations on an hourly basis, or overnight as appropriate. • When data is ratified (by QA/QC Contractors), collect this ratified data and update the data in the database accordingly. Log all changes to the data. • Undertake consistency, integrity and reality checks on data flows. • Store all data and demonstrate how this is done securely and sustainably. • Make this data available to various authorities, scientists and academics, and third-party data modelling suppliers, for various purposes in various forms. • Data is made available as R objects, XML files, RSS feeds and API’s. Data is also to be made available to support the legacy UK-AIR web service. The skills required for Lot 1 include: • Proven ability to manage large and complex datasets, including routine and ad hoc calculation, aggregation, and storage of statistics in accordance with defined standards. • Demonstrated experience working with air quality data, including its collection, processing, and interpretation, or a proven ability to quickly gain domain expertise in air quality datasets. • Strong data engineering and management skills, including the ability to design, implement, and support modern data delivery mechanisms (e.g. APIs, XML feeds). • Ability to respond to technical and non-technical data queries quickly and accurately, demonstrating strong problem-solving and communication skills. • Expertise in producing accurate, timely data outputs fully compliant with legislative and regulatory requirements. • Experience in managing data efficiently and sustainably, ensuring data systems are robust, scalable, and maintainable over time. • Track record of managing external data providers, ensuring data completeness, accuracy, and delivery in line with defined timescales and contractual obligations. Lot 2: The work to be delivered by the successful Contractor is as follows: • Gather supporting information, data and metadata relating to e-Reporting. • Produce and quality assure all data flows required for e-reporting against the AQSR (2010) and EU Exit Regulations. • Deliver on this reporting by 30th September each year in a pre-defined format that meets INSIPRE requirements. • This requires reporting against air quality zones, assessment regimes, assessment methods, validated and provisional measurement data, validated modelled data, and attainment against objectives. • Each dataflow is required as an XML file or shapefile. • The measurement data and aggregations are to be made available as csv files. • Analyse the results and produce an annual compliance assessment report, with visualisations, trends and an accessible summary document. • Report data from ‘supersites’ and five rainwater sites to the EMEP on an annual basis. The skills required for Lot 2 include: • Extensive knowledge of air quality data, including structure, sources, and standards, as well as the reporting requirements under UK and/or EU legislation • Strong data management and processing capabilities, including demonstrated ability to generate and validate XML files, shapefiles, and other structured data formats required for statutory reporting. • Proven experience collaborating with technical specialists, such as air quality modellers, emissions inventory teams, and data providers, to support accurate, timely and coherent data submissions.