PL077-20: NERC Data Labs Phase 3

Description

Summary of the work Provide a flexible team comprising Business Analyst, implementation team, Test Lead and Project Coordinator to support UKCEH's existing Data Labs leadership and implementation teams. Expected Contract Length 12 months with an optional extension of up to 12 months Latest start date Monday 20 July 2020 Budget Range The estimated value of this opportunity for the initial period is a maximum of £350,000.00 excluding VAT. Should the optional extension be utilised, the total value shall not exceed £700,000.00 excluding VAT. Why the Work is Being Done The goal of Data Lab is to extend NERC's Environmental Data Service offering through utilisation of existing NERC Data Centres as data analytics service providers; allowing researches to have accessible, secure, flexible, cloud-based research environments, combining dynamic big data resources, flexible analytical capabilities, and opportunities to collaborate. Phase 3 of the project allows Data Lab to scale out, in storage and compute resources (on-premise and hybrid-cloud) to meet the operational need of our research communities. As Data Lab expands, so the need to manage scale and complexity of users, content and services will increase, requiring simplified administration and robust data discovery. Problem to Be Solved In supporting the anticipated operational scale of Data Labs (in terms of both infrastructure and complexity of content) five key deliverables must be realised: 1. A platform management interface allowing administration of users, infrastructure and data assets. 2. Access to large, performant and highly scalable storage supporting big data analytics. 3. A federated Data Labs asset discovery service. 4. Implement a job scheduling interface enabling access to HPC resources in JASMIN’s managed-cloud infrastructure. 5. The ability to work beyond the confines of Data Labs current hosting environment (CEDA's JASMIN) if an activity demands it. Who Are the Users 1) As a Researcher/Scientist, I need to undertake computationally intensive modelling, simulation and data analysis activites, and/or access research outputs (data, visualisations, etc.) that are computationally demanding to produce, and beyond the capabilities of my personal laptop, workstation or research centre's own infrastructure. 2) As a scientist, I may need to collaborate with other scientists, make use of previous assets, or make my assets discoverable for future researchers, through a federated Data Labs asset discovery service. 3) As a system administrator, I need to be able to manage a growing infrastructure and user base through simple administrative interfaces. Early Market Engagement n/a Work Already Done Data Lab phase 1 (2017) delivered a cloud-based solution (hosted on JASMIN) allowing scientists to access an intuitive self-service, highly scalable data management and computational analytics capability. This phase focussed on delivering on-demand Jupyter and Zeppelin analysis environments and RShiny-based web applications as demonstrators of the Data Labs approach, underpinned by dedicated private storage supporting parallel data processing by Apache Spark and Dask. Data Lab phase 2 (2019) facilitated increased research collaboration and enhanced lab security through project-based environments. The range of data analysis and publication services also increased to include RStudio. Existing Team The project team includes representatives from the UK Centre for Ecology & Hydrology (UKCEH), STFC's Centre for Environmental Data Analysis (CEDA) and end user research scientists from academia, NERC Research Centres, and wider UKRI Councils. The Supplier will primarily work alongside UKCEH's project leadership and implementation team with significant collaboration also anticipated with infrastructure managers at CEDA. Liaison and engagement with a core group of Data Labs Champions across NERC and UKRI more widely is also anticipated, facilitated by UKCEH's project leadership team. Current Phase Beta Skills & Experience • Management of large scale open source projects • Experience of developing self-service data analytics platforms in both academic and commercial research environments, providing cloud-hosted compute and storage access • Experience of cloud-based technologies and the CNCF landscape; including OpenStack, Teraform, Kubernetes, Docker, Auth0, Vault, GlusterFS, and MinIO • Experience of parallel data processing: Dask and Apache Spark • Experience of execuable notebook technologies: RStudio, Jupyter and Zeppelin • Experience in web-based data exploration tools: RShiny • Experience of developing cloud-based APIs in relevant technologies: GraphQL • Appreciation of common research tools (e.g. R, Matlab, IDL, Python, Java Applications, Audio and visualisation techniques), data formats (e.g. NetCDF, CSV, GRIB, Binary, WAV) and processing needs • Software testing (automated and manual testing) including use of Selenium, Swagger, Postman • Experience in user experience user interface implementation • Experience in metadata design and standards implementation: Dublin Core, DCAT, Schema.org; and application to FAIR data principles • Experience in implementing data catalogue solutions including relational and NoSQL-based approaches: PostgreSQL, SQLlite, MongoDB, ElasticSearch • Experience of providing software solutions that meet complex and varied scientific requirements for exploration, manipulation and visualisation of data • Large data volume software solutions for batch and parallel processing • Leading and working in teams using Scrum and other Agile techniques • Cooperating and collaborating with 3rd party academic research organisations such as STFC (JASMIN) and NERC Nice to Haves • Have a strong scientific or technical background (e.g. a Scientific degree or higher) • Engagement with user communities • Understanding of business process planning and requirements to support provisioning, management and administration of a Data Lab • Experience of delivery of agile projects in a mixed supplier / client team Work Location The majority of work will be undertaken from the suppliers own premises, through remote connectivity and collaboration with the UKCEH team (and other stakeholders as required). UKCEH have well established remote collaboration tooling covering project communication/reporting and technical development activites. The successful supplier will be expected to align with UKCEH's approach. When conditions permit travel to UKCEH research sites (Lancaster, Wallingford) and CEDA (STFC Harwell) are anticipated on an ad-hoc, part-time basis. Occassional travel to other project partner locations may be required in support of community engagement and solution roll-out activities. Working Arrangments The Supplier will largely work remotely and report daily to the project implementation team. UKCEH advocate the Scrum agile project management approach. The successful supplier will be expected to engage in UKCEH's Scrum methodology, including participation in its range of project execution ceremonies. A summary of progress, supplier team utilisation and budget tracking will be provided as a written report at the end of each implementation sprint by the supplier. The Supplier will provide consistent representatives in key roles throughout the project's lifetime, although flexibility within the wider team reflecting needs of the project as it progresses may be necessary. Security Clearance Security clearance is not required Additional T&Cs The supplier will incorporate flexibility into the timetable to enable End User involvement to assist in the development of user stories, acceptance testing, and UKCEH-led engagement activities. The Supplier will accommodate breaks in the timetable for customer led end user engagement, decision making and governance. No. of Suppliers to Evaluate 3 Proposal Criteria • The proposed technical solution • The proposed approach and methodology • How the approach or solution meets user needs • Team structure and organisational makeup • Value for money of the proposed solution • How they have identified risks and dependencies and offered approaches to manage them • Flexibility to delivery the proposed approach to ensure end user engagement Cultural Fit Criteria • Work as a team with members of the project and end users to produce the best outcome • Transparent and collaborative • Collaborating and delivering value in an environmental science research environment • Good record keeping and tracking of progress • Be comfortable discussing complex and varied end user requirements with scientists and technicians and confident in managing discussion and keeping focus • Contribute to the ongoing development and improvement of the Data Lab • Take responsibility for their work • Share knowledge and experience with other team members • Challenge the status quo • Can work with clients with low technical expertise Payment Approach Capped time and materials Assessment Method • Case study • Presentation Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers 1. The deadline for applications is Friday 24 July 2020, but the latest start date is listed as Monday 20 July 2020. Should the start date be the 20th of June? This is an error on the form. The preferred start date is as soon as possible, at the latest the end of July.