DfE Data Engineering Delivery Capability

Description

Summary of the work We require a development partner who will be able to provide the technical capability on digital and technical projects as part of the DfE’s Data Transformation Portfolio. You will provide expertise and specialism to boost capacity and enhance capability, working alongside our in-house digital and data teams supporting delivery. Expected Contract Length 2 years Latest start date Saturday 1 April 2023 Budget Range Up to £8,000,000 ex VAT Why the Work is Being Done Data Directorate is undergoing a transformation programme to cultivate our in-house skills. This will help us achieve our strategic goal to offer ‘Data as a Service’, modernising our data collection services and laying foundation for good data governance practices; all of which will allow delivery of high-quality data into our estate enabling greater insight into the sector as well as drive data driven policy and operational decision-making Problem to Be Solved The DfE Data Directorate is looking for a supplier to support a range of user-led data and digital projects, including both existing and new projects, from inception through to live. The supplier will work with blended teams, using agile to deliver outcomes based on user needs and, where applicable, government service standards. The supplier should have a good grounding in large, complex data delivery programmes covering data ingestion, cleansing, transformation and data matching, as well as strong experience in digital delivery to GDS standards. The data pipeline projects will predominantly require: Data Engineers skilled in Informatica DEI and Azure Data Factory/data bricks; QA automation testers (Skills: Azure, Specflow, Rest Sharp, C#), Cloud Infrastructure Engineers who are skilled in Azure and Data Analysts with data visualisation skills. The Digital Projects will predominantly require: Software Developers skilled in C#. net but with some capabilities required in REACT, JAVA; QA automation testers QA automation testers (Skills: ETL Testing, Specflow, Rest Sharp, C#), cloud infrastructure engineers skilled in Azure and DevOps engineers Typical projects include supporting DfE's policy and operational response to the pandemic and recovery by bringing more attendance data rapidly into the department to fill critical gaps Who Are the Users This contract will cover a range of data services for which we have a wide range of internal and external users who contribute to and consume our data pipelines and Digital services. Internal users include policy teams, Departmental analysts and corporate services access our data through analytical data stores and Bi reports Other Internal Users include: Analysts, Policy team, Operations, Service Delivery team As a data collector working in Data Engineering and Operations division, I need to setup data collection in line with department policy External Users including: Academies and Multi Academy Trusts Schools Local Authorities (Education & Children's Services) e.g. As an academy I need to be able to submit and validate my school census return and check my data. Early Market Engagement None Work Already Done There are a number of projects and programmes in flight, that are at various stages of the delivery lifecycle requiring data engineering, software engineering, test engineering and Infrastructure resources. These range from the improvement of existing developed systems to transformation leading to new systems. Examples of completed and ongoing work includes • Search for schools and colleges to compare https://www.compare-school-performance.service.gov.uk/ • Get Information about Schools https://get-information-schools.service.gov.uk/ • Get Information about Pupils Existing Team The supplier will work with blended project teams comprising of permanent civil servants and other supplier teams across multiple strands of work for the data engineering service delivery. Typically, within agile delivery teams, working with service owners, product managers, delivery managers, business analysts, user researchers, architects, data and software engineers, Infrastructure Engineers, QA testers and users. You may collaborate or work with other suppliers who are providing capabilities as part of a team. Our multi-disciplinary agile teams stretch across 9 squads and are typically sized between 8 and 10. Current Phase Not applicable Skills & Experience • Technical agile delivery experience in end-to-end design, development, test and deployment of complex CI/CD data pipelines, using Informatica DEI, ADF & Databricks. • Experience and skills in delivering new cloud data platforms, covering ingestion, transformation, validation, cleansing, matching and storage in a complex, high risk environment, to tight timescales. • Experience and skills in developing full stack digital services using C#.net, REST API build and integration and SQL/non-SQL databases in a cloud environment. • Experience and skills in delivering large digital and data programmes successfully, for example to GDS guidelines, progressing successfully through different phases of the delivery lifecycle including governance and assurance processes • Experience and skills in delivering automated CI/CD integrated data and software testing in large, complex, data intensive organisations within central government or regulated sector using Informatica DEI and ADF tools. • Experience and skills in delivering data lakes, data warehouses, streaming data ingestion and processing, integration with SQL/NoSQL databases, structured, semi-structured, and unstructured data sources, temporal data and time-series analysis • Experience and skills of working well in large organisations, across organisational boundaries – for example public sector or multi-agency departments or similar-departments, delivering domain driven service design/architecture Nice to Haves • Awareness and understanding of the Education sector • Awareness and understanding of educational datasets Work Location Manchester ( and additionally Newcastle / Sheffield / Darlington /Coventry / London ) Working Arrangments For any SoW the work is likely to be in Manchester, Darlington, Coventry, Sheffield or London. Some research will be required in other England locations . Travel may be required to DfE offices in these locations. The primary base will be detailed in each Statement of Work (SoW) and expenses will not form part of the cost unless the supplier is asked to work from a secondary location instead during the work relating to the SoW. e.g. If SoW primary base is Darlington then no expenses will be paid for Darlington. Inside IR35 expected. Security Clearance Proposed team members will have been subject to Baseline Personnel Security Standards checks. However higher levels of Security Clearance maybe required dependent on work agreed. Suppliers will also be required to complete a supplier security assurance form to ensure they meet the required standard for DfE e.g. Cyber Essentials Additional T&Cs Standard Framework and Call Off Terms and Conditions. Expenses must be pre-agreed and comply with CCS Travel and Subsistence Policy. Any expenses shall be submitted in line with DfE standard T&S policy. Primary work location stated in SoW will not attract expenses. Contract and Vendor Management will form a key part of governance and suppliers will be expected to complete a security questionnaire. Suppliers must provide sufficient guarantees to meet the requirements of GDPR in line with Procurement Policy Note 03/17 Changes to Data Protection Legislation & General Data Protection Regulation No. of Suppliers to Evaluate 5 Proposal Criteria • Within the last three years to have experience in end-to-end design, development, and agile delivery of digital products/services to high quality standards (in a digital, data and technology driven environment. • Within the last three years have experience in delivery of new data platforms and data migration projects in a complex and high risk environment. • Within the last three years have demonstrable experience in user-centred design services (user research, content design and interaction design and business analysis) • Proven experience of using a range of agile tools e.g. Azure Dev Ops and techniques including scrum, working in multi-team agile programmes • Evidence of coaching, building capability and upskilling of resources • Manage the full product-lifecycles including business analysis, user research, design, delivery, and the continuous improvement of one or more transactional services or platforms including rapid prototyping and delivering at pace • Evidence of demonstrable experience within the last three years in working on high-profile complex, multi-stakeholder digital, data and technology programmes • Evidence of establishing an Agile delivery capability in both data and software delivery lifecycles • Experience of working with an organisation at the beginning of an agile culture change • Experience of transferring knowledge to permanent staff within a client organisation Cultural Fit Criteria • How will your organisation support educational attainment relevant to the contract, including training schemes that address skills gaps and result in recognised qualifications (e.g. , Apprenticeships, Industry placements, Traineeships) • Describe how you will encourage representation of under-represented groups in the workforce, e.g. Women, BAME, Disabled, LGBTQ+ and how will you monitor and measure this throughout the contract? Payment Approach Capped time and materials Assessment Method • Case study • Work history • Reference • Presentation Evaluation Weighting Technical competence 50% Cultural fit 15% Price 35% Questions from Suppliers 1. Can you give an idea of approximate number of resources required for the this work. Between 25 and 50 are expected for this requirement 2. Could you confirm whether Informatica DEI is a strategic technology for the programme or is it being gradually replaced by the ADF/Databricks combination? Will data engineers thus require more than reverse engineering existing Informatica pipelines to migrate them onto Databricks or will they be actively developing new pipelines in Informatica? Yes, Informatica DEI is a strategic technology used for the programme. The data engineers will actively be developing new pipelines in Informatica.We are not looking to reverse engineer any Informatica pipelines as part of this programme. 3. As part of the collaborative team how much management and leadership is required from the resources to deliver the new cloud data platform ?• There is no direction mention of involvement in data governance and management, data modelling, data lineage, data dictionaries, dashboards and scorecards, data mart/sets, cubes and summarisations – can you please clarify if these remain out of scope or are required ?• Beyond development – Is there any role in transition, support and maintaining the developed solutions and platforms?• Does the requirement around transferring knowledge to permanent staff include training them on technical topics? Leadership is generally provided by the senior capability leadsThis work and the roles are required to successfully deliver our demand.There may be some support required to operationalise some products and or services Training on technical topics could be part of the knowledge transfer 4. What are the expectations around the timelines for mobilising a team after a statement of work has been signed and exchanged?Is there a set of defined SQL/non-SQL databases that are used in the DfE ecosystem? We would work with to provide early indications of any new demand so that you can assess timescales for bringing resources in. No, there is not a set of defined databases