Disaster Recovery Implementation Partner

Description

Summary of work The Pensions Regulator (TPR) are looking to design and deliver a suitable Disaster Recovery (DR) solution; reorganize in Azure to achieve DR Implementation and Cloud Modernization. It will achieve a coherent unified DR solution across our two sides of the business that consist of Auto Enrolment and Levy, as well as increase efficiency and savings in Azure costs. TPR have identified areas where Specialist and Subject Matter Expert (SME) assistance or outcomes are required from a Professional Services (PS) supplier. The work is to provide assistance and outcomes by supplying the technical lead of Professional Services (PS), Specialists and Subject Matter Experts (SME) in Microsoft Azure resiliency, SQL, general Cloud and networking. The size of estate in terms of distribution of compute and storage scale is as below: LEVY All Resources and numbers Analysis Services: 1 Data factory (V2): 4 Disk: 536 Application gateway: 4 Storage account: 114 App Service: 6 App Service plan: 11 Availability set: 53 Service Bus Namespace: 2 API Connection: 16 Logic app: 11 Azure Data Box: 4 Azure Cosmos DB account: 1 ExpressRoute circuit: 5 SQL database: 18 Load balancer: 35 Azure Databricks Service: 1 Function App: 9 Virtual machine: 115 Machine - Azure Arc: 21 Key vault: 30 Logic App (Standard): 1 SQL virtual machine: 23 SQL Server - Azure Arc: 1 Azure Cache for Redis: 1 Recovery Services vault: 4 Route Server: 1 SQL server: 10 SQL elastic pool: 2 Dedicated SQL pool (formerly SQL DW): 1 Traffic Manager profile: 1 Virtual machine scale set: 4 Virtual network: 8 AE All Resources and numbers Load balancer: 2 Application gateway: 2 Availability set: 2 App Service plan: 9 Managed database: 32 Disk: 54 SQL database: 3 Data factory (V2): 6 Function App: 44 Key vault: 8 Logic app: 15 Storage account: 36 SQL managed instance: 4 SQL server: 3 Traffic Manager profile: 5 Recovery Services vault: 2 Virtual machine scale set: 1 Virtual machine: 18 SQL virtual machine: 2 App Service: 27 Working alongside and with the TPR Project Manager (PM) and core project team, the expectation is for the supplier to undertake the following activities: Stage #1 - High level design phase (familiarisation) anticipated during Q4 2023 • Health Check review of the TPR DR Strategy Pack and producing a RAG Report. • Technology workshop, review, and advisory for 'DR Strategy Implementation High Level Design (HLD)' and the new 'DR Standards for Solutions Architecture'. Stage #2 - Detailed design/Dev/Test phase anticipated during Q4 2023 • Provide a Proof of Concept (PoC)/Dev for moving Foundation/AE/Levy Prod across all 3 Availability Zones (AZs) in the UK South region. • Produce any Low-Level Designs(s) (LLD)/build documents required for moving Foundation/AE/Levy Prod to AZs in UK South and non-active DR in UK West. • Produce migration and rollback processes and repeatable templates outputs for moving Foundation/AE/Levy Prod to AZs in UK South. • Assistance to produce Acceptance into Service (AIS) testing documentation. • Assistance to produce identification of pilot and batches, and migration delivery planning. Stage #3 – Migration delivery phase anticipated during Q1 2024 • Assistance to deploy and test AZs pilot. • Assistance to produce the new DR Runbook for Modernised Cloud. • Flexible consulting support during each tranche of batches. • Over the shoulder guidance and screen share during each tranche of batches. • All Stages assistance with a light touch PMO/engagement manager for IT Delivery Partner PS resources. Where the supplied staff will work South East England Why the work is being done This project is to revise TPR DR strategy; ensures it reflects the current operating model and can be flexible when future changes are made. This new strategy will meet TPR’s corporate priorities, whilst ensuring value for money by achieving a coherent unified DR solution across AE and Levy; lead to increase efficiency and savings in Azure costs. The strategy is targeted for completion in March 2024 (or sooner) and from this, a roadmap will be produced, for both AE and Levy. The requirement for an IT Delivery Partner is needed currently to satisfy the project technical design and development activities window of FY 23/24 Q3, and the migration support window of FY 23/24 Q4. The work is being done to implement a DR Strategy, which stretches Single VM and Local redundancies Azure services across three Availability Zones within a region (UK-South) achieving Zonal redundancies, and the Recovery Services Vault footprint and Terraform scripts in a second region (UK-West), according to the customer High Level Design (HLD) Document. The business problem The DR Strategy is to solve the issue appeared due to move of TPR’s IT services, from an on-Premises Delivery to an Azure cloud-based solution was achieved without building and fully testing the requisite DR suite of services. As a result, TPR is unable to guarantee the Disaster Recovery components of Business Continuity commitments: Levy Tenancy – The infrastructure was setup in an active-active configuration with the same services setup in UK West as in the Primary UK South site. The services setup in UK West were not completed and not operational. But since some services are running in UK West, there is a sizeable cost being incurred for services that are not functional. In its current state, the Levy tenancy is not easy to setup in an Active – Active mode due to the complexity of the Extract, Transform, Load (ETL) functions and the intricacies of the data movement. AE Tenancy – The AE tenancy was initially setup and managed by Capita in an Active-Active configuration between UK South and UK West. They also carried out a failover test between the two regions. The tenancy since handed over to TPR, has been through a lot of changes to the UK South region that have not been reflected in the UK West region. It has made the ability to test the active-active configuration more complex. Due to the fact, there is an inter-dependency between the two Tenancies and testing or setting up their DR independently will not work. This situation has been compounded by the addition of PaaS and SaaS services, beside bringing in house of the AE service from Capita. Due to the changes in the TPR operating model, it is important to ensure its DR strategy meets its operational needs and is within its resiliency risk. There is a need to review that the strategic recovery objectives are realistic and achievable in the light of changes to the operating model. It is also important to understand the complexities of the operating model and ensure that they are also in line with the DR strategy objectives. The key deliverable for this project is to unify and enhance the DR strategy for Levy and AE and improve on the current 99.9% availability by moving to Azure Availability Zones with a 99.99% availability which will also support run cost reductions for the DR state. The people who will use the product or service User type: TPR (the whole organisation) Definition: • Solution needed to provide High Availability and High Resilience for the TPR infrastructure state as a part of the implementation of DR Strategy. • So that it ensured the mitigation for any potential outage. Which phase the project is in Alpha Existing team Core Project • Ian Reis (Project Manager) • Aderemi Adejumo (Solutions Architect) • Feroz Siddiqui (Lead Business & Technical Analyst) IT Operations • Gareth Sumpter (Head of IT Operations) • Daniel Hay (Infrastructure Manager) IT Applications • Simon Twigger (Head of IT Solutions) • Richard Difford-Smith (Data Solutions Manager) Address where the work will be done Telecom House, 125-135 Preston Road, Brighton, BN1 6AF Working arrangements TPR operates a hybrid working practice. There may be times where it is appropriate to work in the office but on the whole work is done from home. Technological supply can be arranged if need be. Security and vetting requirements Baseline Personnel Security Standard (BPSS) Latest start date 1 December 2023 Expected contract length Contract length: 0 years 8 months 0 days Optional extension: 0 years 4 months 0 days Special terms and conditions special term or condition: 1. Special Terms to address the Pensions Act 2004– Section 82 Restricted information. 1. In this clause ‘Restricted Information’ means ‘restricted information’ as defined section 82(4) of the Pensions Act 2004 (the 2004 Act). 2. To facilitate the discharge by the Supplier of its obligations under the Call-Off Contract, and in the exercise of its statutory functions as set out in the 2004 Act, the Buyer may disclose Restricted Information to the Supplier. 3. Where the Buyer discloses Restricted Information to the Supplier in accordance with this clause, then, in relation to that Restricted Information: a. for the purposes of section 82 (1) (b), of the 2004 Act, the Supplier will be a ‘person who receives information’ from the Regulator, whether directly or indirectly. b. the Supplier notes the provisions relating to the disclosure of Restricted Information contained in section 82 of the 2004 Act, in particular section 82(5) and (6); and c. the Supplier will be liable for any breach of section 82 of the 2004 Act in relation to that Restricted Information resulting from its acts or omissions or the acts or omissions of any Subcontractors. 4. A disclosure of Restricted Information by the Supplier in breach of section 82 of the 2004 Act will be a material breach of the Call-Off Contract and the Buyer may by notice immediately terminate this Call-Off Contract. Budget Indicative maximum: £135000 Indicative minimum: The contract value is not specified by the buyer Further information: Up to a maximum budget of £135,000 exclusive of VAT (This is inclusive of contingency). Contracted out service or supply of resource? Contracted out service: the off-payroll rules do not apply Terms and acronyms Term or acronym: TPR Definition: The Pensions Regulator - The Buyer Term or acronym: DR Definition: Disaster Recovery Term or acronym: AE Definition: Auto Enrolment Term or acronym: PS Definition: Professional Services Term or acronym: PM Definition: Project Manager Term or acronym: HLD Definition: High Level Design Term or acronym: LLD Definition: Low Level Design Term or acronym: PoC Definition: Proof of Concept Term or acronym: AZs Definition: Availability Zones Term or acronym: AIS Definition: Acceptance into Service Term or acronym: ETL Definition: Extract, Transform, Load Term or acronym: VM Definition: Virtual Machine