Data Engineering Project

Description

Problem statement Ofqual need to improve their maturity in the use of Azure data management and storage products. Additionally, Ofqual require assistance in developing a front-end interface for management of the Portal tagging dataset, primarily for the use of the Risk Team to bulk manage the SVR reporting. The requirement To deliver the below stated requirements by the end of June 2021. To define the development lifecycle process for Azure Data Factory development with a focus on integration with Azure DevOps using Git repositories and build and release pipelines. Support the Ofqual Data Engineering teams' efforts to migrate the existing SQL Server Integration Services (SSIS) based data importer tool into Azure Data Factory. Work with Ofqual to develop a small front-end interface for the Portal's tagging data set. The work package requires external technical expertise to work alongside Ofqual's in-house data services team to develop the outcomes described in this work package. Outcome Required A clear set of practices for working with Azure Data Factory and Data Lake. An assessment of the options for a small front-end interface for bulk record management of the Portal's tagging attributes (assumed to be based on a development-lite technology such as PowerApps, Excel or Master Data Services). Provide expertise to the Data Services team whilst they migrate from SSIS. Provide an approach to structuring and securing an implementation of Azure Data Lake.