DDaT22027 - Janus Analytics Data Platform Design, Build & Deployment Project

Description

Summary of the work We have recognised the need to improve our Data Analytics Platform to enable us to progress our strategic objectives. We wish to design and deploy a consolidated data analytics platform which will hold both internal and external data and enable us to leverage both in our analytics and reporting outputs. Expected Contract Length 3-4 Months Latest start date Friday 20 May 2022 Budget Range Total costs should not exceed £200,000 exclusive of VAT. Why the Work is Being Done Innovate UK is progressing on its journey to becoming a data-driven organisation. We have recognised the need to improve our Data Analytics Platform to enable us to progress our strategic objectives. Specifically we wish to design and deploy a consolidated data analytics platform which will hold both internal and external data and enable us to leverage both in our analytics and reporting outputs. Problem to Be Solved This is a new platform intended to enable interactive data mining and to enable AI and ML data functions to be applied against it in real time. To achive this we need the platform constructed and deployed. This will require a number of current data sources and integration tools to be interfaced with this new platform to ensure it retains current and accurate data from the internal systems sources. Also external data feeds from a variety of providers will be integrated using our existing SIL (message bus) capabilities. Who Are the Users This is a technical delivery platform with a wide array of users and requirements. These are stated in a separate presentation. The analytics platform will be hosted in AWS in line with our strategy and other InnovateUK internal systems and is likely to be based on RDS PostgreSQL. The platform is intended to be flexibly scalable, resilient and highly performant as it will serve interactive Power-BI reporting systems across the organisation. Additional capabilities to utilise other analytics tools and systems in future is in scope of this project as well as integration of current deployed BI reports currently hosted on other and legacy platforms. In addition we require an AWS data lake to be provisioned with supporting EC2 nodes to run python ETL processes against that datalake. Further expansion to include apache spark and apache kafka are considered likely in subsequent phases (not in scope for initial platform delivery) however note should be made of this and provision in design made accordingly. Early Market Engagement N/A Work Already Done Discovery has already been done – this is an alpha phase project. A full presentation on the requirement, strategy, architecture and desired outcome is available separately. Existing Team This project will require close working with the Innovate UK data team, helping to develop capability within the Innovate UK team, as well as delivering the core deliverables of the tender. The main objective of this project is to design, build and deploy a fully operational data platform suitable for intensive query use from a large number of Power-BI dashboard users across the organisation. Additional data loads and automated analytics processes are required to interact with this data based on Python, and a variety of other AWS analytics tools including but not limited to SageMaker. Current Phase Alpha Skills & Experience • Demonstrate your teams experience and understanding of up to date industry developments regarding data management, use, and analysis. • Detail with an example where you have worked with AWS Aurora Postgres. • Detail using an example your experience of RDS. • Detail using an example your experience of MySQL • Detail using an example your experience of using ETL Tools. • Detail using an example your experience in Talend. • Detail using an example your experience of WS02 • Detail using an example your experience of Kafka, Spark, Python and Java. • Detail, using an example your application experience Nice to Haves • Detail your experience of current data manipulation development knowledge • Detail your experience of AWS database portfolio more broadly, AWS analytics and data lake services, ETL tools including some of Knime, • Detail your experience of Talend, Alteryx, WS02 others. You will also have extensive experience of use of BI tools including Power-BI, Tableau, and other tools. Python, Java Work Location Hybrid working will be used. The activity is expected to be performed using hybrid working – mostly remote with some in-office time at our Swindon offices for meetings and design discussion workshops for example. Working Arrangments This project will require close working with the Innovate UK data team, helping to develop capability within the Innovate UK team, as well as delivering the core deliverables of the tender. Security Clearance BPSS required Additional T&Cs N/A No. of Suppliers to Evaluate 5 Proposal Criteria • Describe the approach and methodology you will apply to Alpha, including how priorities will be determined and emerging findings presented • Describe how the approach will achieve the objectives and intended outcomes • How the structure of the team and how their technical-skills and experience will be used to deliver-the-need, including estimated-resource dedicated to the project linked to the overall cost • A project plan with proposed timelines for mobilisation and delivery • Identify the risks and dependencies you associate with the project and provide mitigation actions to manage them • Value for money, including how you will take previous learning to deliver better value for this project Cultural Fit Criteria • Demonstrate how your proposed team will work successfully with UKRI as part of a remotely distributed multi-disciplinary team. • Demonstrate how your proposed team will deliver in an open, collaborative, agile way. Detail your plan for sharing knowledge and experience • Demonstrate how you will work with stakeholders at all levels, within the programme and across the wider UKRI landscape. • Detail how you will work within given constraints (including but not limited to technology, policy, regulatory, financial, legal, ethical, social, user constraints). Payment Approach Capped time and materials Assessment Method Presentation Evaluation Weighting Technical competence 65% Cultural fit 10% Price 25% Questions from Suppliers 1. Was the Discovery exercise conducted by an incumbent supplier or in-house? Discovery exercise was conducted in-house and led by Innovate' s Head of Data Management, MI & Analytics. 2. Who is the incumbent supplier who has conducted Discovery phase? There is no incumbent supplier, internal team and lead conducted discovery. 3. Would it be possible to outline how you are using WS02 Products/Services? Is this for API management, Identity and Access Management, Digital Asset Governance and/or using Ballerina? API and ESP services - intending to replace it - need to examine it closely and then replace. 4. Can small business bid for this project? Yes, small businesses can bid for this requirement 5. Please can we enquire about who completed the Discovery phase. Was this Innovate UK or a third party? As per question 2 answer, discovery was completed in house by Innovate 6. Was Discovery conducted by the internal team? If not, who was the supplier involved? As per question 2 answer, discovery was completed in house by Innovate 7. Who is currently delivering the Discovery for this? As per question 2 answer, discovery was completed in house by Innovate 8. What was your reasoning behind choosing technologies such as AWS Aurora Postgress, WS02, Talend, Spark? Some are legacy we are migrating from, aws aurora postgres, spark, kafka, we are moving to - latter is strategic - others needed to understand to help with migration (talend, ws02 legacy for example) 9. We don’t specifically have WSO2 experience, however, we leverage an associate network who have a wide range of different technology experience so we can bring in that skillset for this project. Would this be considered as meeting the essential skills for this submission? Yes if you can do it within budget and maintain quality and time to deliver 10. The requirement mentions that a Discovery has already been done, and a full presentation on the requirement, strategy, architecture and desired outcome is available separately. How do we obtain a copy of this presentation? Outcomes from previous stages will be shared with shortlisted suppliers 11. Why are all three of postgres, MySQL, and RDS being used? Are there multiple legacy systems and data that will be accessed? MySQL is legacy, rds postgres is target future platform - postgres flavour of rds will be used - both mentioned for completeness. 12. Is there an incumbent and are they eligible to bid? There is no incumbent supplier. 13. Which supplier delivered the Discovery phase of the project? As per question 2 answer, discovery was completed in house by Innovate 14. Can you clarify/correct question 9 Give an example project where you have used some or all of the relevant skills and experience to deliver a data infrastructure solution 15. Is Knime experience a ‘must’ requirement for your new Analytics Platform? It is part of our current platform but not a must have - desirable to understand as many tools involved as possible (talend, ws02, alteryx, knime, spark, kafka, plus any others in that space) 16. Are you considering other ETL technologies to replace Knime? Yes 17. Is the Knime integration productionalised? Yes - but has limited deployed scale in our use case 18. Are you using any streaming technologies at a production level?If yes, what is the high-level use case for streaming data? And are you currently using live dashboards? Not yet - data doesn’t really lend itself to streamed processing. yes we are using live dashboards based on powerbi and some tableau 19. Are you using licenced versions of Talend Cloud in your solution? To be confirmed during project - we are licensed and have production services and adminstrator workspaces live with talend 20. Do you have SMEs within your business already utlising Apache Spark? Yes - limited use at present - expecting to increase this 21. Does your team have experience with Databricks? Not known 22. To what extent do you implement DevOPs code management procedures? Handled by another team - we have a devops service delivery team that handles this - we anticipate some change to that to increase our hands-on devops capability in this project