Procurement of Data Science services - Testing

Description

Summary of the work A requirement for a flexible team of multiskilled Quality Assurance Test analysts (QATs), Test Engineers (TEs), Senior QATs, Test Leads and Test Managers to work on Data Science projects within fast-paced Agile software development teams delivering Government Digital Services. Expected Contract Length 18 Months Latest start date Thursday 1 October 2020 Budget Range The budget could range up to a maximum of £4.8m consisting of multiple Statements of Work, but that scope could expand or contract. DfE does not commit to any minimum spend at this point. A pricing template will be given at the proposal stage to those suppliers we shortlist. Why the Work is Being Done ESFA Data Science Data Collections and downstream systems and services are used to collect, manage and report financial, participation, attainment and building information from Academy Trusts, Local Authorities, FE and Apprenticeship Providers and Diocese to inform its funding allocations, payments, services and market oversight remit. Data is collected through bulk upload, online forms, and document exchange platforms. Data Collections systems are the source and starting point of data flow into Data Science, via the IT products detailed in the Online Collections Service Product Register, essentially Achieve Forms, Financial Reporting products, .NET (Child Safe Guarding, Report Extremism in Education, National Non Domestic Rates), OPA (Related Party Transactions, College Financial Planning) . Functional and Non-Functional testing is required at the point of Online Collection services and Learner Records Service, and throughout the software and data delivery service processing and software mechanisms, including the Master Data Repository and Analysis and Modelling, Data Operations and Analysis and modelling data storage and data warehousing databases, including systems and software that handle data transformation & MI/BI reporting. Where testing is completed to meet Data Science quality assurance requirements and the quality assurance of product delivery demanded by GDS assessments. Problem to Be Solved Development of new and improved Data Science IT software systems, applications and services is a continuing expectation, from 2020 onwards. The ESFA will upgrade it’s software and consolidate platforms, to improve the quality and timeliness of its data, to reduce the burden on its data suppliers and customers. The ESFA is also consolidating its data storage and data management systems and automating data flows to reduce reliance on manual processes to transform, upload and manipulate data for its data scientists. We need to assure that data is processed correctly and data is reported correctly, poor output will impact users referenced above and risks financial and reputational damage of ESFA and DfE. Who Are the Users As a provider of Primary/Secondary/Tertiary education, I need to submit financial returns to the DfE so that our financial status can be established for approved funding and the DfE can provide accurate reports on our Funding Allocations/ Achievement Rates/ Risk Assessment / Qualitative Reports. As a DfE civil servant, I need to support critical services and support the development and improvement of essential Data Collections, Data Processing and Data Reporting systems, so that our users receive the highest quality ESFA IT systems and services, and our customer requirements are satisfied. Early Market Engagement N/A Work Already Done Functional and non-functional testing, utilising MTS testing staff, has previously been done for many Data Science programmes of software development and online collection services. Including the Better Financial Reporting Programme, with the Accounts Return 4 (2019), AR5 (2020) and Better Future Reporting Programme (19/20); the Land and Building Collection tool (2019); many Achieve forms projects; Qualification Achievement Rates (18/19/20); various View Your Education Data projects and many Data Operations team projects, covering data warehousing and reporting. Existing Team The supplier will be working with civil servants and multiple suppliers across various strands of work for delivery of the service. Typically, within Agile Scrum and Kanban delivery teams, working with Product Owners, Delivery Managers, Business Analysts, Developers, other Test Engineers and Test Analysts. Current Phase Not applicable Skills & Experience • Demonstrable experience in applying Agile testing (preferably Scrum and Kanban) methodology, skills, and techniques to projects. To confidently discuss Agile testing processes from an overall software development perspective. • Demonstrable experience of Test-Driven Development (TDD), Behaviour Driven Development (BDD), Continuous Integration (CI) and pair-testing activities. Experience of component integration testing (CIT) and system integration testing (SIT). • Demonstrable experience of analysing and executing Risk-based and Context-driven approaches to testing, together with demonstrable experience of using Exploratory testing techniques. • Demonstrable experience of automation testing using Agile with architecture, design and configuration of testing frameworks. Including Selenium with IDEs for C# and Java development. Also, SpecFlow; Cucumber; and other tooling. • Demonstrable experience of MS SQL database testing skills, including validating stored procedures, jobs, and triggers. • Demonstrable experience of data warehousing, ETL, data migration processing and data quality assurance management, testing. • Demonstrable experience of Application Program Interface (API) testing within Agile sprints, using tools such as Postman and/or other industry recognised API testing tools. • Demonstrable experience of accessibility testing in compliance with UK Government accessibility regulations, using automated accessibility testing tools and assistive technologies. Including SortSite and accessibility assistive technologies, e.g. JAWS, Fusion, Dragon. • Demonstrable experience of performance, scalability, and security testing of a variety of digital services, especially with Apache JMeter and OWASP top-ten security testing. • Demonstrable experience of presenting test outcomes to technical and non-technical teams. Presenting all types of findings to a range of audiences, including Demos, Show and Tells, Retrospectives and walkthroughs. • Demonstrable experience of producing Test Approaches and Sprint Test Plans collaboratively with the team and suppliers, in accordance with the Department's overall approach and test strategy. • Testing certification: ISTQB Certified Tester Foundation Level (100% of MTS provided testers certified passed) ISTQB Certified Tester Foundation Level – Agile tester extension (75% of MTS provided testers certified passed)” Nice to Haves • A good understanding of the Government Digital Service Standard and Service Manual. • Experience and knowledge of cloud infrastructure and service technologies, such as Azure PaaS, IaaS, their sourcing and other IT service model attributes that may have impact for testing. • In-depth experience of using Microsoft Azure DevOps as a tool for test planning, execution and reporting. • Demonstrable experience of defining or breaking down user stories and acceptance criteria using test design techniques, and contributing to the "Definition of Done", from an overall software quality perspective.” • Experience of Operational readiness testing including, Helpdesk and Support readiness testing, backup/restore and Disaster Recovery testing. Dependent on the IT infrastructure service models. Work Location Primary location will be Education and Skills Funding Agency, Cheylesmore House, 5 Quinton Road, Coventry CV1 2WT, with occasional travel to other locations nationwide as and when required. Working Arrangments All Managed Testing Service testing staff would be required to be located full-time (5 days a week) at the ESFA offices in Coventry in co-located Agile delivery teams. If in exceptional circumstances the office is closed staff would be asked to work from home until the office re-opened. Occasional travel to other ESFA (or partner organisation) offices may be required, subject to DfE / ESFA travel and subsistence policy. A copy of the T&S policy can be provided on request. Security Clearance Personnel from the provider need to have a BPSS check. The supplier must be able to provide verification to ESFA that all proposed team members have been subject to Baseline Personnel Security Standards checks within the last twelve months. Higher levels of Security Clearance may be required. Additional T&Cs Expenses must be pre-agreed and comply with DfE Travel and Subsistence Policy. Any expenses shall be submitted in line with DfE standard T&S policy. Primary work location stated in Statement of Work will not attract expenses. No parking is available on site. No. of Suppliers to Evaluate 5 Proposal Criteria • Demonstrate/provide evidence of experience and knowledge of Agile testing (mainly SCRUM and Kanban methodologies), have experience of providing comprehensive resourcing of blended skilled Quality Assurance testing teams and individual testers • Demonstrate/provide evidence of experience and knowledge of testing automation within CI pipelines and as part of a DevOps team. • Demonstrate/provide evidence of experience and knowledge of exploratory testing. • Demonstrate/provide examples of how the supplier has worked with multi-disciplinary delivery teams in a QA role • Demonstrate/provide evidence of how the supplier contributes to building an effective multi-disciplinary team • Demonstrate/provide evidence of the supplier’s approach to incorporating user needs and government needs into the product backlog and prioritises work effectively and efficiently • Demonstrate/provide evidence of how the supplier actively promotes understanding and acceptance of Agile, user-centred delivery • Demonstrate/provide evidence of how the supplier has worked in a multi-team environment with multiple product owners; collaborating, delivering broader service goals and achievements, in a successful, quality IT delivery programme. Cultural Fit Criteria • Be able to operate as a definitive voice of testing and quality assurance within any software development and testing team. • Ensure the Agile Manifesto key principles are followed, wherever possible. • Individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; responding to change over following a plan. • Establish excellent working relationships and generate good team spirit • Make regular contribution to the DfE / ESFA test engineering professional community(s ) • Show a commitment to ensuring the wider team understands the user needs, over simply delivering work. • Be innovative and act on initiative; drive improvements; balance risks with gains, and, costs and benefit. • Look to build capability and encourage learning, knowledge sharing; address capability gaps and act on feedback in a timely manner • Demonstrate responsibility (e.g. be a leader, serve the team, attempt to provide solutions to problems, engage in external/internal environments, seek the best for the organisation, beyond your team) • Focus on delivery goals; set stretching targets; take responsibility for delivery • Be a part of the "One Team" approach Payment Approach Capped time and materials Assessment Method Presentation Evaluation Weighting Technical competence 30% Cultural fit 10% Price 60% Questions from Suppliers 1. Is there an incumbent software testing provider, and if so, who is it? The incumbent supplier for work within scope for this opportunity is Flatworld Works. 2. Is this a new service or is there an incumbent supplier – If so will they be applying ? The incumbent supplier for work within scope for this opportunity is Flatworld Works. This opportunity is open to any supplier on the DOS4 Framework to submit should they choose. 3. How many people are you looking for? Not clear what this question is referring to. This will be an outcomes based requirement where the successful supplier provides a solution and dictates the make up of teams to deliver services 4. Can we ask why this opportunity, to supply Testing Services, has not gone out via the "Quality Assurance and Testing for IT Systems 2" Framework?Apart from Government Digital Services, the opportunity only refers to Digital once and even then as an overarching service. Thanks The following link outlines the coverage of testing through the DOS4 Framework https://www.gov.uk/guidance/digital-outcomes-and-specialists-digital-outcomes-capabilities#testing-and-auditing . The "Quality Assurance and Testing for IT Systems 2" is complimentarty to the DOS4 Framework, not a replacement. On assessing options DfE chose the DOS4 frameowrk as the most suitable route. 5. Is there an incumbent in place and are we permitted to know who ? Is this the end of their existing contract ? The incumbent supplier for work within scope for this opportunity is Flatworld Works. This current contract for these expertise is due to expire, DfE have a continued need for delivery of outcomes in scope of this requirement. 6. Would you be able to confirm the size of the initial requirement in terms of total number of resources and blend of expertise/experience? Is it expected that the team size will remain relatively constant throughout the proposed 18 month engagement? This will be an outcomes based requirement where the successful supplier provides a solution and dictates the make up/size of teams to deliver outcomes. The size of teams required will flex over the 18month period. 7. Could you please outline the current teams structure that this bid relates to. Thanks The supplier will be working with civil servants and multiple suppliers across various strands of work for delivery of the service. Typically, within Agile Scrum and Kanban delivery teams, working with Product Owners, Delivery Managers, Business Analysts, Developers, other Test Engineers and Test Analysts. 8. In the next stage of this procurement, suppliers will be required to propose a team make-up, approach and associated costs to successfully deliver the specified outcomes. This provides the incumbent with a clear advantage in this competition, as they already have detailed knowledge of the environment, culture, work and challenges involved, and hence the team size and approaches that the buyer will perceive to work best. How do you propose to level the playing field to give all of the suppliers a fair chance with their proposal? The second stage of this opportunity will assess suppliers against the proposal criteria and cultural fit criteria outlined in the publication on the digital marketplace, along with an assessment of price. Suppliers who are successful in reaching stage 2 will be able to ask any clarification questions to support their proposals. Suppliers will be scored against how well they are able to evidence they meet the criteria. 9. Please clarify on the possibility of the remote work using video conferencing tools. Thank you. Due to current working restrictions in relation to Covid-19 it is likely that there will be remote working initially. Once these restrictions are lifted delivery of service will continue to be from the primary location unless otherwise agreed with statements of work 10. Is mobile testing included in the scope of engagement Future software and digital services development will most likely require cross-skilled testing service resources, who are capable to utilise and learn mobile apps testing skills and toolsets. Certain webforms and services will be begin to be developed and designed to require mobile device requirements testing, certainly covering functional and non-functional testing, especially where this would also concern the testing of accessibility requirements for meeting the Government accessibility regulations. 11. To what extent is testing automated for the in-scope applications – we realise this would be an estimate Currently around 20% functional automated regression testing, but also testers experienced in working with and helping in the development of CI automation regression test suites, would be most advantageous. 12. Is non-functional testing restricted to performance and security testing only? Not wholly restricted. Usability would be incorporated into all functional testing. Accessibility is defined as Item 8 within these MTS requirements. Also, Reliability; Maintainability; Compatibility; Interoperability; Ready for Operations / service readiness. 13. Please list tools being used foro Test and Defect Managemento Test Automationo Performance Testingo Security Testingo ETL and Database Testingo Mobile Testing Test and Defect ManagementAnswer: Azure DevOpsTest AutomationPerformance Testing JMeterPerformance TestingJMeterSecurity TestingZap Zed Attack Proxy and potentially other opensource toolsETL and Database TestingMainly MS SQL Database tools, SSMS etc. for test data analysis, data management and test execution.Mobile TestingCurrently ESFA Data Science are exploring BrowserStack and SauceLabs suitability for testing our digital services with mobile device simulators and emulators. 14. Are suppliers free to propose tool recommendations and/or include tool licensing in the service? We would be happy for the MTS supplier to offer any recommendations and advice on potential testing tools and other supplementary services. Though any decision and authorisation for such proposals would be ESFA’s alone. 15. Can you please clarify what this question is asking? “Testing certification: ISTQB Certified Tester Foundation Level (100% of MTS provided testers certified passed) ISTQB Certified Tester Foundation Level – Agile tester extension (75% of MTS provided testers certified passed)” All proposed MTS testing service reps would be required to meet the 100% minimum ISTQB certified tester foundation level standard. Then 75% of testing service reps supplied, would be required to be ISTQB Agile tester extension certified passed. 16. Can you clarify what you mean by "To confidently discuss Agile testing processes from an overall software development perspective." in the following question. Are you looking for examples where our team has suggested how we should implement Agile ways of working?"Demonstrable experience in applying Agile testing (preferably Scrum and Kanban) methodology, skills, and techniques to projects. To confidently discuss Agile testing processes from an overall software development perspective." It would be examples of where the supplier MTS representatives’, in their usual working practice, could provide detail and example of how they would work to provide suitable Agile software testing practices, as integral part of an Agile software development team.