Enterprise Data Platform

Tender description

Work done so far Work has already been carried to understand the data maturity of TfW. The government framework was used and TfW was classed as within the 2-3 range under the framework score matrix. This work also identified a number of business requirements from across the whole of TfW . A data strategy refresh was also carried out to identify the capabilities that need to be developed in order for TfW to move its data maturity to a level where it can deliver on its vision. Which phase the project is in Discovery Existing team Team consists of 4 data analysts, recruitment underway for a business analyst, PM, data engineer and data architect. Capability within the wider team consists of Devops in-house engineering teams for infrastructure deployment, solution architects, software engineers, cyber security engineers. However, there is not committed time from these resources. Address where the work will be done Remotely and in the TfW head office: 3 Llys Cadwyn, Pontypridd, Rhondda Cynon Taf, CF37 4TH Working arrangements The supplier will be required to work in the Pontypridd office 2 days week for F2F meetings and collaboration. The remaining 3 days can be remote within the UK. Travel and subsistence expenses will require prior approval. Ideally the supplier should have offices in South West/South Wales to limit such expenses." Provide more information about your security requirements: Baseline Personnel Security Standard (BPSS) Latest start date 2025-10-01 Enter the expected contract length: 1 year, 6 months Extension period: 4 months and 15 days Special terms and conditions All expenses must be pre-agreed and in line with TfW expenses policy. Are you prepared to show your budget details?: Yes Indicative maximum: 462000 Confirm if you require a contracted out service or supply of resource Contracted out service: the off-payroll rules do not apply Summary of work TfW require a team with experience in designing and delivering an Enterprise data platform within a Microsoft first organisation. Build will include an agreed number of business use cases that need to be delivered end to end including quick wins to demonstrate business value. Key outcomes will be automating the ingestion, integration and transformation of TfW data from a number of technologies, systems/sources(internal & external), that are structured and unstructured. Providing TfW with access to trusted, controlled, compliant, secure and consumable data products, augmented with AI/ML for analytics and reporting. The supplier must ensure the entire solution is documented and also time in their project plan to work along side TfW colleagues to upskill their capabilities to ensure they are confident in maintaining and future developing the delivered solution. Where the supplied staff will work South West England Where the supplied staff will work Wales Why the work is being done Transport for Wales (TfW) is embarking on a transformational journey to achieve its strategic ambitions, as outlined in the T network (https://tfw.wales/about-us/t-network#:~:text=That%E2%80%99s%20why%20we%E2%80%99re%20building%20an%20integrated%20multimodal%20transport,ticket%20to%20get%20you%20from%20A%20to%20B). Data is a crucial enabler in this journey, and its value must be fully realized. However, the current environment, shaped by the merger of multiple organizations, has resulted in a fragmented array of systems, data sources (both structured and unstructured), dashboards, and reports. This fragmentation poses significant challenges in consolidating data to produce a single, accurate version of the truth in a secure, compliant, efficient, and effective manner. Additionally, it hampers the ability to advance beyond diagnostic and descriptive analytics to predictive and prescriptive modelling, as well as leveraging a broader range of machine learning (ML) and artificial intelligence (AI) capabilities. A recent data maturity assessment highlighted several challenges and provided recommendations. One key recommendation was the need for an enterprise data platform, recognizing that the current environment cannot deliver the required pace, security, and contextual understanding of data. The existing approach is no longer scalable, especially with the ambition to integrate AI and advanced analytics. The current setup lacks a unified data model, standards, and controls, leading to inefficiencies and inconsistencies. The business problem you need to solve Complex, quality relevant data needs to be brought together, presented in the format required for different stakeholders, so they can engage, plan, analyse, deliver and measure how successful TfW is. The following needs to be considered and addressed. 1.Data Collection: Data comes from internal and external sources, SaaS apps, APIs, SQL dbs, or any type of data format. Unstructured , semi – structured and structured data. Batch, real time processing , change data capture Scalable for new data sources variety, velocity and volume Automated DM controls 2. Data storage/management Stored in multiple locations across the business, leading to data silos No central repository cataloguing data, its type, use, lifecycle, quality, provenance, lineage, business definitions, security and compliance requirements. Limited automated controls that govern the data Scalable storage 3. Analysing and Processing: Prolonged time to insight due to manual data acquisition, transformation, integration, and analysis. Duplication of data and reporting due to lack of awareness of existing data. Manual processes lead to high resource time and low feasibility of real-time data and decision-making. No documented single version of the truth or single views of core business areas Limited scalability , existing systems may struggle to handle increasing data volumes, changes and complexity Reporting and Dashboards Inconsistent data quality and duplication lead to unreliable reports and dashboards. Lack of a centralized data repository results in multiple versions of the truth and conflicting reports. Ensuring data security and compliance with regulations can be challenging with disparate systems. Speed to respond to requests and changing regulatory and customer requests. Resourcing and capability The capability to solve these business problems is not currently available in-house. Outside resources are required to provide solutions and support the creation of appropriate internal capabilities. First user type: Devops Engineers Questions and Clarifications 1. - What specific compliance frameworks or security certifications must the solution adhere to? - Technology Constraints: Are there any preferred or mandated tools, technologies, or architectural principles (e.g., use of specific Azure components, data lake/lakehouse preferences)? - You state that: "Work has already been carried to understand the data maturity of TfW. The government framework was used and TfW was classed as within the 2-3 range under the framework score matrix. This work also identified a number of business requirements from across the whole of TfW . A data strategy refresh was also carried out to identify the capabilities that need to be developed in order for TfW to move its data maturity to a level where it can deliver on its vision." Can you: - Share copies of the outputs from these activities - to help us better understand your state, requirements and ensure we're best placed to support. - Confirm how these were delivered - entirely in house or were any third parties engaged to support their delivery? - Have you worked with any third parties in developing the scope of this project? If so who? - Is there any flexibility on the number of days per week that the suppliers team will need to be in the Pontypridd office? - On Nice-to-Have Q2, should this read: "Experience working within organisations with organic and inorganic growth through acquisition and have legacy systems and strategic technology." - Cyber Essentials, or ISO 27001 .A SOC2 report if available. - Decision to be made as part of the project. - Initial documentation has been shared to provide a high level view of the TfW data strategy and the data maturity. Any furher documentation will be shared for those taken to the second stage. - No, the scope of this project was defined by TfW. - This can be discussed, but at least 2 days per week. - TfW as an organisation has grown ,in a large part, as a result of organisations coming together. With that comes the systems and technology from these organisations. Therefore, suppliers having experience of working with organisations that have a complex technology landscape across a mixture and age of technologies is seen as a 'nice to have'. Last Updated: 30 June 2025, 16:05 2. - Can TfW provide a high-level list of core systems or data sources (internal and external) expected to be integrated in the initial phase of the EDP? - Can you confirm how many initial business use cases are expected as part of the “quick wins” referenced in the Statement of Requirements? Are there any priority domains (e.g. operations, customer, commercial)? - Are suppliers permitted to propose third-party technologies or platforms as part of the solution, provided they align with a Microsoft-first approach? - Can you confirm when the detailed Stage 2 technical questions will be released to shortlisted suppliers, and whether a template or character limit will be applied? - Will there be an opportunity to submit CQs following the release of the detailed stage 2 questions. - There will be data sources from systems typical to an Enterprise. For instances, Finance, Customer, Health & Safety, Assets, Engineering etc .There will also be sources from suppliers and partners. However, the prioritisation will be based on agreed use cases within the project. - This will need to be discussed as part of the project. - This will need to be discussed as part of the project. - Yes, further details will be provided to the shortlisted suppliers as part of Stage 2. - Yes, shortlisted suppliers will have the opportunity to ask clarification questions specific to stage 2 of the further competition. Last Updated: 30 June 2025, 15:58 3. - How many different transport operators' systems need integration? - Could you provide an inventory of the mains systems requiring integration? Please include details of any legacy systems with limited connectivity options, SQL Databases, SaaS Applications and APIs etc. - Is this a greenfield implementation or is there any existing work completed in bringing data assets together that need to be migrated/adopted to the new platform/solution? - Have any external data sources have been identified for integration, and if so what are they? - Are there any specific requirements for data residency and/or sovereinty? - Please provide indicative data volumes, including total data estate, daily ingestion volumes, number of concurrent users expected, peak transaction/event volumes - What are the known data quality issues that need attention? - Can you confirm if licencing costs and Cloud consumption costs are budgeted separately from the prescribed £462,000 transformation budget? - Do you have an existing MS Enterprise Agreement? If so, What is your current Azure footprint? (eg Existing subscriptions, approx. monthly spend, current use of Data Services, Power BI implementations etc) - What architectural constraints or standards should we consider in formulating a solution? - For real-time data processing, are there any minimum latency requirements? - Would it be acceptable to build a hybrid team to deliver this project i.e. mix of on-shore and remote off-shore delivery capability? - The requirement for 2 days per week in Pontypridd, is this for the entire team or onshore/project lead personnel only? - What disaster recovery and business continuity requirements are there? - Is there a preferred DevOps toolchain or deployment model for managing infrastructure and data pipelines? (e.g., Azure DevOps or GitHub Actions for CI/CD, Terraform or Bicep for Infrastructure as Code) - We are not integrating other transport operators' systems. However, we have to integrate systems from our suppliers, partners and company group. - Systems span the operation of the business and will include legacy systems. Details will be discussed within the project. - Yes, it is to be treated as Greenfield implementation. - External data sources will be included and will come from open data sources and partners. - Only UK - To be discussed within the project. - Data quality issues fall under all of the dimensions and so a framework will need to be implemented and designed into this solution. - TfW maintains its licenses . Discussion will need to take place with TfW around licenses. - TfW maintains its licenses . Discussion will need to take place with TfW around licenses. - We are looking for a cloud native, Microsoft first solution. Detailed principles for TfW's Azure Cloud will be shared as part of the project. - Not all data will be required near real time and so will vary depending on use case. - Hybrid would be considered however off-shore access could be restricted based on security protocols. - The purpose of this is to talk through the business requirements and development as well as for project governance . It is for the wider team not just PM. - Decision to be made as part of the project . - Infrastructure, including its deployment will be managed by TfW. Data pipelines (for the application tier) must utilise TfW's Azure DevOps solution and the vendor is expected to work with TFW engineers to agree appropriate data pipelines. Last Updated: 30 June 2025, 15:43 4. Which Azure components are currently being/proposed (by your solution architects) to be used? The approach is up for discussion as it has not been determined. Last Updated: 27 June 2025, 14:52 5. - Could you supply an inventory of on-prem, SaaS and cross-cloud sources (system name, technology, location, daily volume, update frequency, criticality)? - Performance sizing - please supply estimates for users, concurrency, comments about any major performance bottlenecks in current systems that the new system will need to address. - For a cloud-based deployment, can the supplier assume new Azure subscriptions can be provisioned, and if so can these be done using a new CSP? - Does TfW have Microsoft 365 licences across the organisation? If so what SKU or type are they (e.g. E3, E5 etc)? - Whilst a Microsoft-first strategy is noted, is TfW open to considering non-native solutions that intergration with Microsoft services? - Is TfW seeking a solution that can be deployed and managed using DevOps technologies (e.g. ARM/Bicep templates, Terraform)? - Is there a requirement or preference for the solution to be a traditional "data replication" (copy and paste EDW), or instead a "data virtualisation" (proxy-based access) type solution? - Are there any requirements at this stage relating to backup, disaster recovery and resilience such as RPO and RTO? - The details will be discussed within the project . On-prem, cloud and external sources will be included. - At present there are a number of powerBI users within TfW . - TfW maintains its licenses and has a strict policy on the provision of Azure subscriptions. the solution will be required to adhere to these. We are not expecting the supplier to provision any agreement CSP or otherwise as part of this tender. -TfW does have Microsoft 365 licenses. -This will need to be discussed as part of the project. - Yes, noting the requirement to collaboratively work with TFW engineering teams. All Azure infrastructure will be deployed by TfW teams. Application code is also required to be deployed via automated methods. - Decision to be made as part of the project. - Decision to be made as part of the project . Last Updated: 27 June 2025, 14:49 6. - Is it a strict requirement for the platform to be built on Azure, or are you open to considering other cloud providers? - How is C# currently being used in your existing tech stack and what role do you expect it to play going forward? - Is it required for all new development, or only in specific areas? - TfW operates a single cloud set-up with Azure as it's cloud provider. No other cloud providers are permitted. - TfW has selected C# as it's primary software development language. This language will be used for all future software development, with existing non C# applications migrated over time. - It is used within software development. Last Updated: 27 June 2025, 14:42 7. Are there any upcoming initiatives during the delivery period of the project, that this new platform should support and integrate with? (eg existing/new source system/platform upgrades) Dependencies to be discussed as part of the project. Last Updated: 26 June 2025, 10:36 8. What metrics do you want to be able to track for passenger journeys? (eg end-to-end multi-modal journeys, real-time passenger flow, revenue across operators etc) Metrics will be decided and prioritised with each business area based on the use cases. Last Updated: 26 June 2025, 09:59 9. Could you elaborate on the "quick wins" and specific use cases and their prioritisation that you have identified in your discovery phase that would demonstrate immediate business value? Decision to be made as part of the project. Last Updated: 26 June 2025, 09:55 10. Have you made a decision if you want to migrate data to new consolidate platform(s) and retire legacy platform(s), or are you in search for an integration approach to access disparate systems in their current form? Or are you open to suggestions to approach as this has not been discussed? We are open to suggestions. Last Updated: 26 June 2025, 09:29 11. Evaluation Clarity: Can you confirm whether a more detailed breakdown of Stage 2 evaluation criteria and presentation expectations will be shared post shortlisting? Yes, more detailed instructions on the stage two criteria will be shared with the shortlisted suppliers. Last Updated: 26 June 2025, 09:26 12. AI/ML Scope: Do you have any specific use cases or priority areas identified for AI and machine learning deployment at this stage? This is being developed and decisions to be made as part of the project. Last Updated: 26 June 2025, 09:25 13. Existing Data Strategy: Can the refreshed data strategy and/or maturity assessment reports be shared during the bid or Discovery phase? A copy of the current data strategy was issued as part of the tender. We may refresh this during the discovery phase. Last Updated: 26 June 2025, 09:23 14. Access to Internal Teams: Will the supplier have dedicated access to internal technical staff (e.g., DevOps, Security, Business SMEs), or is access expected to be ad hoc? This will be based on need. The supplier is required to scope out what resource may be needed from our in-house team and we will look to augment the team's capability subject to internal capacity. Last Updated: 26 June 2025, 09:21 15. Engagement Expectations: Can you clarify the expected duration and scope of the initial phase (e.g., is this limited to Discovery or intended to include Alpha/Beta delivery)? The expected duration of this engagement will cover until go live and it will include delivery of use cases. Last Updated: 26 June 2025, 09:18 16. Data Governance Tools: Is there a current data governance or metadata management tool in place (e.g., Microsoft Purview), or should the solution include selection and implementation of one? Microsoft Purview is in place . Tools to be discussed as part of the project. Last Updated: 26 June 2025, 09:17 17. Data Sources & Integration: Can you confirm whether a full inventory or data catalogue of existing internal and external data sources is available, or will discovery and mapping form part of the initial engagement? An initial inventory was undertaken. However, this will need to be verified and validated against agreed use cases. Last Updated: 26 June 2025, 09:15 18. Will post-implementation support be performed between TfW and the product vendor (e.g. Microsoft), or is ongoing support from the supplier required? Ongoing support will be discussed within the project. This was covered within the ITT. The supplier must ensure the entire solution is documented and also have time in their project plan to work along side TfW colleagues to upskill their capabilities. This is to ensure they are confident in maintaining and the future development of the delivered solution. Last Updated: 26 June 2025, 09:13 19. Is there a minimum commitment term that TfW would agree to in terms of running the solution (e.g. long term commitment for 3rd party licence spend/support/maintenance and/or cloud consumption ) This will be discussed within the project. Last Updated: 26 June 2025, 09:11 20. If TfW is using an EA for it's Microsoft Azure consumption, is the minimum spend commitment currently being met / on target? We do not believe this is relevant information for a supplier to apply for this opportunity. Last Updated: 26 June 2025, 09:09 21. What SLAs apply for data freshness (pipeline latency) and interactive query response for business-intelligence users? Not all data will be required near real time and so data freshness will vary depending on use case. Last Updated: 26 June 2025, 09:07 22. For live queries against on-prem data, what latency threshold (< 10 s, < 60 s, etc.) is acceptable? Not all data will be required near real time, and so will vary depending on use case. Last Updated: 26 June 2025, 09:04 23. Capacity sizing - what volume of data is in scope, across how many systems? Please summarise by type / location if possible Scope and systems will be determined based on priority aligned to agreed use case delivery within the project. Last Updated: 26 June 2025, 09:02 24. Is it anticipated that the solution will be deployed purely to the cloud, or is a hybrid, or even physical infrastructure deployment possible? We expect the solution to be based on cloud only. Last Updated: 26 June 2025, 08:53 25. Is it anticipated that the solution will be deployed purely to the cloud, or is a hybrid, or even physical infrastructure deployment possible? We expect the solution to be based on cloud only. Last Updated: 26 June 2025, 08:52 26. Is it anticipated that the solution will be deployed purely to the cloud, or is a hybrid, or even physical infrastructure deployment possible? We expected the solution to be based on cloud only. Last Updated: 26 June 2025, 08:52 27. Does TfW already hold licences for Power BI Premium, Databricks or other analytics services that we can leverage? Decisions on licenses will be made as part of the project. Last Updated: 26 June 2025, 08:49 28. Will TfW procure any required third-party platform components itself, or should suppliers include a separate pricing line for those licences? These will be discussed as part of the project. Last Updated: 26 June 2025, 08:47 29. Is the £462k cap intended to cover professional services only, or does it also include hardware purchases, licensing costs, cloud consumption fees, post-implementation support, managed services? The budget is to cover the production of the agreed solution. In terms of licensing and cloud consumption costs these will be discussed with TfW as part of the project. Last Updated: 26 June 2025, 08:46 30. Can the Authority please outline which phase in the development process the current budget for this procurement is for? For example is this for the Alpha phase? The budget figure outlined is for all the development cycle until go live and agreed acceptable product to be defined within the project. Last Updated: 26 June 2025, 08:43 31. Will there be a further period allowed for clarification questions post the down-selection of suppliers to Stage 2? Yes, the three potential shortlisted suppliers will be allowed to submit clarification questions on the proposal/presentation requirements. Last Updated: 23 June 2025, 14:45 32. In bid pack attachment 1, Overview section, there are 2 contract lengths mentioned. Not sure why there are 2. Can you please elaborate a bit more on this ? 1 years, 6 months, 0 days 0 years, 4 months, 15 days. The expected contract length is 1 year and 6 months. The second figure is included as a 4 months optional contingency extension if needed. Last Updated: 23 June 2025, 14:43 33. What do the two different figures stated in the expected contract length refer to? The expected contract length is 1 year and 6 months. The second figure is included as a 4 months optional contingency extension. Last Updated: 23 June 2025, 14:39 34. What do the two different figures stated in the expected contract length refer to? The expected contract length is 1 year and 6 months. The second figure is included as a 4 months optional contingency extension. Last Updated: 23 June 2025, 14:38 35. Could the Authority please provide further clarity on the following 'nice to have' question: Organisations with organic and inorganic growth through acquisition and have legacy systems and strategic technology. Are you looking for a bidding organisation to have grown through acquisition and have both legacy systems and strategic Technology partners? TfW has grown by organisations coming together and continues to grow . With that comes the systems and technology from these organisations. Therefore, it would be a nice-to-have for a Bidder to have experience of working with organisations that have a complex technology landscape with a mixture of technologies. These could be legacy systems with ageing technology as well as new strategic technology. Last Updated: 23 June 2025, 12:55 36. Can the Authority please share who has completed the Discovery? Please refer to clarification question 1 above. Last Updated: 23 June 2025, 12:35 37. Under the "Nice-to-have skills and experience" section of Bid pack Attachment 3, one of the criteria requests “Experience working within the Transport sector,” which carries a significant weighting (50%). As an SME with extensive and demonstrable experience delivering similar services across other government departments and sectors, we are concerned that this criterion may disadvantage SMEs that have not yet had the opportunity to work directly within the Transport sector, despite possessing highly transferable skills and sector-relevant expertise. Could you please confirm whether bidders may provide evidence of analogous work in other public sector domains where the nature, complexity, and impact of the work are relevant and comparable to the challenges described in the Statement of Requirements? We believe this would allow for a broader and fairer assessment of supplier capability, in line with the stated aims of fairness and transparency in the competition. We are committed to reduce barriers for SMEs to tender for this work. That requirement is a nice-to-have and not an essential one, meaning you would not be disqualified if you cannot meet that requirement. The 50% that you reference is the weighting allocation between the two nice-to-have criteria and not the overall weighting of the question. Nice-to-have skills have been allocated 10% of the total technical competency score of 50% of the overall weighting. In simple terms someone that scores highly on this criterion will only be allocated 2.5% of the total weighting for this event. Last Updated: 23 June 2025, 12:31 38. Under the "Nice-to-have skills and experience" section of Bid pack Attachment 3, one of the criteria requests “Experience working within the Transport sector,” which carries a significant weighting (50%). As an SME with extensive and demonstrable experience delivering similar services across other government departments and sectors, we are concerned that this criterion may disadvantage SMEs that have not yet had the opportunity to work directly within the Transport sector, despite possessing highly transferable skills and sector-relevant expertise. Could you please confirm whether bidders may provide evidence of analogous work in other public sector domains where the nature, complexity, and impact of the work are relevant and comparable to the challenges described in the Statement of Requirements? We believe this would allow for a broader and fairer assessment of supplier capability, in line with the stated aims of fairness and transparency in the competition. We are committed to reduce barriers for SMEs to tender for this work. That requirement is a nice-to-have and not an essential one, meaning you would not be disqualified if you cannot meet that requirement. The 50% that you reference is the weighting allocation between the two nice-to-have criteria and not the overall weighting of the question. Nice-to-have skills have been allocated 10% of the total technical competency score of 50% of the overall weighting. In simple terms someone that scores highly on this criterion will only be allocated 2.5% of the total weighting for this event. Last Updated: 23 June 2025, 12:30 39. Under the "Nice-to-have skills and experience" section of Bid pack Attachment 3, one of the criteria requests “Experience working within the Transport sector,” which carries a significant weighting (50%). As an SME with extensive and demonstrable experience delivering similar services across other government departments and sectors, we are concerned that this criterion may disadvantage SMEs that have not yet had the opportunity to work directly within the Transport sector, despite possessing highly transferable skills and sector-relevant expertise. Could you please confirm whether bidders may provide evidence of analogous work in other public sector domains where the nature, complexity, and impact of the work are relevant and comparable to the challenges described in the Statement of Requirements? We believe this would allow for a broader and fairer assessment of supplier capability, in line with the stated aims of fairness and transparency in the competition. We are committed to reduce barriers for SMEs to tender for this work. That requirement is a nice-to-have and not an essential one, meaning you would not be disqualified if you cannot meet that requirement. The 50% that you reference is the weighting allocation between the two nice-to-have criteria and not the overall weighting of the question. Nice-to-have skills have been allocated 10% of the total technical competency score of 50% of the overall weighting. In simple terms someone that scores highly on this criterion will only be allocated 2.5% of the total weighting for this event. Last Updated: 23 June 2025, 12:30 40. Can you please confirm the incumbent who delivered the data maturity work? We do not believe this is relevant information for tenderers to submit a bid as we are still in the discovery stage and we are open to all capable suppliers. Last Updated: 23 June 2025, 12:26 41. Your budget is listed as £462,000, which is obviously quite specific. Can you share how you arrived at that figure, a breakdown and any assumptions you made? The estimated budget figure was included as a guide following an internal desktop estimation with the assumption of having at least a minimum acceptable end-to-end product, and there was no breakdown populated for the different elements, but rather a ballpark figure. Last Updated: 1 July 2025, 12:39