Data Engineerring and Extraction Specialists

Description

Summary of work GDS require a team with good knowledge PHP and Ruby to develop, run and support the GovWifi from public beta to a live service. Where the supplied staff will work London Where the supplied staff will work No specific location (for example they can work remotely) Who the organisation using the products or services is National Crime Agency Why the work is being done As a law enforcement agency, we collect data from a range of sources and we need to decrease the amount of time taken to ingest this. We have built several tools within AWS which seek to extract this data from a variety of formats and make it available to analyse for users. The volume of data is increasing and it is currently a manual process to extract data from its original format. The business problem We currently have an existing framework for data extraction which we want to expand and utilise to automate as much of the data extraction phase as possible to speed up our data ingestion process by having an agile team of data engineers to work alongside internal developers and data scientists to bring value to users at pace. The NCCU does not currently have much dedicated data extraction resource as such our delivery to users is slowed. This requirement will address this. The people who will use the product or service User type: Users Definition: National Cyber Crime Unit and wider policing colleagues Work done so far There is an existing environment/framework and part of the process which we are looking to build on. Which phase the project is in Beta Existing team Some internal developer resource Address where the work will be done Spring Gardens, Units 1-6 Citadel Place, Tinworth Street, London SE11 5EF Working arrangements Hybrid working resources required to attend Spring Gardens on a regular basis (2 days per week) to collaborate and sync with internal development teams. No expenses can be charged for attendance at the stated location address. Other travel expenses will be charged at cost and within the NCA Travel and Subsistence Policy Security and vetting requirements Security Check (SC) More information about the Security requirements: SC with NCA enhanced checks. The NCA will fund the cost of obtaining the NCA SC Enhanced checks Latest start date 4 September 2023 Expected contract length Contract length: 0 years 8 months 0 days Optional extension: 0 years 4 months 0 days Special terms and conditions special term or condition: 1. Official Secrets Act 1911 to 1989, S182 of the Finance Act 1989 1.1 The Contractor shall comply with, and shall ensure that its Staff comply with: 1.1.1 the provisions of the Official Secrets Acts 1911 to 1989; and 1.1.2 the provisions of Section 182 of the Finance Act 1989. 1.2 In the event that the Contractor or its Staff fail to comply with this clause, the Authority reserves the right to take any or all of the following actions: 1.2.1 terminate the Contract by giving notice in writing to the Contractor; 1.2.2 to initiate criminal proceedings special term or condition: 2. Contractor’s Staff 2.1 The Contractor must have policies and processes in place covering pre and post-employment checks for all staff including compliance with Employment Legislation relating to identity checks and establishing ‘right to work’ in the UK. 2.2 At the Authority’s written request, the Contractor shall provide a list of the names and addresses of all persons who may be employed on the Contract specifying the capacities in which they are concerned with the Contract and giving such other particulars as the Authority may reasonably request. 2.3 The Contractor shall comply with the Authority’s Staff Vetting Requirements as detailed in the specification in respect of all persons employed or engaged in the provision of the Services. 2.4 The Authority may, by written notice to the Contractor, refuse to admit onto, or withdraw permission to remain on any of the Authority’s sites: 2.4.1 any member of the Contractor’s Staff; or 2.4.2 any person employed or engaged by any member of the Contractor’s staff, whose admission or continued presence would, in the reasonable opinion of the Authority, be undesirable. 2.5 The Authority shall not disclose its reasons for considering an individual unsuitable. Any such unsuitability is not an indication that the individual is unsuitable to work on other Contracts for the Contractor. 2.6 The decision of the Authority as to the suitability of any person employed upon this service shall be final and conclusive. special term or condition: 3. Publicity and Branding 3.1 The Contractor shall not (and shall procure that no Sub-contractor shall) make any advertisement, public statement or press announcement in relation to this Contract or the provision of the Goods or Services without the Authority’s prior written consent. 3.2 The Contractor shall not use any of the Authority’s Intellectual Property Rights, including (but not limited to) the Authority's names, branding, domain names, logos or trade marks on any of its products or services or in any promotional, marketing or similar material or advertising without the Authority’s prior written consent. 3.3 The Contractor hereby agrees and acknowledges that the unauthorised use of the Authority’s names, branding, domain names, logos and/or trade marks may be breach of Crown copyright and that legal action may be taken against it in the event of any unauthorised use of such material. This applies to all Authority-owned brands including (but not limited to) the Authority's corporate identity, Child Exploitation and Online Protection (CEOP) command brands and/or campaign brands. All requests for external use of a NCA brand must be approved in writing by the Authority branding team and the Contractor shall in each case seek to obtain such approval in advance by submitting an approval request to branding@nca.gov.uk. 3.4 Nothing in this Contract constitutes an endorsement by the Authority of the Contractor's goods or services, and the Contractor shall not conduct itself in a way that implies any endorsement or authorisation by the Authority. special term or condition: 4. Freedom of Information 4.1 For the purpose of the FOIA, the Authority is not a 'public authority'(as listed in Schedule 1 of the Act), and is not under any duty to disclose information. The Authority is also not listed as a ‘Scottish Public Authority’ in the Freedom of Information (Scotland) Act 2002. In addition any information from, or relating to the Authority has an absolute exemption from disclosure by other public authorities by virtue of Section 23 of the Act (as amended by the Crime and Courts Act 2013). Any request for information must be referred to the Authority’s FOI single point of contact, by email to PICUEnquiries@nca.gov.uk or by telephoning 0870 268 8677. 4.2 Additionally, the Authority will not provide a public authority with any Information in respect of a Request for Information they receive. However, Information received from, (either directly or indirectly), or “relating to” the Authority that is already held by the public authorities or their sub-contractors, has additional protection from release to the public by virtue of section 23 of the FOIA. The Authority’s Information may also be covered by the FOIA Section 24 exemption, this is applicable to Information the non-disclosure of which is necessary to safeguard national security. 4.3 It is important that the Contractor understands the different legislation covering the Authority under Section 23 of the FOIA, so as not to inadvertently breach the FOIA. 4.4 The Contractor shall acknowledge the Authority’s position in respect of the FOIA and agree to assist and co-operate in order to enable the Authority to best protect its Information. The Contractor shall also ensure that any sub-contract the Contractor enters into also contains a condition in similar terms. 4.5 Failure to comply with this policy may result in an unauthorised disclosure of the Authority’s information and a potential breach of the Official Secrets Act. 4.6 The provisions of this Clause shall survive termination or expiry of the Contract. special term or condition: 5. Security Measures Please see attached Document special term or condition: 6. Security Requirements and Information Security 6.1 The Contractor must meet all the Authority’s relevant security policies as notified by the Authority, from time-to-time. 6.2 All Staff employed on the Contract, including the Contractor’s and Sub-contractor’s staff, must hold the appropriate level of security clearance, as stated by the Authority. 6.3 The Contractor must comply with Her Majesty’s Government Security Policy Framework and the Government’s Security Classification and any amendments or revised editions to this document which may be issued from time-to-time. 6.4 The Contractor shall bring to the attention of the Authority any business development outside of the Authority’s areas of responsibility which is likely to give rise to a security risk. 6.5 The Contractor shall have in place appropriate security systems and policies at all sites within its organisation, and at any relevant Sub-contractors, in relation to the handling and storage of any Authority related documentation and/or assets. 6.6 The Contractor shall ensure, and provide written evidence, that an Information Security Policy is in place. The policy shall also cover Confidential Information and Authority Data risk. The policy must be communicated to any individuals with access to systems where any Confidential Information and/or Authority Data are processed. 6.7 The Contractor shall ensure that all access to, and use of, any Authority Data or Confidential Information involving Authority business (including, but not limited to, Personal Data, financial records such as invoices and any other data, as well as the tracking of the assets) shall be controlled and that only authorised personnel shall have access. Robust controls and capabilities for access, auditing and monitoring shall be in place to ensure: 6.7.1 the prevention of attempts to gain access, and 6.7.2 identification of actual unauthorised access. 6.8 The Contractor shall allocate responsibilities for Information Security wherever Authority business or information or assets are involved and the Contractor shall take all reasonable steps necessary to protect Information Security. 6.9 Where the Contractor uses computer systems to process any Authority Data (including financial records such as invoices and any other data as well as the tracking of the assets), the Contractor must ensure the confidentiality, integrity and availability of any of Authority Data. 6.10 Where the Contractor is involved with the transferring of any Authority related data, such transfer must be controlled, and conform with protection measures appropriate to the impact level and or risk assessment. 6.11 The Contractor shall ensure that Authority Data is held within the United Kingdom and is not capable of being accessed from outside of the United Kingdom. Budget Indicative maximum: £800000 Indicative minimum: The contract value is not specified by the buyer Contracted out service or supply of resource? Contracted out service: the off-payroll rules do not apply Questions and Clarifications 1. what are the data source types? Mobile data records, Internet Connection Records, Video Feeds etc. Mostly relational databases from many different sources Last Updated : <strong>07/07/2023</strong> 2. What is the current presentation layer? Presentation layer custom application produced by collaboration with other international law enforcement partners Last Updated : <strong>07/07/2023</strong> 3. Good Afternoon, Please can you double check and confirm if the documents are the correct ones? The brief doesn't seem to match the client? Many thanks Meera Updated and amended accordingly Last Updated : <strong>07/07/2023</strong> 4. Hi Can you confirm if there is an incumbent supplier and who they are, please? There is no incumbent supplier Last Updated : <strong>07/07/2023</strong> 5. Please can you confirm that at this stage, you just need responses to Essential Skills and Experience, and Nice-to-have Skills and Experience? You should only provide a response to the Essential and nice to have skills Last Updated : <strong>07/07/2023</strong> 6. Is the requirements solely for data engineers or would the authority be looking for additional supporting resource such as a Cloud developer or SCRUM master? Please refer to Essential skills and experience. We have requested expertise in Terraform and AWS skills and expect resources to have strong competence in using AWS services for data engineering tasks. we only need very light touch management of resources. Last Updated : <strong>07/07/2023</strong> 7. Please can you confirm if the 750 character count includes or excludes spaces? The 750 character count includes spaces Last Updated : <strong>07/07/2023</strong> 8. can you confirm that a team comprised of full-time and part-time data engineers is acceptable? If so, is there flexibility for part-time staff to work 1 day a week on-site? A team of both full time and part time is acceptable. We are willing to discuss flexibility around on site working. Last Updated : <strong>07/07/2023</strong> 9. what are the data source types? Mobile data records, Internet Connection Records, Video Feeds etc. 2. What is the current presentation layer? 3. Are the data sources fed into the current system live or offline? 4. Do you maintain the raw data form after processing? 5. Is Kafka part of the solution? 6. Do you also employ a graph database (relationship – nodal based data)? 1. Mostly relational databases from many different sources 2. Presentation layer custom application produced by collaboration with other international law enforcement partners 3. Mostly offline some aspiration to ingest live but not a priority 4. Yes we do but we would only be interested in the winner working on working copies of the data, raw data is handled separately. 5. No we use RabbitMQ Last Updated : <strong>07/07/2023</strong> 10. Can you please confirm if this is an outcome or specialist opportunity? If it is specialists, how many do you require and could you provide max day rate exc vat? Many thanks in advance This is an outcome opportunity Last Updated : <strong>07/07/2023</strong> 11. Please can you confirm if the 750 word count is 750 WORDS for the total response (spread across the 7 sections), or if it is 750 WORDS per question After clarification with the CCS it is agreed that the response should be 750 characters including spaces per Question. Please bear in mind this is a maximum. Please not this is a change to the previous advice. Please use the template when responding to the requirement. Last Updated : <strong>07/07/2023</strong> 12. What percentage of the data by volume (gigabytes) and number of data sources (e.g. tables) is structured (e.g. SQL) or unstructured (e.g. EnCase, XML, etc.)? Backlog of 50+ SQL files which are low tens of gigabytes generally (This varies and some are bigger/ smaller). Most of the data arrives packed in E01 file formats so extraction is required Last Updated : <strong>07/07/2023</strong> 13. can you confirm that you are able to sponsor SC as well as the SC Enhanced checks? The NCA would be looking for the supplier to provide SC cleared staff. Last Updated : <strong>07/07/2023</strong> 14. Is the team that built the current framework available to work alongside the project team? Mostly Last Updated : <strong>07/07/2023</strong> 15. 1. To what extent is the infrastructure required for the project defined as code (e.g. terraform, CDK) or will we have to provision resources manually or have them provisioned for us by other teams? If so, what types of resources are provisioned by a different team? 2. Can you confirm how the K8s clusters are currently hosted (e.g. EKS, self-managed on EC2 etc.)? If self managed, will the project team be responsible for the cluster administration? 3. Your documentation mentions Elasticsearch is air gapped. Air-gapped implies no physical network link. Is this the case for your ES setup or does it mean isolated on a logical network level, except for needed ports/protocols? Can you briefly describe the characteristics of this setup? Specifically how is data loaded from AWS into ES, how is the API queried and data safely returned, how is the cluster administered and upgraded? 4. Will this project's infrastructure be expected to be deployed on a Hybrid solution (e.g. AWS Outposts) or AWS Cloud Native? If hybrid, can you please confirm if this is a third party, the hypervisor (e.g. VMware), and the product used for a secure connection from on-prem to the cloud (e.g. AWS Direct Connect) 1. We generally seek to use Terraform for all projects. We are a small team and you will have access to our internal DevOps/ platform team who are able to provision infrastructure, 2.EKS 3. logical separation as AWS based. 4. AWS Cloud Native. We are reviewing AWS storage gateway as a mechanism to move data into the cloud. We have used AWS Snowball to move large volumes of data into the cloud where necessary. Last Updated : <strong>07/07/2023</strong> 16. 1.Can you briefly describe the process of reviewing the infrastructure code on this project? For example are internal peer reviews enough, or do we need to get approval from other internal people or teams. 2. Can you briefly describe the process of promoting code to higher environments? For example, does going from QA to production require extensive manual testing by highly qualified internal resources. 3. Can you briefly describe the process of introducing new tools (e.g. Docker) to the project? For example, does it require review or approval by an internal technical domain authority. 4. Can you briefly describe the process of introducing new libraries (e.g. pytest) to the project? For example, does it require review or approval by an internal technical domain authority. 1. We would generally do an internal peer review 2. We provide opinionated guide lines and run some automated checks via our CI/CD pipelines 3. NCCU would act as the domain authority and generally we would seek to assess based on ongoing supportability/ commercial ease of procurement where necessary. 4. Proxied access to internet reps with scanning Last Updated : <strong>07/07/2023</strong> 17. The issued DOS 6 notice states 750 characters per question response. 'Clarification Questions - Set 1 - Issued' states "Please can you ensure your response is no more than 750 WORDS". However, the issued Excel 'Response Template - DOS' has a 750 CHARACTER limit placed on each Excel cell. Please can you communicate: 1. whether NCA is stipulating 750 characters or 750 words, 2. (if using characters) whether the limit is with or without spaces, and 3. a new 'Response Template - DOS' (if required based on answers to the above) It is 750 characters including spaces and please use the Response template Last Updated : <strong>07/07/2023</strong>