Data Test Engineer (SC Cleared)

# About Scrumconnect [Scrumconnect Consulting](https://www.scrumconnect.com/?utm_source=chatgpt.com) is a UK-based digital transformation consultancy delivering agile, secure, and scalable technology solutions for public and private sector clients. This is a fully remote role based in India, supporting UK operations and client engagements. The successful candidate will work closely with UK leadership teams and must be comfortable operating within UK regulatory, governance, and delivery frameworks. * * * # About the Role We are looking for an experienced **Test Manager** to lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines **hands-on technical leadership** with **test strategy ownership**. You will be responsible for defining and driving the overall testing approach across automated pipelines, data workflows, cloud infrastructure, and release delivery while managing risks, improving testing standards, and mentoring engineering teams. The ideal candidate will have strong experience in **Big Data and Data Lake environments**, including testing both structured and unstructured datasets, and will be comfortable operating in highly automated CI/CD-driven delivery environments. You will work closely with engineering, DevOps, architecture, and stakeholder teams to ensure quality is embedded throughout the software delivery lifecycle. * * * # Key Responsibilities ## Test Strategy & Leadership - Define, implement, and manage the overall test strategy across cloud-native data platforms and applications - Lead end-to-end testing activities across data pipelines, cloud services, APIs, and distributed systems - Establish quality standards, governance processes, and testing best practices across teams - Drive a quality-first engineering culture across the programme - Provide leadership, mentoring, and coaching to Test Engineers and QA team members ## Automated Testing & Frameworks - Design, implement, and oversee scalable automated test frameworks for data and application testing - Build and maintain automated test suites across AWS cloud infrastructure and data workflows - Enhance existing automation frameworks to improve reliability, coverage, maintainability, and reporting - Ensure automated testing is fully integrated within GitLab CI/CD pipelines for rapid feedback and release confidence ## Big Data & Data Lake Testing - Lead testing activities across Big Data and Data Lake solutions containing structured and unstructured datasets - Validate large-scale data ingestion, transformation, and processing workflows - Ensure data integrity, transformation accuracy, reconciliation, and performance under load - Oversee PySpark-based data validation and interrogation activities on AWS EMR - Support testing and monitoring of workflows orchestrated through Apache Airflow - Drive root cause analysis for production data issues and platform defects ## Risk Management & Reporting - Identify, assess, and communicate quality risks across releases and environments - Analyse test execution metrics, defects, trends, and quality indicators - Translate technical findings into clear, prioritised reporting for technical and non-technical stakeholders - Provide regular status reporting to programme leadership and client stakeholders ## CI/CD & DevOps - Collaborate with DevOps and engineering teams to improve release quality and deployment confidence - Ensure automated test execution within GitLab CI/CD pipelines - Support infrastructure testing and validation within Terraform-managed cloud environments - Contribute to continuous improvement initiatives across testing and delivery processes * * * # Technology Stack - Apache Airflow, Amazon Athena, AWS S3, AWS Glue, AWS EMR, AWS EC2, AWS ECR, AWS DynamoDB, AWS CloudWatch, AWS IAM, Python, SQ…