Senior Data Engineer (Remote)

Job Title: Senior Data Engineer / Developer (QA focused)

Job Overview

We are seeking a highly experienced Senior Data Engineer / Developer to establish and lead a robust Quality Assurance (QA) practice within our Data Engineering team. This role will be pivotal in improving QA execution and coverage for our modern, Azure-based data platform, which leverages Microsoft Fabric and Power BI for data visualization.

The ideal candidate will bring extensive Data Engineering experience, sharp development skills, and experience testing data sets and applications at scale. They will be responsible for driving best practices and collaborating effectively with team members to ensure high-quality deliverables across all stages of the data lifecycle.

Key Responsibilities

• Data Engineering: extensive hands-on experience on a modern data platform, illustrating both independent skills and capabilities as well as team leadership experience.
• BI / Data Reporting Development: Demonstrable experience in modern coding platforms, especially PySpark, C#, M-SQL, etc. Hands-on, day-to-day coding skills are a must.
• Database architecture: Strong knowledge of database architecture, including data modeling, indexing, partitioning, and query optimization for large-scale data environments.
• QA Practice Leadership: Define and implement QA best practices tailored to data engineering workflows, ensuring thorough testing coverage and quality assurance across the platform.
• Coding: Active hands-on coding expertise, with current proficiency in C#, PySpark, Python, T-SQL/M-SQL, and other relevant programming languages and frameworks.
• Data Application Testing: Develop and execute comprehensive test plans for data applications built on modern architectures, including Data Lake and Delta.
• Expertise in Azure Synapse and Data Lake architecture, including designing, implementing, and optimizing scalable data solutions within these platforms.
• Testing Tools & Frameworks: Utilize tools such as Great Expectations or Python-based testing harnesses to automate and enhance data quality testing.
• Collaboration & Communication: Provide mentorship to junior QA analysts, clearly articulating testing strategies and objectives. Collaborate with data engineers and business stakeholders to translate functional requirements into detailed test cases.
• UAT Management: Facilitate user acceptance testing (UAT) by organizing artifacts, leading meetings, and providing regular status updates.
• Data Validation: Conduct rigorous data validation using intermediate-to-advanced SQL and/or Python skills to ensure data quality and accuracy.
• ACA: Experience with Azure Container Apps (ACA), including deploying, managing, and optimizing containerized applications in a cloud-native environment.
• Compliance Support: Support compliance requirements, including SOX testing, where applicable.
• Continuous Improvement: Monitor QA execution results, identify areas for improvement, and drive solutions to enhance efficiency and effectiveness.

Required Qualifications

• 6–10 years of experience as a Data Engineer/Developer on a modern data platform.(developer background and current coding skills used in daily practice are mandatory)
• Demonstrated expertise in defining and driving QA best practices for large-scale data applications.
• Proficiency with testing tools such as Great Expectations or Python testing harnesses.
• Advanced SQL and/or Python skills for data validation and automation.
• Hands-on experience with Data Lake/Delta architecture and QA testing in these environments.
• Ability to effectively translate business and functional requirements into detailed and actionable test cases.
• Proven experience leading UAT processes, including documentation, communication, and reporting.
• Familiarity with ETL development (preferred).
• SOX compliance experience (a plus).
• QA focus must be data-focused, not UI/UX

Preferred Skills

• Strong problem-solving skills with attention to detail and quality.
• Excellent verbal and written communication skills.
• A proactive approach to identifying risks and resolving issues collaboratively.