Data Engineer With Extensive Scala Experience Ta X (Remote)

Our Customer is looking for Data Engineers who have extensive experience with Scala.

Main Responsibilities

• Data Pipeline Maintenance: Continuously monitor and maintain data pipelines for ingesting and transforming data using Scala and SQL on Spark. Diagnose and resolve errors and performance bottlenecks, addressing data discrepancies, ambiguities, and inconsistencies as needed.
• Technical Support and Version Control: Provide technical support for data analysis while managing source code and configuration artifacts via GitHub. Deploy code artifacts through GitHub Workflows/Actions.
• Technical Leadership: Offer hands-on technical guidance and leadership in developing Spark-based data processing applications using Scala, with a focus on Microsoft Azure Synapse Spark Runtime.
• Pipeline Optimization: Design and enhance data pipelines to streamline processing across various stages of the Medallion architecture using Azure Synapse Pipelines.
• Data Management: Oversee data ingestion processes, enforce data quality checks using tools like DQ, and manage validation and error-handling workflows.
• Configuration Management: Develop and manage configuration settings using JSON-based configurations (e.g., ApplicationConfig, TableConfig) for multiple data zones.
• Collaboration: Work closely with data scientists, analysts, and cross-functional teams to ensure smooth integration of data engineering efforts with marketing and business strategies.
• Logging and Auditing: Implement and manage logging, auditing, and error-handling practices to maintain data processing integrity, leveraging tools like Azure Log Analytics and KQL queries where applicable.
• Testing and Quality Assurance: Conduct unit testing with tools like ScalaTest and maintain rigorous data quality checks to ensure dependable processing outcomes.

Technical Requirements

• 7+ years of professional experience in Data Engineering
• Minimum 6-7 years of Experience in Data Engineering space on Scala & Spark + Azure.
• Fluent English
• Strong proficiency in programming languages such as Python and Scala
• Hands-on experience with data pipeline tools
• Expertise in relational and non-relational databases, including SQL
• Solid knowledge of cloud platforms and their data services.
• Familiarity with containerization and orchestration (e.g., Docker, Kubernetes) is a plus.
• Strong problem-solving skills and the ability to work in a fast-paced environment.
• Excellent communication and collaboration abilities.