Senior Data Engineer Databricks (Remote)

Our client, a boutique Data & AI Consultancy in London is actively seeking multiple Senior Data Engineer Consultants to play a pivotal role in delivering innovative Databricks technical solutions to their end clients.

This client-facing role requires a unique blend of technical prowess in Databricks and exceptional communication skills to understand and address clients’ diverse data challenges..

Key Responsibilities:

Client Engagement:

• Collaborate directly with our clients to understand their business objectives, data challenges, and technical requirements.
• Translate our client’s needs into effective and scalable Databricks-based data engineering solutions.
• Foster strong client relationships through clear and proactive communication.

Solution Design and Implementation:

• Lead the design and implementation of robust data processing workflows using Databricks.
• Develop and deploy ETL (Extract, Transform, Load) processes to ensure efficient and accurate data delivery.
• Optimize data workflows for performance, scalability, and cost-effectiveness.

Technical Expertise:

• Demonstrate in-depth expertise in Databricks, Apache Spark, and related big data technologies.
• Stay informed about the latest industry trends and advancements in data engineering.

Quality Assurance:

• Conduct thorough testing and validation of Databricks solutions to ensure data integrity and reliability.
• Collaborate with our clients and internal teams to troubleshoot and resolve technical issues.

Documentation:

• Create and maintain comprehensive technical documentation for implemented Databricks solutions.
• Provide training and knowledge transfer to our clients and internal team members.

Innovation and Collaboration:

• Identify opportunities for innovation and improvement in data engineering processes.
• Collaborate with cross-functional teams to contribute to the overall success of our projects.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Proven experience in data engineering with a focus on Databricks and Apache Spark.
• Strong programming skills, preferably in Python or Scala.
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and associated data services.
• Excellent communication skills with the ability to convey technical concepts to non-technical stakeholders.
• Proven problem-solving skills and a proactive approach to addressing challenges.

Preferred Qualifications:

• Databricks certification or relevant professional certifications.
• Experience in a client-facing role within a consultancy or similar environment.
• Familiarity with start-up culture and the ability to thrive in a dynamic and fast-paced environment.