Integration Engineer | Blufeather Solutions | Worldwide

Role & responsibilities

Integration Engineer Azure Event Grid, Azure Logic Apps and Azure API Management

We are seeking a talented and experienced Integration Engineer to join our growing team. You will play a critical role in designing, implementing, and managing the integration of numerous source systems into a Databricks-hosted data warehouse following a medallion architecture. This position will involve leveraging event-driven architecture and technologies such as Azure Event Grid, Azure Logic Apps, Azure Functions, and Azure API Management to ensure seamless data flow and integration. The successful candidate will collaborate closely with technical teams, business stakeholders, and third-party vendors to ensure the successful delivery of this project which is part of a much larger transformation program.

Responsibilities:

• Assess and understand the structure, format, and volume of data from various source systems.
• Design, build, and maintain real-time data pipelines using Azure Event Grid, Azure Data Factory, and Azure Databricks.
• Architect event-driven workflows to process data from various source systems without intermediate storage, leveraging custom events in Azure Event Grid.
• Develop and manage event-based triggers in Azure Data Factory to automate the flow of data in near real-time from Event Grid to Databricks.
• Implement and manage Databricks notebooks or jobs to process, clean, and transform event data.
• Implement data transformation logic to cleanse, normalize, and enrich data before loading it into Databricks.
• Ensure data quality throughout the integration process, implementing data validation and cleansing mechanisms.
• Develop and maintain APIs using Azure API Management to expose data integration services to other applications or systems.
• Monitor and optimize data integration performance, identifying and addressing bottlenecks or inefficiencies.
• Create comprehensive documentation of data integration processes, including data flow diagrams, API specifications, and configuration details.
• Work closely with data architects, integration engineers, data quality engineers, and other stakeholders to understand data requirements and ensure alignment with the overall data architecture.

Preferred candidate profile

Qualifications:

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 3+ years of experience in data integration or a related field.
• Strong understanding of data warehousing and data lake concepts.
• Proficiency in data modeling and database design.
• Strong expertise in event-driven architectures and working with custom events in Azure Event Grid.
• Proficient in designing and deploying pipelines in Azure Data Factory, including the use of event-based triggers.
• Experience in developing scalable ETL/ELT data flows in Databricks, utilizing Spark for large-scale data processing.
• Solid understanding of SQL, Python, and Spark for data transformation and manipulation in Databricks.
• Extensive experience with Databricks using Spark.
• Proficiency in GitHub and GitHub Actions.
• Experience with Azure Log Analytics Kusto Query Language (KQL).
• Knowledge of data quality and validation techniques.
• Strong analytical and problem-solving skills.
• Excellent communication and documentation skills

Perks and benefits

Bonus Points:

• Certifications in data engineering or cloud technologies (e.g., Azure Data Engineer Associate).
• Experience with data integration tools and technologies (e.g., SSIS, Informatica, Talend).
• Knowledge of data governance and compliance frameworks.

Regards,

Narender K

narenderk@blufeathersolution.com

Direct: +91 8639123266

www.Blufeathersolutions.com

Tagged as: remote, remote job, virtual, Virtual Job, virtual position, Work at Home, work from home