Data Engineer / Databricks Expertise Engineer/Azure

COMPEST SOLUTIONS INDIA PRIVATE LIMITED

Compest Solutions India Private Limited is on the lookout for a skilled Senior Data Engineer with Databricks expertise. This position exists to advance the company's data infrastructure and capabilities. The candidate will work within a dynamic team to optimize and scale up data processes to meet business needs.

In this role you can expect to have the responsibilities:

• Develop and maintain scalable ETL/ELT pipelines using Databricks.
• Utilize PySpark/Spark and SQL for data transformation and processing.
• Integrate data from various platforms such as Azure Blob Storage and ADLS.
• Collaborate with teams to prepare data for dashboards and BI tools.
• Optimize and monitor Databricks workloads for efficiency and reliability.
• Implement governance standards and manage data security with Unity Catalog.
• Deploy Databricks jobs and configurations using Databricks Asset Bundles.

This role comes with the following benefits:

This role requires you to have:

• Expertise in Databricks
• Proficiency in Azure Cloud Services
• Strong understanding of Spark and PySpark
• Experience with relational databases
• Knowledge of GitLab

You would benefit from having:

• Familiarity with streaming frameworks like Spark Streaming
• Experience in real-time data solutions development
• Azure Data Engineer Associate or Databricks Data Engineer certification

The role requires significant experience in Azure Data Factory, Azure Data Lake, Azure Synapse, and more. Certifications like Azure Data Engineer Associate are optional but preferred.