NEWPosted 3 hours ago

Job ID: JOB_ID_6675

Job Summary

Azure Databricks Senior/Lead Engineer to join our team for a 12-month engagement based in the US (preferably NY/NJ area). The ideal candidate will bring strong expertise in Azure Databricks, Apache Spark, and data engineering best practices, along with experience in platform administration and optimization.

This role requires close collaboration with client stakeholders and offshore teams to design, build, and manage scalable data solutions on Azure.

What You’ll Bring:

  • Strong hands-on experience with Azure Databricks and Apache Spark.
  • Proficiency in Python and/or Scala with strong SQL skills.
  • Experience designing and implementing scalable data pipelines.
  • Hands-on experience with Databricks cluster configuration and administration.
  • Experience with CI/CD pipelines and DevOps practices.
  • Strong troubleshooting and performance tuning skills.
  • Excellent communication and stakeholder management skills.

Roles & Responsibilities

What You’ll Do:

Data Pipeline Design & Development
  • Design and build scalable, high-performance data ingestion pipelines using Azure Databricks and Apache Spark.
  • Develop robust ELT/ETL workflows to prepare data for analytics and reporting.
  • Leverage Python, SQL, or Scala for data transformation and processing.
  • Working knowledge of DBT is a plus.
Data Transformation & Processing
  • Develop and maintain transformation scripts and Databricks notebooks.
  • Implement reusable and modular data processing frameworks.
  • Ensure efficient handling of large-scale datasets.
Databricks Platform Administration
  • Perform Databricks platform administration activities.
  • Configure and manage clusters for optimal performance and cost efficiency.
  • Manage cluster policies including autoscaling, instance types, and runtime versions.
  • Monitor cluster health and tune Spark configurations to improve job performance.
Data Quality & Governance
  • Implement data quality checks and validation frameworks within notebooks and workflows.
  • Ensure data accuracy, consistency, and reliability across pipelines.
Automation & CI/CD
  • Develop and maintain CI/CD pipelines for automated deployment of notebooks, jobs, and configurations.
  • Promote DevOps best practices for version control and release management.
Monitoring, Troubleshooting & Optimization
  • Monitor Databricks jobs, clusters, and pipelines for failures and performance bottlenecks.
  • Analyze platform usage metrics and troubleshoot processing or infrastructure issues.
  • Optimize Spark jobs and cluster configurations to improve execution time and efficiency.
Cost Management
  • Track and analyze Databricks platform usage and associated costs.
  • Optimize resource utilization and cluster configurations to reduce expenses.
  • Implement governance policies to prevent cost overruns.
Collaboration & Leadership
  • Collaborate with client and offshore teams to deliver scalable solutions.
  • Provide technical leadership, best practices guidance, and platform support.
  • Act as a subject matter expert for Azure Databricks implementations.

Special Requirements

Hybrid Onsite, NY/NJ area

Compensation & Location

Salary: $120,000 – $160,000 per year (Estimated)

Location: New York, NY

Recruiter / Company – Contact Information

Email: jobs@metarpo.com

Interested in this position?
Apply via Email

Recruiter Notice:
To remove this job posting, please send an email from
jobs@metarpo.com with the subject:

DELETE_JOB_ID_6675

to delete@join-this.com.