Job ID: JOB_ID_1938
Position Summary
KK Software Associates, LLC is looking for a skilled Databricks Engineer with over 8 years of experience to join our team in Pleasanton, CA. This role is focused on designing, developing, and optimizing sophisticated data pipelines and analytics solutions using the Databricks platform across various cloud environments including Azure, AWS, and GCP. The ideal candidate will be a technical expert in big data processing, capable of building scalable data architectures that support advanced business intelligence and machine learning initiatives.
Core Responsibilities
- Design and implement robust ETL/ELT pipelines using Databricks and Apache Spark to handle massive datasets.
- Develop and optimize data workflows to ensure high-performance, large-scale data processing and delivery.
- Integrate diverse data sources into unified data lakes and data warehouses, ensuring data consistency and accessibility.
- Collaborate closely with data architects and business analysts to translate requirements into scalable technical solutions.
- Implement comprehensive data quality checks, monitoring systems, and performance tuning strategies for big data jobs.
- Utilize Delta Lake to manage ACID transactions, data versioning, and schema enforcement within the Lakehouse architecture.
- Deploy and manage Databricks clusters, ensuring optimal resource allocation and cost-efficiency.
- Maintain strict compliance with organizational security and data governance standards throughout the data lifecycle.
- Participate actively in Agile ceremonies, contributing to continuous improvement and rapid delivery cycles.
Technical Skill Set
- Hands-on expertise with Databricks and Apache Spark is mandatory.
- Strong programming proficiency in Python and/or Scala for data engineering tasks.
- Advanced SQL skills for complex data manipulation, analysis, and optimization.
- Deep understanding of Delta Lake and the Lakehouse architectural pattern.
- Familiarity with cloud storage and compute services such as Azure Data Lake, AWS S3, or GCP Cloud Storage.
- Knowledge of CI/CD pipelines and DevOps practices tailored for data engineering (DataOps).
- Experience in performance tuning, troubleshooting, and debugging complex Spark jobs.
Professional Expectations
In this role, you will be expected to serve as a subject matter expert on Databricks best practices. You will not only build pipelines but also mentor junior engineers and contribute to the evolution of our data platform. The ability to work in a fast-paced, collaborative environment is essential. You will be responsible for ensuring that our data infrastructure is resilient, secure, and capable of meeting the growing demands of our data-driven organization. Your work will directly impact the ability of our business units to derive actionable insights from complex data sets.
Compensation & Location
Salary: $160,000 – $215,000 per year (Estimated)
Location: Pleasanton, CA
Recruiter / Company – Contact Information
Recruiter / Employer: KK Software Associates, LLC
Email: sushant.s@kksoftwareassociates.com
Recruiter Notice:
To remove this job posting, please send an email from
sushant.s@kksoftwareassociates.com with the subject:
DELETE_JOB_ID_1938