Job ID: JOB_ID_2002
Position Summary
We are seeking an elite AWS Databricks Data Engineer with a deep specialization in DevOps to join our high-performance data engineering team in Los Angeles. This hybrid role is designed for a technical leader who can navigate the complexities of the Databricks Lakehouse Platform on AWS while simultaneously driving modern CI/CD and infrastructure-as-code practices. In 2026, the boundary between data engineering and operations has blurred, and we need a professional who can design, build, and optimize multi-terabyte data pipelines with an automated, scalable mindset. You will be a ‘player-coach,’ providing hands-on engineering excellence while offering technical leadership to the broader team.
Core Responsibilities
- Design and implement sophisticated Databricks-based Lakehouse architectures on AWS, ensuring a clear and efficient separation between compute and serving layers for maximum performance.
- Develop high-performance data processing workflows using PySpark and advanced SQL, integrating diverse data sources including Amazon S3, relational databases, and semi-structured datasets.
- Implement Delta Lake best practices, including schema evolution, ACID compliance, and advanced optimization techniques like Z-Ordering and partitioning to handle massive data volumes.
- Drive the adoption of DevOps principles within the data team by implementing CI/CD pipelines using GitLab, GitHub Actions, or AWS-native tools, ensuring rapid and reliable deployments.
- Build and manage automated deployment frameworks using Databricks Asset Bundles and manage version control for all notebooks, workflows, and environment configurations.
- Automate infrastructure provisioning and configuration management using Terraform or CloudFormation, ensuring that our data environments are reproducible and secure.
- Collaborate with data analysts, BI teams, and product owners to translate complex business requirements into scalable technical implementations that support real-time and batch analytics.
- Tune Spark clusters and SQL queries for maximum cost-efficiency, monitoring pipeline health and implementing robust logging and observability frameworks.
- Maintain rigorous data governance and security standards using Unity Catalog, enforcing access controls and ensuring data lineage and auditability across all flows.
- Apply advanced caching strategies and workload optimization techniques to support low-latency API and data access patterns beyond traditional Spark execution.
Critical Skills & Architectural Competencies
The ideal candidate must demonstrate a mastery of multi-terabyte structured and time-series datasets. You should possess a keen ability for requirement probing, identifying the architectural nuances that matter most for performance and scalability. We value a mindset that prioritizes security, governance, and cost-optimization as much as technical functionality. Your experience with Unity Catalog and Delta Live Pipelines will be essential in building the next generation of our data platform.
Special Requirements
NO GC / NO GC EAD; Local Only for California; Hybrid work mode; FTE / CTH.
Compensation & Location
Salary: $155,000 – $210,000 per year (Estimated)
Location: Los Angeles, CA
Recruiter / Company – Contact Information
Recruiter / Employer: Quantum World Technologies Inc.
Email: gopal@quantumworldit.com
Recruiter Notice:
To remove this job posting, please send an email from
gopal@quantumworldit.com with the subject:
DELETE_JOB_ID_2002