Job ID: 1310
Role Overview
The Data Databricks Architect is a pivotal leadership and technical role responsible for the strategic design and implementation of the Databricks Lakehouse Platform. In this capacity, you will lead the transition from traditional data warehousing to a modern, unified architecture that supports data engineering, data science, and business intelligence on a single platform. This role requires a deep understanding of distributed computing, cloud-native services, and the evolving landscape of big data technologies.
Key Responsibilities
- Architecture & Design: Architect end-to-end data solutions using the Medallion Architecture (Bronze, Silver, Gold layers) to ensure data quality and reliability. Design scalable ingestion frameworks for structured and unstructured data.
- Engineering Excellence: Develop high-performance ETL/ELT pipelines using PySpark and Spark SQL. Implement Delta Lake features such as ACID transactions, schema enforcement, and time travel to maintain data integrity.
- Platform Governance: Leverage Databricks Unity Catalog to implement fine-grained access control, data lineage, and centralized governance across the enterprise.
- Performance Optimization: Conduct deep-dive performance tuning of Spark jobs, utilizing Z-Ordering, Data Skipping, and Photon engine capabilities to minimize latency and cloud costs.
- Cloud Integration: Seamlessly integrate Databricks with cloud-native ecosystems including Azure Data Factory, AWS Glue, or GCP Dataflow, ensuring secure connectivity via Private Links and managed identities.
- DevOps & Automation: Establish robust CI/CD pipelines using Databricks Repos and Terraform to automate infrastructure provisioning and code deployment.
Technical Environment
The successful candidate will operate in a high-velocity environment utilizing the latest Databricks features. You will be expected to lead Proof of Concepts (PoCs) for emerging tools like Delta Live Tables (DLT) and Serverless SQL Warehouses. Your expertise will guide the organization in adopting MLOps best practices through MLflow and Feature Store integration, enabling data scientists to move models from experimentation to production with minimal friction.
Strategic Impact
As a subject matter expert, you will bridge the gap between technical execution and business strategy. You will collaborate with stakeholders to define data retention policies, disaster recovery strategies, and cost-allocation models. By optimizing the Databricks environment, you will enable real-time analytics and advanced machine learning use cases that drive competitive advantage in the Los Angeles market and beyond.
Special Requirements
Hybrid role; Databricks Certified Data Engineer Professional preferred; experience in regulated industries like BFSI or Healthcare is a plus.
Compensation & Location
Salary: $185,000 – $245,000 per year (Estimated)
Location: Los Angeles, CA
Recruiter / Company – Contact Information
Recruiter / Employer: Digital Dhara LLC
Email: keshav.singh@digitaldhara.com
Recruiter Notice:
To remove this job posting, please send an email from
keshav.singh@digitaldhara.com with the subject:
DELETE_1310