Job ID: JOB_ID_2003
Role Overview
As a Senior Data Engineer in the 2026 landscape, you will be a pivotal member of our technical team in Pleasanton, CA. This role is designed for a visionary engineer who can bridge the gap between complex data architectures and actionable business insights. You will be responsible for the end-to-end lifecycle of data, from ingestion to consumption, ensuring that our data infrastructure is not only scalable but also highly resilient and secure. In an era where data is the lifeblood of the enterprise, your expertise in Databricks and cloud-native technologies will drive our competitive advantage.
Key Responsibilities
- Design, develop, and maintain sophisticated end-to-end data pipelines leveraging the Databricks platform, specifically utilizing PySpark, Spark SQL, and Delta Lake for both high-velocity real-time processing and massive batch workloads.
- Architect and implement robust data lake and data warehouse solutions across major cloud environments including Azure, AWS, and GCP, ensuring seamless cross-cloud data mobility and integration.
- Develop, optimize, and automate complex ETL/ELT workflows using industry-standard orchestration tools such as Apache Airflow, Azure Data Factory, and AWS Glue to ensure high availability and reliability of data assets.
- Build and manage modern data lakehouse architectures that integrate Databricks, Snowflake, and Delta Lake, providing a unified and high-performance environment for advanced analytics and machine learning.
- Collaborate deeply with business stakeholders, analytics teams, and data scientists to translate complex business requirements into scalable, high-performance data solutions that meet the evolving needs of the organization.
- Ensure the highest standards of data quality, consistency, and reliability through the implementation of comprehensive validation frameworks, rigorous data governance practices, and master data management (MDM) strategies.
- Implement and enforce best practices for data security and strict compliance with global regulations, including CCPA, GDPR, and HIPAA, ensuring that data privacy is maintained at every stage of the pipeline.
- Support continuous performance tuning and optimization of Spark clusters, SQL queries, and storage layers to minimize latency and maximize cost-efficiency in cloud resource utilization.
- Participate in the implementation of CI/CD pipelines for data engineering using modern DevOps tools such as Jenkins, Git, and Terraform, fostering a culture of automated testing and deployment.
- Contribute to the establishment of Data Centers of Excellence (CoE) and champion data democratization initiatives to empower users across the organization with self-service analytics capabilities.
Technical Environment
The successful candidate will work within a cutting-edge ecosystem that prioritizes automation and scalability. You will be expected to stay ahead of the curve in the 2026 data engineering space, exploring new features in Delta Lake and Spark to solve complex data challenges. Your role involves not just technical execution but also strategic thinking regarding how data flows through the organization to create value.
Special Requirements
Locals Preferred; Compliance with CCPA, GDPR, and HIPAA regulations required.
Compensation & Location
Salary: $145,000 – $195,000 per year (Estimated)
Location: Pleasanton, CA
Recruiter / Company – Contact Information
Recruiter / Employer: Smart TechLink Solutions Inc.
Email: iswarya.k@smarttechlink.com
Recruiter Notice:
To remove this job posting, please send an email from
iswarya.k@smarttechlink.com with the subject:
DELETE_JOB_ID_2003