Job ID: JOB_ID_1542
Role Overview: Lead Databricks Engineer
Veda Info is seeking a seasoned and visionary Lead Databricks Engineer to join our engineering team in San Jose, CA. This is a high-impact, 12-month contract position that requires an onsite presence in the heart of Silicon Valley. As a Lead Engineer, you will be responsible for architecting, building, and optimizing our client’s next-generation data platform. With the rapid evolution of AI and Big Data in 2026, this role demands a professional who is not only a master of the Databricks Lakehouse platform but also a strategic leader capable of driving complex data initiatives from conception to production. You will lead a team of talented engineers to deliver scalable, reliable, and high-performance data solutions that power critical business analytics and machine learning models.
Key Responsibilities and Duties
- Data Architecture and Design: Lead the design and implementation of robust data architectures using Databricks, Delta Lake, and Unity Catalog. You will ensure that the data platform is scalable, secure, and optimized for both batch and real-time processing.
- Pipeline Development: Architect and build complex data pipelines using Spark, PySpark, and Python. You will be responsible for the end-to-end ETL/ELT process, ensuring data is ingested, transformed, and delivered with high quality and low latency.
- Technical Leadership: Provide expert-level guidance and mentorship to the data engineering team. You will establish engineering best practices, conduct code reviews, and drive the adoption of modern DevOps and DataOps methodologies.
- Data Modeling and Optimization: Develop and maintain sophisticated data models that support diverse business use cases. You will optimize Spark jobs and SQL queries to ensure maximum performance and cost-efficiency within the Databricks environment.
- Quality and Governance: Implement comprehensive data quality frameworks, monitoring systems, and alerting mechanisms. You will ensure data integrity across the entire lifecycle and maintain compliance with data governance policies.
- Stakeholder Collaboration: Work closely with Data Scientists, Product Managers, and Executive Leadership to understand data requirements and translate them into technical solutions that drive business value.
- Problem Solving: Act as the primary point of escalation for complex technical issues. You will provide hands-on troubleshooting and root cause analysis for performance bottlenecks and system failures.
Required Skills and Qualifications
- A minimum of 12 years of total experience in data engineering, software development, or a related technical field is required.
- Expert-level proficiency in the Databricks platform, including Spark, PySpark, and Delta Lake.
- Advanced programming skills in Python and SQL are essential.
- Proven experience in building and managing large-scale data lakes and data warehouses in cloud environments (Azure, AWS, or GCP).
- Strong understanding of data modeling techniques (Star Schema, Snowflake, Data Vault) and data architecture patterns.
- Experience with CI/CD pipelines, version control (Git), and automated testing for data workflows.
- Excellent leadership and communication skills, with the ability to influence stakeholders and lead cross-functional teams.
- Strong analytical and problem-solving abilities, with a focus on delivering high-quality, production-ready code.
About Veda Info and the San Jose Opportunity
Veda Info is a premier technology services firm dedicated to delivering innovative solutions to the world’s most successful companies. By joining our team in San Jose, you will be at the epicenter of technological innovation, working on projects that define the future of data engineering. San Jose offers a unique professional landscape where you can collaborate with industry leaders and contribute to groundbreaking advancements in AI and Big Data. At Veda Info, we value expertise, creativity, and a commitment to excellence. We offer a highly competitive compensation package, reflecting the seniority and critical nature of this role. Our culture is built on collaboration and continuous improvement, providing you with the tools and environment needed to succeed as a leader in the field. If you are a data engineering expert with a passion for Databricks and a proven track record of leading successful data transformations, we invite you to join us in San Jose and help us build the future of data.
Special Requirements
Onsite requirement in San Jose, CA; 12 years total experience required; Expert proficiency in PySpark and Databricks mandatory.
Compensation & Location
Salary: $195,000 – $255,000 per year (Estimated)
Location: San Jose, CA
Recruiter / Company – Contact Information
Recruiter / Employer: Veda Info
Email: raj.kumar@vedainfo.com
Recruiter Notice:
To remove this job posting, please send an email from
raj.kumar@vedainfo.com with the subject:
DELETE_JOB_ID_1542