Job ID: 3179802
Overview of the Lead Databricks Platform Admin Role
This is an exceptional opportunity for a highly skilled and experienced Lead Databricks Platform Administrator to join a dynamic team in a hybrid capacity, based in St. Louis, MO. This contract position, with an initial duration of 12 months, is critical for designing, implementing, and managing scalable, secure, and reliable Databricks platforms. The successful candidate will play a pivotal role in ensuring the optimal performance, cost-efficiency, and governance of the Databricks environment, supporting various data science and machine learning initiatives. This role demands a blend of deep technical expertise, strong leadership capabilities, and excellent communication skills to engage effectively with both technical teams and client stakeholders. The ideal candidate will be a proactive problem-solver, capable of balancing technical depth with strategic decision-making in an Agile delivery model.
Key Responsibilities and Accountabilities
- Platform Design and Implementation: Lead the design, implementation, and ongoing management of robust, secure, and highly available Databricks platforms tailored to organizational needs. This includes creating and managing Databricks workspaces and clusters, focusing on rightsizing, lifecycle management, and performance optimization.
- Infrastructure Automation: Drive the automation of infrastructure provisioning and deployments using cutting-edge Infrastructure-as-Code (IaC) tools such as Terraform, integrated with CI/CD pipelines like GitLab CI/CD, and scripting languages including Python and Bash/PowerShell.
- Monitoring and Troubleshooting: Establish and maintain comprehensive monitoring solutions to track platform health, performance metrics, and cost utilization through intuitive dashboards and integrated monitoring tools. Proactively troubleshoot and resolve complex cluster, job, and performance issues across various Databricks environments.
- Security and Governance: Implement stringent security and governance policies, including advanced user management, fine-grained access controls, and compliance adherence utilizing Databricks Unity Catalog and other native security features.
- Cost Optimization: Continuously optimize compute and storage costs without compromising performance, leveraging advanced Databricks capabilities such as Delta Lake, Photon engine, and intelligent cluster management strategies.
- Team Leadership and Mentorship: Provide strong leadership and mentorship to both onsite and offshore technical teams, fostering a collaborative and high-performing environment. Balance technical guidance with strategic decision-making to achieve project goals.
- Stakeholder Engagement: Engage confidently and effectively with client stakeholders, articulating complex system designs, architectural patterns, and platform strategies to varied audiences, including senior leadership.
- Documentation and Knowledge Transfer: Own the creation and maintenance of comprehensive platform documentation, operational runbooks, and knowledge transfer materials to ensure platform sustainability and empower junior engineers.
- Agile Delivery: Work effectively within Agile delivery models, participating in sprint planning, stand-ups, and retrospectives to ensure timely and efficient project execution.
- Continuous Improvement: Stay abreast of the latest Databricks feature releases and coordinate with product teams to evaluate and strategically adopt new capabilities that enhance platform functionality and efficiency.
Required Skills and Experience
The successful candidate will possess a strong background in Databricks platform administration and engineering, coupled with proven leadership abilities. This includes extensive experience with cloud platforms, data warehousing, and big data technologies. A deep understanding of data governance, security best practices, and cost management in a cloud-native environment is essential. The ability to communicate complex technical concepts clearly and concisely to both technical and non-technical audiences is paramount. Experience with large-scale data pipelines and machine learning operations (MLOps) is highly desirable. This role requires a proactive individual who can drive initiatives, solve complex problems, and contribute significantly to the organization’s data strategy.
Special Requirements
Hybrid role requiring onsite presence in St. Louis, MO. Candidates must possess strong communication and customer management skills. Experience with Agile delivery models is essential.
Compensation & Location
Salary: $120,000 – $180,000 per year (Estimated)
Location: St. Louis, MO
Recruiter / Company – Contact Information
Recruiter / Employer: Agile Enterprise Solutions Inc.
Email: bhavanesh@aesincus.com
Recruiter Notice:
To remove this job posting, please send an email from
bhavanesh@aesincus.com with the subject:
DELETE_3179802