Job ID: JOB_ID_6577
Job Summary:
We are seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines, models, and infrastructure that power analytics, business intelligence, and machine learning products across the company. This role involves close collaboration with business, product, and analytics teams to translate complex requirements into elegant, reliable data solutions and drive the delivery of innovative data products. The position reports to the Senior Manager, Data Engineering.
Duties and Responsibilities:
- Build end-to-end Azure Databricks-based data solutions.
- Design, develop, and maintain scalable ETL and streaming data pipelines on Azure Databricks, leveraging PySpark, Delta Lake, and Azure Data Lake Storage (ADLS Gen2) to enable reliable Lakehouse architectures and ensure efficient ingestion, transformation, and storage of data.
- Build and optimize data models and schemas for analytics, reporting, and operational data stores.
- Build and optimize Delta Lake / Lakehouse patterns (Bronze/Silver/Gold), including schema evolution and time travel.
- Develop high-quality Python / PySpark / Spark SQL transformations, optimize joins, partitioning, caching, and shuffle behavior.
- Implement and maintain data quality frameworks, including data validation, monitoring, and alerting mechanisms.
- Collaborate closely with data architects, analysts, data scientists, and product teams to align data engineering activities with business goals.
- Leverage cloud data platforms (Azure, AWS, or GCP) to build and optimize data storage solutions, including data warehouses, data lakehouses, and real-time data processing.
- Develop automation processes and frameworks for CI/CD supported by version control, linting, automated testing, security scanning, and monitoring.
- Troubleshoot and resolve complex Azure Databricks platform, data infrastructure, and pipeline issues, ensuring minimal downtime and optimal performance.
Experience:
Required:
- A minimum of 12+ years of hands-on experience in data engineering, designing, and building scalable data pipelines, ETL/ELT processes.
- Extensive experience with cloud data platforms in Azure, AWS, or Google.
- Strong proficiency with Python and SQL for data processing.
- Proven experience building reusable, metadata-driven data ingestion frameworks using Python and/or Scala.
- Hands-on experience with modern data-platform components (object storage, Lakehouse engines, orchestration tools, columnar warehouses, streaming services).
- Proven experience with data modeling, schema design, and performance tuning of large-scale data systems.
- Deep understanding of data engineering best practices: code repositories, CI/CD pipelines, test automation, monitoring, and alerting systems.
Preferred:
- Experience building data pipelines in an Azure Databricks environment.
- Knowledge of Databricks architecture and core components, including Databricks Lakehouse, Delta Lake, Databricks SQL, Apache Spark Clusters, Unity Catalog, Databricks Workflows (Jobs), and Databricks Notebooks.
- Proficiency with Apache Spark for data processing.
- Hands-on experience integrating Azure Databricks with Azure DevOps, Azure Blob Storage / ADLS Gen2, Azure Key Vault, and Azure Data Factory.
- Experience migrating to or building data platforms from the ground up.
- Experience working in an Agile delivery model.
Special Requirements
In-person interview required. Locals only to Texas.
Compensation & Location
Salary: $120,000 – $160,000 per year (Estimated)
Location: Lewisville, TX
Recruiter / Company – Contact Information
Recruiter / Employer: Techstar group
Email: ul@techstargroup.com
Recruiter Notice:
To remove this job posting, please send an email from
ul@techstargroup.com with the subject:
DELETE_JOB_ID_6577