Job ID: JOB_ID_692
Role Overview
VBeyond Corporation is seeking a highly skilled and hands-on Lead Data Engineer to join our client’s team in Chicago, IL. This is a pivotal leadership role designed for a senior developer who possesses deep technical expertise in Snowflake and Python. As a Lead Data Engineer, you will be at the forefront of architecting, developing, and optimizing cutting-edge data platforms that drive business intelligence and advanced analytics. This position requires a local candidate who can collaborate effectively within the Chicago tech ecosystem while leading a team of talented engineers toward technical excellence.
Key Responsibilities
- Lead the end-to-end development and deployment of Azure-based data platforms, leveraging ADLS, Azure Data Factory (ADF), and Databricks.
- Design and implement scalable data ingestion pipelines using Snowflake and Snowpipe to ensure real-time data availability.
- Develop complex data processing logic using Python and PySpark, focusing on performance and scalability.
- Architect and manage Kafka-based streaming solutions and EventHub integrations for high-throughput data processing.
- Drive CI/CD automation initiatives to streamline the software development lifecycle and ensure robust deployment processes.
- Perform advanced query optimization and performance tuning within Snowflake environments to reduce latency and cost.
- Evaluate and implement modern data tools, specifically articulating the advantages of DBT (Data Build Tool) over traditional alternatives.
- Collaborate with cross-functional teams to define data requirements and translate them into technical specifications.
- Mentor junior and mid-level engineers, providing guidance on best practices in SQL, Python, and cloud architecture.
Technical Requirements
- Extensive experience with Snowflake, including Snowpipe, Tasks, and Triggers.
- Proficiency in Python and PySpark for large-scale data manipulation.
- Strong command of SQL and deep understanding of query optimization techniques.
- Hands-on experience with Azure cloud services (ADLS, ADF, Databricks).
- Proven track record in streaming technologies such as Kafka and EventHub.
- Familiarity with CI/CD pipelines and automation tools.
- Deep understanding of the differences between virtual warehouses and serverless compute models.
- Experience with DBT for data transformation and modeling.
Leadership and Strategy
Beyond technical execution, the Lead Data Engineer will play a strategic role in shaping the data roadmap. You will be expected to provide thought leadership on emerging technologies and industry trends. Your ability to communicate complex technical concepts to non-technical stakeholders will be essential for the success of data-driven initiatives. We are looking for a professional who is not only a coder but a visionary capable of building resilient data architectures that can scale with the evolving needs of a global enterprise.
About VBeyond Corporation
VBeyond is a global recruitment company specializing in hiring top-tier IT professionals. We are fully committed to Diversity and Equal Employment Opportunity, ensuring that all qualified applicants receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, or national origin. Joining our network means gaining access to some of the most innovative projects in the technology sector.
Special Requirements
Local candidates only; Must have hands-on experience with Snowflake, Python, and Azure; Domain: Data Engineering & Cloud Platforms.
Compensation & Location
Salary: $165,000 – $215,000 per year (Estimated)
Location: Chicago, IL
Recruiter / Company – Contact Information
Recruiter / Employer: VBeyond Corporation
Email: shraddhap@vbeyond.com
Recruiter Notice:
To remove this job posting, please send an email from
shraddhap@vbeyond.com with the subject:
DELETE_JOB_ID_692