Job ID: JOB_ID_6116
Job Description: Data Engineer – Azure
We are seeking a strong core Data Engineer with hands-on experience in PySpark, Databricks, and Azure data platforms to design, build, and support end-to-end data pipelines. The ideal candidate will develop and optimize data transformations, build production-grade Python components, and maintain cloud-native Azure environments while collaborating with application teams and ensuring high-quality, reliable data delivery. This role offers the opportunity to work with large-scale datasets, implement ETL/ELT best practices, optimize Databricks clusters, and leverage modern cloud technologies to support AI/ML initiatives.
Responsibilities:
- Design, build, and support end-to-end data pipelines, including ingestion, transformation, validation, and publishing.
- Develop and optimize SQL and PySpark/Databricks transformations for large datasets.
- Build production-grade Python modules with logging, error handling, testing, and integration with APIs/files.
- Create, maintain, and operate Azure Data Factory (ADF) pipelines, including triggers, parameterization, monitoring, and failure handling.
- Work within Azure environments: ADLS Gen2 (Blob Storage), Azure SQL, Azure App Service, and resource groups.
- Provision and maintain Azure components using Pulumi (Infrastructure as Code).
- Optimize Databricks clusters, workflows, and jobs for performance and reliability.
- Participate in code reviews, documentation, and operational support, including triage and root cause analysis.
- Collaborate with application teams for integration, troubleshooting, and operational coordination.
Qualifications:
- Education: Bachelors degree in Computer Science, Engineering, or a related technical field (or equivalent experience).
- Experience: 7+ years as a Data Engineer; 3+ years in ETL/ELT concepts, PySpark, and SQL.
- SQL: Advanced querying, CTEs, views, joins, complex transformations, and performance tuning.
- Python: 2+ years building production-quality modules, unit/integration testing, logging, and CI/CD integration.
- Databricks: 1+ year working with notebooks, jobs, workflows, external/managed tables, Delta Lake, and basic cluster configuration.
- Azure Data Factory (ADF): 1+ year creating and maintaining pipelines, including triggers, parameterization, monitoring, and error handling.
- Azure Cloud: Hands-on with ADLS Gen2, Azure SQL, Azure App Service, and general Azure portal/resource group operations.
- Infrastructure as Code: Experience provisioning Azure resources with Pulumi.
- ETL/ELT Concepts: Strong understanding of pipeline patterns, incremental loads, data validation, and troubleshooting.
Preferred Skills:
- Additional Skills (nice-to-have): R for data validation, TypeScript for Pulumi pipelines, Java/.NET for integration, Angular/Spring Boot for minor troubleshooting.
Employment Details:
- Location: Chicago, IL
- Work Schedule: Hybrid Schedule
- Duration: 6+ Month Contract
- Employment Type: Contract
- Client: NetworkPedia, Inc (Direct Client || Prime Vendor || Implementation Partner)
- Preference: ONLY LOCALS
Interview Process:
- F2F INTERVIEW REQUIRED
Special Requirements
ONLY LOCALS. F2F INTERVIEW REQUIRED. Hybrid Schedule.
Compensation & Location
Salary: $60 – $80 per year
Location: Chicago, IL
Recruiter / Company – Contact Information
Recruiter / Employer: NetworkPedia, Inc
Email: araj@networkpedia.com
Recruiter Notice:
To remove this job posting, please send an email from
araj@networkpedia.com with the subject:
DELETE_JOB_ID_6116