NEWPosted 10 hours ago

Job ID: JOB_ID_8098

Role:

As a Senior Data Engineer, you will play a hands-on role in designing, building, and operating high-performance batch and streaming data platforms. You will:

  • Design, develop, and maintain large-scale batch and streaming pipelines using PySpark and Python.
  • Build real-time and near real-time streaming applications with stateful processing, windowing, and checkpointing.
  • Develop production-grade Python microservices for complex data transformations and business logic.
  • Design and manage modern data lake architectures using Apache Iceberg on AWS S3, implementing schema evolution, partitioning, compaction, and time travel.
  • Develop and deploy pipelines across AWS services including S3, EMR, Glue, Lambda, Athena, Redshift, and Aurora.
  • Optimize Spark workloads for performance, scalability, and cost efficiency.
  • Implement monitoring, logging, alerting, and recovery mechanisms for robust production operations.
  • Contribute to CI/CD pipelines, participate in architecture discussions, and uphold engineering best practices.

Qualifications:

  • Bachelors or Masters degree in Computer Science, Engineering, or a related discipline.
  • Over 10+ years of experience in IT and strong hands-on expertise in PySpark, Spark SQL, and distributed data processing.
  • Advanced proficiency in Python for building scalable, production-grade data solutions and microservices.
  • Proven experience building and running Kafka-based streaming applications in production environments.
  • Deep understanding of streaming fundamentals, including stateful processing and fault tolerance.
  • Hands-on experience with Apache Iceberg in production data lake environments.
  • Solid experience with AWS data services (S3, EMR, Glue, Lambda, Redshift, Aurora).
  • Advanced SQL skills and strong knowledge of data modeling and modern data lake architectures.
  • Strong troubleshooting skills in distributed data systems with a focus on reliability and performance.

Mandatory:

  • Experience with Tier 1 and 2 Companies
  • Skillset: Python, Spark, Microservices and AWS

Special Requirements

Onsite-Locals only. Experience with Tier 1 and 2 Companies.

Compensation & Location

Salary: $130,000 – $170,000 per year (Estimated)

Location: Reading, PA

Recruiter / Company – Contact Information

Email: praveen.palla@3sbc.com

Interested in this position?
Apply via Email

Recruiter Notice:
To remove this job posting, please send an email from
praveen.palla@3sbc.com with the subject:

DELETE_JOB_ID_8098

to delete@join-this.com.