NEWPosted 2 hours ago
Job ID: JOB_ID_10732
Job Overview:
We are seeking a skilled Data Engineer to join our datastore-migration Factory team. This is a high visibility and crucial project for Goldman Sachs, focused on the end-to-end datastore migration from on-prem DataLake to an AWS hosted LakeHouse. You will play a key role in ensuring a smooth and accurate transition of data assets.
Responsibilities:
- Pipeline Migration: Refactor and migrate extraction logic and job scheduling from legacy frameworks to the new Lakehouse environment. Execute the physical migration of underlying datasets while ensuring data integrity. Act as a technical liaison to internal clients, facilitating ‘hand-off and sign-off’ conversations with data owners to ensure migrated assets meet business requirements.
- Consumption Pattern Migration: Translate and optimize legacy SQL and Spark-based consumption patterns (raw and modeled) for compatibility with Snowflake and Iceberg. Analyze usage patterns to deliver required data products. Act as a technical liaison to internal clients, facilitating ‘hand-off and sign-off’ conversations with data owners to ensure migrated assets meet business requirements.
- Data Reconciliation & Quality: Implement a rigorous approach to data validation. Work with reconciliation frameworks to build confidence that migrated data is functionally equivalent to that already used within production flows.
- Collaboration: Work with internal data management platforms team and demonstrate an aptitude for learning new workflows and language constructs as necessary.
Technical Skills:
- Basic Qualifications:
- Bachelors or Masters in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.
- Minimum of 3-5 years of professional ‘hands-on-keyboard’ coding experience in a collaborative, team-based environment. Ability to troubleshoot (SQL) and basic scripting experience.
- Professional proficiency in Python or Java.
- Deep familiarity with the full Software Development Life Cycle (SDLC) and CI/CD best practices & K8s deployment experience.
- Core Data Engineering Competencies:
- Temporal Data Modeling: Managing state changes over time (e.g., SCD Type 2).
- Schema Management: Expertise in Schema Evolution (Ref: Iceberg Apache) and enforcement strategies.
- Performance Optimization: Advanced knowledge of data partitioning and clustering.
- Architectural Theory: Balancing Normalization vs. Denormalization and the strategic use of Natural vs. Surrogate Keys.
- Technical Stack Requirements:
- Extraction & Logic: Kafka, ANSI SQL, FTP, Apache Spark
- Data Formats: JSON, Avro, Parquet
- Platforms: Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ
Core Competencies:
- Demonstrates strong integrity and consistently models good conduct and ethical decision-making.
- Acts as a trusted team player who collaborates effectively across multiple teams and functions.
- Communicates with clarity and confidence – concise written updates, structured verbal briefings, and proactive stakeholder management.
- Works effectively with global teams across time zones and cultures; builds alignment and resolves issues constructively.
- Delivery-focused with a strong sense of ownership; drives work to closure and meets commitments.
- Brings high energy and urgency to achieve targets while maintaining quality and professionalism.
- Shows intellectual curiosity; asks thoughtful questions, surfaces risks early, and seeks feedback to continuously improve.
Special Requirements
Onsite from Dallas, TX (5 Days Onsite)
Compensation & Location
Salary: $100,000 – $150,000 per year
Location: Dallas, TX
Recruiter / Company – Contact Information
Email: adilmohd9084@gmail.com
Recruiter Notice:
To remove this job posting, please send an email from
adilmohd9084@gmail.com with the subject:
DELETE_JOB_ID_10732