Job ID: JOB_ID_993
Role Overview
The Lead AI Engineer will be at the forefront of a major technological shift within our organization. We are moving away from traditional, rigid regex and keyword-based search systems toward a dynamic, context-aware, and highly intelligent search experience. This role is not just about maintenance; it is about architectural transformation. You will be responsible for integrating state-of-the-art semantic search, dense retrieval, and Large Language Model (LLM) powered ranking techniques into our existing ElasticSearch infrastructure. The ideal candidate is a visionary engineer who understands the nuances of information retrieval and can bridge the gap between cutting-edge AI research and production-grade software engineering.
Key Responsibilities
- Modernizing the Search Platform: You will analyze the current limitations of our keyword-only search implementation. By applying BM25 tuning, custom tokenizers, and sophisticated boosting strategies, you will significantly improve search relevance.
- LLM-Driven Search & RAG Integration: You will implement advanced workflows including query expansion and embedding generation using industry-leading models from OpenAI, Cohere, or Sentence Transformers. A major part of the role involves building Retrieval Augmented Generation (RAG) flows to provide users with precise, context-rich answers.
- Search Infrastructure Engineering: Beyond the models, you will build the APIs that power these experiences. This includes designing scalable pipelines for indexing both structured and unstructured data, maintaining high-performance embedding stores, and ensuring real-time updates.
- AWS Cloud Delivery: Leveraging the full power of AWS is essential. You will manage OpenSearch services, utilize SageMaker for model deployment, and orchestrate services using Lambda, ECS/EKS, and API Gateway.
- Evaluation and Metrics: You will establish a data-driven culture by developing search evaluation metrics such as nDCG, MRR, and precision@k. Continuous improvement through A/B testing will be a core part of your workflow.
- Strategic Leadership: Partner with product teams to refine search behaviors based on real usage patterns and business goals. You will lead the technical roadmap for search innovation.
Technical Environment
The successful candidate will work extensively with Python and the AWS ecosystem. You will be expected to containerize applications using Docker and deploy them via robust CI/CD pipelines. The role requires a deep understanding of vector databases, approximate nearest neighbor (ANN) techniques, and the ability to tune ranking functions for hybrid search scoring. You will also explore cross-encoder and bi-encoder architectures to further refine re-ranking processes. This is a high-impact role that requires both deep technical expertise and the ability to communicate complex AI concepts to stakeholders across the organization. We are looking for someone who is passionate about the intersection of search and artificial intelligence and who can drive our platform into the next generation of intelligent discovery.
Special Requirements
Hybrid work model: 3 days onsite in Austin, TX. Initial 4-week discovery phase requires 5 days onsite. Domain: AI/Search Engineering.
Compensation & Location
Salary: $185,000 – $245,000 per year (Estimated)
Location: Austin, TX
Recruiter / Company – Contact Information
Recruiter / Employer: NorthITE
Email: tej@northite.com
Recruiter Notice:
To remove this job posting, please send an email from
tej@northite.com with the subject:
DELETE_JOB_ID_993