Aipplify
JobsCompaniesBlogFor Recruiters
Sign InCreate Account
Ruby Labs

AI Engineer

6.0/10

Ruby Labs

Not specified
Remote
senior
10 days ago
aitechNode.jsNext.jsTypeScriptLangChainLlamaIndexLangfuseOpenRouter

AI Summary

The vacancy is well-defined but lacks compensation details, which may deter applicants.

Check Match — Just drop your CV

See your fit for AI Engineer in seconds.

Description

Key Responsibilities

  • •Advanced Prompt Engineering: Designing complex, dynamic prompt templates with conditional logic and efficiently reusing information and context within prompts to maximize generation quality and reasoning.
  • •Structured Outputs & Schemas: Implementing various response schemes (JSON mode, function calling, Zod/JSON schemas) to ensure AI outputs are predictable and ready for seamless integration into application logic.
  • •Prompt Engineering & Evaluations: Building robust evaluation pipelines and using Langfuse to collect feedback and score the quality of responses in real time.
  • •Tracing & Debugging: Performing deep debugging of complex LLM chains using Langfuse traces to identify bottlenecks and optimize for cost, latency, and context window usage.
  • •AI A/B Testing: Running systematic experiments across different models via OpenRouter (e.g., comparing Claude 3.5 Sonnet vs. GPT-4o) and analyzing results based on quantitative metrics.
  • •Data-Driven Decisions: Making deployment decisions for new prompts or models strictly based on quantitative benchmarks and trace data, rather than intuition.
  • •Output Scoring & Analysis: Developing scoring systems to analyze the “Problem → Solution” chain and identify root causes of hallucinations or logic errors using Langfuse analytics.
  • •Model Performance & Fine-Tuning: Regularly re-evaluating model performance as new architectures emerge and performing fine-tuning when necessary to meet specific domain requirements.

Requirements

Qualifications

  • •Node.js & Next.js: Deep knowledge of the stack to build reliable services and handle complex LLM-generated data.
  • •Dynamic Prompting Skills: Proven experience in building prompts where content is highly dependent on input variables and context injection.
  • •OpenRouter Experience: Experience working with unified APIs, managing rate limits, and selecting the most cost-effective models for specific tasks.
  • •Langfuse (or similar): Understanding of LLM observability principles — setting up tracing, creating test datasets, and integrating scoring systems.
  • •Evaluation Methodology: Experience with frameworks like RAGAS or building custom “LLM-as-a-judge” systems.
  • •Analytical Mindset: Ability to transform raw generation logs into actionable business metrics and technical insights.
  • •Iterative Mindset: Focus on continuous product improvement through constant feedback loops.
  • •Fluency in Russian and/or Ukrainian.
Loading similar jobs...