QA Automation Engineer (TypeScript)

Summary

Andersen is hiring a QA Automation Engineer (TypeScript) for a project developing AI-powered platforms and ensuring quality of intelligent automation solutions.

The customer is a technology-focused company specializing in AI-driven solutions and digital innovation for businesses across multiple industries. The organization develops intelligent platforms and automation tools designed to improve operational efficiency, streamline workflows, and support data-driven decision-making. The company operates in an international environment with a strong focus on scalable architecture, modern engineering practices, and the practical implementation of advanced AI technologies.

The project is focused on developing a platform for creating AI agents without technical expertise. It enables businesses to deploy plug-and-play AI assistants that handle communication, reporting, analysis, and collaboration tasks across channels such as email, phone, etc.

Responsibilities

• Designing, developing, and maintaining automated testing frameworks for AI/LLM-powered SaaS platforms and agentic systems.
• Validating end-to-end autonomous workflows, including prompt orchestration, tool calling, memory and context handling, RAG pipelines, and human-in-the-loop processes.
• Building automated evaluation pipelines to assess reasoning quality, hallucinations, accuracy, consistency, and response safety across non-deterministic AI systems.
• Creating and maintaining regression suites covering long-running workflows, retry and fallback logic, conversation state persistence, and failure recovery scenarios.
• Performing API, async pipeline, webhook, queue, and event-driven system testing across distributed architectures.
• Investigating, reproducing, and debugging issues across frontend, backend, infrastructure, and LLM layers.
• Collaborating closely with AI engineers, backend developers, and product teams to improve quality, reliability, observability, and scalability of AI-driven features.
• Contributing to CI/CD automation, tracing and monitoring solutions, synthetic and golden dataset generation, response scoring, and token/cost analysis for production AI workflows.

Requirements

• Strong experience in QA Automation for AI/LLM-powered SaaS platforms and agentic systems for 3+ years.
• Hands-on experience with automation frameworks such as Playwright, Cypress, Selenium, or similar tools.
• Solid understanding of autonomous and multi-agent workflow validation, including prompt orchestration, tool calling, memory/context handling, RAG systems, and human-in-the-loop processes.
• Experience testing non-deterministic AI behavior and building reliable validation strategies for probabilistic outputs.
• Ability to design and maintain LLM evaluation frameworks focused on reasoning quality, hallucination detection, accuracy, consistency, and response safety.
• Strong experience with API testing, async pipelines, webhooks, queues, and event-driven architecture.
• Experience validating long-running workflows, retry/fallback logic, conversation state persistence, and failure recovery scenarios.
• Strong debugging and troubleshooting skills across frontend, backend, infrastructure, and LLM layers.
• Experience with CI/CD pipelines, automated regression testing, tracing, observability, and log analysis.
• Strong understanding of how testing agentic AI systems differ from traditional QA approaches and the ability to build scalable automated evaluation systems for AI agents.
• Level of English – from Intermediate+ and above.

Desired skills

• Experience with synthetic test generation, golden datasets, response scoring, and token/cost monitoring.
• Familiarity with OpenAI or Anthropic APIs, LangChain, LangGraph, CrewAI, AutoGen, vector databases, and RAG architectures.
• Knowledge of AI safety, guardrail validation, performance/load testing, and cloud or Kubernetes-based infrastructure.

Reasons to join us

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc..
• The opportunity to change the project and/or develop expertise in an interesting business domain.
• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
• The opportunity to earn up to an additional 1,000 USD per month, depending on the level of expertise, which will be included in the annual bonus, by participating in the company's activities.
• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
• Certification compensation (AWS, PMP, etc).
• Referral program.
• English courses.
• Private health insurance and compensation for sports activities.

Join us!

Lokalizacje

Worldwide