Building an Automated AI QA Agent to Test Jira Tickets in Real Browsers

Manual end-to-end testing is notoriously time-consuming. Writing Cypress or Selenium scripts for every new user story often creates an unmaintainable codebase of fragile selectors and flaky assertions. Autonomous AI agents present a modern alternative: by combining visual inspection, task reasoning, and browser orchestration, an agent can read acceptance criteria directly from a Jira ticket, navigate your Web application in a real browser, execute the test steps, and evaluate success or failure on its own.

In this step-by-step tutorial, we will construct an end-to-end AI QA testing suite using two open-source frameworks:

1. Agent Factory: A visual node-based studio for editing, prompt engineering, orchestrating, and inspecting LLM-powered multi-agent workflows.
2. BaaS (Browser as a Service): A Go-based browser automation engine that exposes control APIs to manage headful or headless Chrome instances, take DOM snapshots, execute navigation scripts, and record screenshots.

By leveraging high-speed, reliable LLM endpoints provided by unified API aggregators like n1n.ai, we ensure our vision and text reasoning steps execute with minimal latency.

System Architecture Overview

The autonomous QA pipeline operates as a multi-stage directed acyclic graph (DAG). Below is the logical architecture of how Jira, Agent Factory, BaaS, and the LLM backbone interact:

+------------------+         +-------------------------------+         +-----------------------+
|   Jira Ticket    | ------> |  Agent Factory Orchestrator   | ------> |   BaaS Chrome Engine  |
| (SCRUM-165 Criteria)|      |  (Workflow & Prompt Nodes)    |         | (Headful Browser Session)|
+------------------+         +-------------------------------+         +-----------------------+
                                             |                                    |
                                             v                                    v
                                 +-----------------------+              +-------------------+
                                 | Unified LLM API       |              | DOM & Screenshots |
                                 | (n1n.ai Endpoint)     |              | Verification      |
                                 +-----------------------+              +-------------------+

Core Multi-Agent Task Pipeline

1. Jira Fetcher Node: Retrieves story details via Jira REST API using Base64 HTTP Basic Authentication.
2. Step Synthesis Node (llm-generate-steps): Converts story text into clear, sequential QA manual execution steps.
3. UI Layout Adaptor Node (llm-0): Analyzes the latest page screenshot and maps logical step instructions to visual targets on screen.
4. BaaS Script Generator Node (llm-1): Converts natural language actions into BaaS execution primitives (navigate(), click(), type(), waitReady()).
5. Visual Assertion Evaluator Node (llm-2): Compares rendered state screenshots against expected acceptance criteria to issue a PASS, FAIL, or BROKEN verdict.

Comparison: Traditional E2E vs. AI-Driven Browser Testing

Prerequisites & Requirements

Before starting the deployment, ensure your local development environment meets the following specifications:

• Required Capabilities:
  • A valid GitHub account to clone repository modules.
  • An API key for an LLM provider (OpenAI, Anthropic, or unified access via n1n.ai).
  • Google Chrome installed locally.
• Recommended Tools:
  • Docker and Docker Compose ( simplifies database and frontend runtime hosting).
  • Go 1.20+ installed locally (required for running the BaaS engine in headful debug mode).
  • An Atlassian Account with Jira API Token generation privileges.

Step 1: Deploying Agent Factory Studio

First, we will pull down the Agent Factory repository and launch the containerized database and backend UI framework.

# Clone the unified agent factory repository
git clone https://github.com/Ursa-Minor-Beta/agent-factory-docker-api-ui.git
cd agent-factory-docker-api-ui

# Create local environment configuration
cp .env.example .env

# Edit .env to set your admin credentials
# Ensure you configure ADMIN_EMAIL and ADMIN_PASSWORD
nano .env

# Launch services with PostgreSQL/MongoDB persistence profile
docker-compose --profile with-db up --build -d

Once the containers build and launch successfully, verify that the studio UI is reachable by opening http://localhost:8080 in your web browser.

Step 2: Setting Up the BaaS (Browser as a Service) Engine

To observe the agent interacting with your target app in real-time, run BaaS directly on your host system in headful mode.

# Navigate to the BaaS engine directory
cd baas

# Start MongoDB dependency in background
docker compose up -d mongodb

# Copy environment configuration template
cp .env.example .env

Open baas/.env and update the parameters as follows:

API_KEY=your_secure_baas_secret_key_here

# Point to your local Google Chrome executable path
# macOS Example:
BROWSER_EXECUTABLE='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
# Linux Example: BROWSER_EXECUTABLE='/usr/bin/google-chrome'
# Windows Example: BROWSER_EXECUTABLE='C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe'

BROWSER_HEADFUL=true

LLM_CLIENT=openai
OPENAI_TOKEN="your-llm-api-token"
OPENAI_ORGANIZATION=