RAG Knowledge Base

Policies, Code, and Documents Managed Separately Across Teams

“Where was that spec?” “Which commit changed this bug?” Developers and POs kept looking for the same context throughout the day.

Requirements and discussions lived in Jira, background and policy in Confluence, and implementation and change history in Git. After searching each tool, someone still had to compare the policy, decisions made in the ticket, and code. Using AI development tools also required a person to find, copy, and paste the relevant material.

QA faced the same problem. E2E reduced repeated checks of core behavior and manual work in Google Sheets, but new and changed features still required reading requirements, tracing their effects, and writing test cases. Accounting for earlier bugs and exceptions meant returning to the three sources, with coverage affected by what the reviewer remembered.

I built an internal RAG platform to reduce repeated searching and comparison across these sources. I applied it first to QA, then extended it so developers could check relevant policies and code before implementation and planners could review missing conditions.

One RAG Platform for Distributed Policies, Code, and Documents

Building on the Jira, Confluence, and Git connectors, I collected and indexed tickets, policies, code, commit history, and accumulated QA records. One question can retrieve related sources together so the reason behind a policy change can be compared with its implementation.

I implemented the server in Go and collection, indexing, and model services in Python. Search returns source links and code locations, while the client constructs test cases and answers. This reduces repeated copying and investigation without losing the ability to check original sources.

Plain Vector Search Where Similar Documents Push the Right Answer Down

Collecting sources in one place would not remove manual investigation unless search found the right material. Semantic similarity alone could return plausible documents without sufficiently distinguishing the required code or exception.

Policy questions need relevant material even when the wording differs, while code and error searches must retain the terms actually used. To support both, I generated dense and sparse vectors together with BGE-M3 and combined semantic and lexical retrieval. Because the two searches use different score scales, I used RRF in Qdrant to combine their rankings rather than simply adding raw scores.

Combining search results alone does not guarantee that documents containing the required conditions or exceptions will rank first. I separated candidate retrieval, which aims to reduce omissions, from a cross-encoder stage that reads each candidate together with the question and reassesses relevance. Applying this more detailed comparison only to the retrieved candidates, rather than every document, bounds the additional computation.

The search context also mattered. Similar code from another repository or an in-development branch could be mistaken for evidence of current behavior, so I applied repository, path, and branch filters during candidate retrieval. When the target was already known, such as a ticket number or function name, semantic similarity was unnecessary. I routed these lookups through a separate keyword-search path without embedding or reranking.

More relevant passages were not enough if they all came from one document and crowded out related policy or code. I limited chunks per document and reserved source-specific candidates so cross-source questions retained the material needed for comparison.

How this meaning-plus-terms approach actually works and looks shows up below — in the retrieval flow and in the vector distribution of the production index. In production, though, dense and sparse are searched separately and their ranks fused by RRF, so hybrid is not really a single vector. For the figure below I approximate it by concatenating the L2-normalized dense vector with an L2-normalized truncated-SVD of the sparse vector.

E2E for Baseline Checks, RAG for New and Changed Features

I connected these retrieved sources to QA preparation, reviewing requirements, earlier bugs, code, and verification records before defining role, state, boundary, exception, and regression cases. I then execute the browser workflow and cross-check saved results and states through APIs. Re-verification starts from earlier reproduction conditions and the intent of the fix.

E2E repeatedly checks core behavior, while RAG supports test-case design and verification for new and changed features. I make the final call on new policies and exceptions, verifying the actual browser behavior and persisted results.

With E2E, test cases, RAG-assisted verification, and Jira follow-up working together, I now own QA alone in about three days on average, compared with three POs spending one to two weeks per release initially. RAG contributes by supporting the investigation and verification of new and changed features.

The automated execution environment and test workflows are covered in E2E QA Automation.

QA Knowledge, Extended into Development

I exposed RAG through MCP so developers could review relevant policies and past bugs before implementation. Environments such as Claude and Codex can query dependencies, change history, policies, and existing consumers together to check implementation plans and likely defects.

The development workflow starts with inspecting current code and proposing a change. RAG then checks the plan against past regressions, related requirements, and consumers in other repositories. It returns up to three sources that could change the plan and five priority checks. If no new information is found, work returns to editing, compilation, and tests.

Up to +6.7% Higher Implementation Accuracy with RAG

I measured whether the knowledge used in QA also lifted the accuracy of what developers built. I reconstructed shipped backend tickets using the pre-change code and the knowledge available at that point, then had Opus and Sonnet each produce outputs in a baseline environment and one combining RAG, development instructions, and code-structure tools. Three fixed LLM evaluators scored the outputs against the same functional criteria.

With RAG, Opus rose from 79.5 to 84.8 and Sonnet from 78.2 to 82.6 — gains of 6.67% and 5.63%.

The trade-off was output-token usage: approximately 1.22× baseline for Opus and 1.16× for Sonnet. I therefore prioritized evidence that could change an implementation plan and avoided repeatedly investigating facts already established in the current code.

Policy Gaps and Conflicts Caught Before Code

A developer can follow the requirements and still produce a defect if those requirements omit a condition or contradict each other. The next extension was to review the planning ticket before implementation.

A planner requests a review with #qa-check in a Jira comment and receives the result in the same ticket. The worker collects comments, field history, and directly related tickets so later decisions are not missed. Linked PRs are inspected through synchronized PR head snapshots. Missing code or an unestablished base branch remains an explicit unknown.

Code is evidence for the review, but the output is written for the PO. It prioritizes missing rules and decisions, rather than telling a non-engineer which function to edit. Findings include priorities and source links.

The review model has read-only tools; only the worker can post Jira comments. Results retain the trigger comment’s visibility. SQLite persists request IDs and progress so retries and restarts do not post the same result repeatedly.

Actual Deployment State That Retrieved Code Alone Cannot Reveal

Using the same knowledge in QA, development, and planning also raised a practical question: is this code deployed to the test environment? Finding code or a commit cannot answer that. I connected dedicated read-only tools to check deployment versions and database change history separately.

Indexed sources also need a visible update history. I separated retrieval from periodic indexing, retained stable document IDs for incremental synchronization, and exposed source freshness. Baseline and unmerged development branches stay distinct so older material and work in progress are not mistaken for the current state.

A shared lock coordinates embedding and reranking on a single GPU. Reranker failures fall back to another model or the fused retrieval order. The Jira worker polls independently of review execution and caps failed jobs at five total attempts. Prometheus and Grafana track retrieval latency, errors, worker progress, stalled activity, and exhausted retries.

Post-deploy Bugs Down from 7.0% to 3.0% with E2E + RAG

Today the platform is genuinely used in QA preparation, development, and planning-policy review. It cuts the repeated searching across Jira, Confluence, and Git, surfacing the background of new or changed features in one place as a basis for verification and implementation.

Since adopting RAG, more bugs are caught earlier. Checking the relevant policy, past bugs, and consumers before implementing reduces easy-to-miss regressions and exceptions, and in QA the background of new and changed features is cross-checked to widen coverage.

E2E automation lifted average QA volume, and adding RAG brought the post-deploy bug rate down another step. Comparing regular releases before E2E, after E2E, and after RAG on the same basis: