RAG Knowledge Base
Summary
I built a RAG platform to reduce the repeated searching and cross-checking of knowledge scattered across Jira, Confluence, Git, and other tools. It supports the test-case design and verification of new and changed features that remained after E2E testing, and extends further into development assistance and policy verification at the planning stage.
- Role
- InitiatorDeveloper (POQA)
- Outcome
- Implementation correctness up to +6.7%post-deploy bugs 3.9% → 2.9%
Policies, Code, and Documents Managed Separately Across Teams
"Where was that spec?" "Which commit changed this bug?" Developers and POs kept looking for the same context throughout the day.
Requirements and discussions lived in Jira, background and policy in Confluence, and implementation and change history in Git. After searching each tool, someone still had to compare the policy, decisions made in the ticket, and code. Using AI development tools also required a person to find, copy, and paste the relevant material.
QA faced the same problem. E2E reduced repeated checks of core behavior and manual work in Google Sheets, but new and changed features still required reading requirements, tracing their effects, and writing test cases. Accounting for earlier bugs and exceptions meant returning to the three sources, with coverage affected by what the reviewer remembered.
I built an internal RAG platform to reduce repeated searching and comparison across these sources. I applied it first to QA, then extended it so developers could check relevant policies and code before implementation and planners could review missing conditions.
One RAG Platform for Distributed Policies, Code, and Documents
Building on the Jira, Confluence, and Git connectors, I collected and indexed tickets, policies, code, commit history, and accumulated QA records. One question can retrieve related sources together so the reason behind a policy change can be compared with its implementation.
I implemented the server in Go and collection, indexing, and model services in Python. Search returns source links and code locations, while the client constructs test cases and answers. This reduces repeated copying and investigation without losing the ability to check original sources.
From Dense Retrieval to Hybrid Search
I started with dense retrieval: embedding questions and documents, then finding passages with similar meaning. The aim was to let people find material without knowing its exact terminology—for example, finding the temporary-suspension scheduling policy by asking how to pause waste collection.
A document on the same topic, however, is not necessarily an answer. Consider a question about the conditions for transferring work quantities during contract renewal. A general guide to copying contracts is not enough; the relevant transfer conditions and restrictions are needed. Ranking by semantic similarity alone can put other renewal documents first and leave the required passage outside the top-k candidates. That distinction matters when the results are used to prepare QA.
I kept dense retrieval and added sparse retrieval to compare terms actually present in the question and documents, such as "work quantity" and "transfer." BGE-M3 generates both representations; Qdrant searches them separately and combines their rankings with RRF. Rather than adding scores with different scales and distributions, RRF uses the position of each result in the two rankings. Dense retrieval helps with different wording, while sparse retrieval adds a route for material sharing the important terms.
Matching terms still does not establish whether a passage covers the required conditions. A cross-encoder therefore compares each candidate with the question and reassesses its relevance. Reranking cannot recover a document that never reached the candidate set, so the two searches first help reduce omissions, then reranking refines the order. Restricting this more expensive comparison to retrieved candidates bounds the additional computation.
The search context also mattered. Similar code from another repository or an in-development branch could be mistaken for evidence of current behavior, so I applied repository, path, and branch filters during candidate retrieval. When the target was already known, such as a ticket number or function name, semantic similarity was unnecessary. I routed these lookups through a separate keyword-search path without embedding or reranking.
More relevant passages were not enough if they all came from one document and crowded out related policy or code. I limited chunks per document and reserved source-specific candidates so cross-source questions retained the material needed for comparison.
The 3D distribution below compares how dense and sparse vectors represent the same material. The hybrid view concatenates normalized dense vectors with normalized, dimension-reduced sparse vectors for visualization only. Production retrieval combines rankings through RRF; distances and positions in the plot are not measurements of retrieval accuracy.
E2E for Baseline Checks, RAG for New and Changed Features
I connected these retrieved sources to QA preparation, reviewing requirements, earlier bugs, code, and verification records before defining role, state, boundary, exception, and regression cases. I then execute the browser workflow and cross-check saved results and states through APIs. Re-verification starts from earlier reproduction conditions and the intent of the fix.
E2E repeatedly checks core behavior, while RAG supports test-case design and verification for new and changed features. I make the final call on new policies and exceptions, verifying the actual browser behavior and persisted results.
With E2E, test cases, RAG-assisted verification, and Jira follow-up working together, I now own QA alone in about three days on average, compared with three POs spending one to two weeks per release initially. RAG contributes by supporting the investigation and verification of new and changed features.
The automated execution environment and test workflows are covered in E2E QA Automation.
QA Knowledge, Extended into Development
I exposed RAG through MCP so developers could review relevant policies and past bugs before implementation. Environments such as Claude and Codex can query dependencies, change history, policies, and existing consumers together to check implementation plans and likely defects.
The development workflow starts with inspecting current code and proposing a change. RAG then checks the plan against past regressions, related requirements, and consumers in other repositories. It returns up to three sources that could change the plan and five priority checks. If no new information is found, work returns to editing, compilation, and tests.
Up to +6.7% Higher Implementation Accuracy with RAG
I measured how much using RAG improved the quality of development outputs. I reconstructed shipped backend tickets using the pre-change code and the knowledge available at the time, then had Claude Opus and Sonnet each implement the same tickets in isolated environments with and without RAG. Three independent agents evaluated the outputs and assigned scores against five criteria: whether they met the tickets’ core requirements, covered related features and exception-handling paths, prevented previously observed follow-up bugs, introduced unnecessary changes, or created regression risks through overengineering.
Opus
Sonnet
With RAG, Opus rose from 79.5 to 84.8 and Sonnet from 78.2 to 82.6 — gains of 6.67% and 5.63%.
The trade-off was output-token usage: approximately 1.22× baseline for Opus and 1.16× for Sonnet. I therefore prioritized evidence that could change an implementation plan and avoided repeatedly investigating facts already established in the current code.
Policy Gaps and Conflicts Caught Before Code
A developer can follow the requirements and still produce a defect if those requirements omit a condition or contradict each other. The next extension was to review the planning ticket before implementation.
A planner requests a review with #qa-check in a Jira comment and receives the result in the same ticket. The worker collects comments, field history, and directly related tickets so later decisions are not missed. Linked PRs are inspected through synchronized PR head snapshots. Missing code or an unestablished base branch remains an explicit unknown.
Code is evidence for the review, but the output is written for the PO. It prioritizes missing rules and decisions, rather than telling a non-engineer which function to edit. Findings include priorities and source links.
The review model has read-only tools; only the worker can post Jira comments. Results retain the trigger comment’s visibility. SQLite persists request IDs and progress so retries and restarts do not post the same result repeatedly.
Actual Deployment State That Retrieved Code Alone Cannot Reveal
Using the same knowledge in QA, development, and planning also raised a practical question: is this code deployed to the test environment? Finding code or a commit cannot answer that. I connected dedicated read-only tools to check deployment versions and database change history separately.
Indexed sources also need a visible update history. I separated retrieval from periodic indexing, retained stable document IDs for incremental synchronization, and exposed source freshness. Baseline and unmerged development branches stay distinct so older material and work in progress are not mistaken for the current state.
Making Slow Searches and Stalled Reviews Visible
As the platform became part of daily QA and development, I needed to investigate slow responses and reviews that did not return a result. I collected metrics from the Go retrieval server and Jira worker with Prometheus and built separate Grafana dashboards for retrieval and policy QA.
The retrieval dashboard pairs requests by tool with p50 and p95 latency so slow periods can be compared with RSS, Go heap, goroutines, and open file descriptors. Qdrant and container CPU and memory, along with GPU usage, help relate retrieval behavior to resource use. Separate repository indexing checkpoints and branch update indicators reveal stale sources even when the service itself is healthy.
A shared lock coordinates embedding and reranking on a single GPU. Reranker failures fall back to another model or the fused retrieval order.
A running Jira worker does not necessarily mean reviews are progressing. I separated the last successful poll, active reviews, job heartbeats, and Claude response activity, and defined Prometheus rules for stale polling, stalled jobs, and exhausted retries. This distinguishes a failure to collect requests from a review that stops after a request has been accepted.
The Jira worker polls independently of review execution and caps failed jobs at five total attempts.
An Operations Runbook: Targets and Response per Signal
I wrote an operations runbook defining which metrics to check, what counts as abnormal, and how to respond when searches slow down or errors occur.
I defined retrieval latency, error rate, index update status, and Jira worker health as key service-level indicators (SLIs), and documented what to check and in what order when a problem occurs. Thresholds were based on observed values, actual timeout settings, polling intervals, and retry counts.
| Metric (SLI) | Checks and Criteria | Response Steps |
|---|---|---|
| Retrieval latency | p95 latency; reranker timeout: 30s; HTTP timeout: 60s | Check RSS, Go heap, goroutines, and GPU contention → verify fallback to fused retrieval rankings if reranking fails |
| Retrieval errors | Errors by tool (rag_errors_total); rag_backend_up == 0 for 5m | Identify the tool reporting errors → check embedder and Qdrant health → inspect error logs and fallback paths |
| Index update status | Event processing lease: 300s; reconciliation: every 1m; branch sync: every 30m; index state: error/pending | Check per-source indexing checkpoints and branch updates → inspect jobs moved to dead-letter after five failed attempts |
| Jira ticket polling | More than 3 min since the last successful ticket poll (critical) | Check worker health and the last successful poll time → restart the worker |
| Jira review status | No heartbeat update for more than 1 min (critical); no Claude response for more than 10 min (warning) | Check the review stage and job lease (45m) → retry the stalled review |
| Retry limit reached | Exhausted after 5 failed attempts; retry delay starts at 1m and increases to a maximum of 1h | Investigate the failure on the exhausted ticket → manually request another review |
Post-deploy Bugs Down from 7.0% to 2.9% with E2E + RAG
RAG is now used in QA, development, and policy review during planning. It has reduced repeated searches across Jira, Confluence, and Git, allowing us to review the context of new and changed features together when deciding QA coverage and implementation direction.
During development, developers checked relevant policies, past bugs, and existing usages to review easily missed regressions and edge cases. In QA, I compared requirements with changed code to define coverage, and extended the platform to check for missing or conflicting policies during planning.
In the regular releases after RAG was applied (2.16.2–2.22.0), 23 post-deploy bugs were found among 794 verified tickets. The period with E2E testing alone had 39 among 992.
With E2E, RAG-assisted verification, and Jira follow-up working together, average QA volume rose from 22.3 to 33.1 tickets per day, while the post-deploy bug rate fell from 7.0% to 2.9%. Automating repeated checks of core features with E2E freed up time to examine the impact and edge cases of new and changed features.
| Metric | Before E2E testing | After E2E testing | After RAG |
|---|---|---|---|
| Verified tickets | 712 | 992 | 794 |
| Average QA volume (tickets/day) | 22.3/day | 29.2/day (up ~31%) | 33.1/day (up ~49%) |
| Post-release bug ratio | 7.0% (50 / 712) | 3.9% (39 / 992) | 2.9% (23 / 794) |