Improving AI accuracy: integrating Graph RAG and human oversight

As artificial intelligence rapidly evolves, ensuring the reliability and accuracy of generated content is a growing challenge. One promising solution is retrieval-augmented generation (RAG), a method that combines the power of large language models (LLMs) with a knowledge retrieval system to improve the quality of AI-generated responses. This blog explores the concept of RAG, its integration with knowledge graphs, and how human oversight can play a crucial role in refining the process, ensuring even more reliable outputs.

What is RAG?

RAG is a method used in natural language processing that enhances the capabilities of generative AI models, such as large language models (LLMs), by combining them with a knowledge retrieval component. This approach is designed to improve the quality and reliability of the information produced by generative models, such as those used for text generation, by rooting their responses in domain knowledge.

In a typical RAG setup, the system first retrieves relevant documents or pieces of information from a large database or knowledge source when tasked with generating text. The model then uses this retrieved information as a context or reference to produce more accurate and contextually relevant responses.

Read more about RAG as a response to hallucinations here

Graph RAG and integrating RAG with a knowledge graph

Graph RAG typically refers to an adaptation of the RAG approach that specifically incorporates graph-based data structures into the retrieval process. The graph-based data serves as a source of contextualized information that is sent to the LLM.

Using graph data has two major advantages over a simpler document look-up setup. Firstly, where traditional RAG provides information from a single document or documents, a graph seamlessly combines knowledge from many different documents or data sources, providing a much richer context for the LLM to work with. Secondly, the graph can provide the LLM with more information about how different facts and concepts relate to each other, allowing it to generate more reliable, appropriate, and informed text.

In this way, integrating a RAG model with a knowledge graph combines the strengths of both generative language models and structured, relational data to enhance the quality and relevance of generated text. The approach also allows an LLM to check the accuracy of its response against the knowledge graph thereby supplementing its own internal knowledge with a more structured, flexible, and well-curated data source. There are four stages to the process:

1. Question: The user asks a question of the LLM, usually through a chat interface.
2. Key concepts: The LLM extracts the key concepts from that question and passes this to the knowledge graph.
3. Additional knowledge: The knowledge graph sends the text back to the LLM with any potential new input.
4. Augmented response: The LLM processes the additional information from the knowledge graph and creates a response.

A human-in-the-loop for Graph RAG

The "human in the loop" (HITL) approach for AI involves integrating human expertise into various stages of AI development and deployment. Human oversight over the validation and operation of an AI model ensures proper safeguarding while improving the reliability of AI-based decisions and outputs, adjusting and correcting as necessary to maintain performance and accuracy.

The integration of the HITL approach with RAG presents a powerful combination for enhancing performance and reliability. In this setup, human expertise is used to supervise and refine the retrieval processes within RAG, ensuring that the information fetched from databases or knowledge graphs is relevant and accurate. Humans can validate and correct the data used for generating responses, particularly in complex or nuanced scenarios where the context is crucial.

How DISQOVER uses human-in-the-loop

DISQOVER, ONTOFORCE’s powerful knowledge discovery platform, uses a staggered approach to GenAI assisted querying and summarization. This approach ensures that our users:

• Are in control at every stage
• Able to understand exactly what information is being retrieved and where it has come from
• Decide which information should be used for a particular purpose

These are the main steps in DISQOVER’s model:

1. A user asks a question in natural language.
2. The DISQOVER Assistant translates this question into a structured graph query, producing up to three options.
3. Rather than executing queries immediately, the Assistant shares its work with the user and asks them to select the most appropriate option.
4. After the query is executed, the user can inspect the results and choose exactly what to feed back to the LLM for further summarization or questioning.

By taking this approach, DISQOVER puts our users in full control of the process, and crucially allows them audit exactly what is happening, and precisely which information is being used to generate content. By providing this transparency, we avoid the ‘black box’ nature of other approaches and give them full confidence in both the results and the process used to obtain them.