Overview

Retrieval Augmented Generation (RAG) is a technique that combines the strengths of traditional retrieval-based and generative AI-based approaches to question-answering (QA) systems. In a RAG chatbot, the user’s query is first used to retrieve relevant documents from a knowledge base. The retrieved documents are then passed to a large language model (LLM), which generates a response based on the query and the retrieved context.

RAG chatbots offer a number of advantages over traditional generative AI chatbots. First, RAG chatbots can provide more accurate and informative responses, as they have access to a knowledge base of relevant information. Second, RAG chatbots are more efficient, as they do not need to generate responses from scratch each time.

In one of our previous articles here, we had a look at how to build a chatbot with a RAG system using CSSF data with Azure OpenAI and Pinecone. The objective was to see how effectively we could source a foundation model using the CSSF public data, such as circular documents, faqs, website pages and many others. And also in this article (link), how to build a medical GenAI bot on AWS.

In this article, we will discuss how to build a RAG chatbot on AWS. We will compare different solutions and the results with our previous article. We will also discuss how to store embeddings and what services can be used to implement a RAG source on AWS.

How does a RAG system work

A RAG system works by first retrieving relevant documents from a knowledge base for a given query. The retrieved documents are then passed to a large language model (LLM), which generates a response based on the query and the retrieved context.This allows the LLM to generate more accurate responses. And also will be able to answer even questions that it has not been trained with. Here is a step-by-step overview of how to build a RAG system and include documents:

1. Gather documents: Collect all sources of information relevant to the task, such as documents (text, pdf, docx), web pages, email etc.. In our case we created a csv file where we listed all public documents (mostly pdfs and web pages) with their formats and source urls

2. Deploy an Embedding Model: This step includes converting gathered documents into embeddings to that store and query them after an user input. Many options are available at AWS, such as using Amazon’s Titan model via the Bedrock service or alternatively you can use SageMaker JumpStart where you can create Model Endpoints for prebuilt text embedding models such as BERT, RoBERTa etc.. Here’s the full list of available models: link

3. Generate Embeddings: Use a pre-trained text embedding model to generate embeddings for each document in the knowledge base. Embeddings are vector representations of documents that can be used to measure the similarity between documents and to retrieve relevant documents for a given query. Depending on the size of the document you might want to split them into chunks (by single page or by number of characters). In this demonstration, we splitted the documents into chunks of 3000 characters.

4. Index embeddings: We have to store those generated embedding somewhere so that we can query later on. Most vector stores are using what we call the K-Nearest Neighbors (KNN) search algorithm to search through the embeddings. KNN provides a scalable and efficient way to search through large datasets, ensuring that the most relevant documents are retrieved quickly.

5. Retrieve relevant documents: Now that we have an embedding database we should be able to retrieve relevant documents to the user’s input. But to query a vector store we also need to convert the user’s input to embeddings so that we can query our vector store and retrieve nearest files. Based on the query’s embeddings, we can retrieve the top “k” most similar documents from the indexed knowledge source.

6. Combine the final prompt to LLM: Final step here is to combine all of the information together. This includes the LLM prompt that we are using to interact with our foundation model, the user’s query, and the relevant documents. For example when a user asks: “Can you summarize the CSSF circular 23/34”, we will retrieve the circular document with number 23/34 and append it to the prompt.

RAG systems offer a number of benefits over traditional generative AI chatbots:

• Accuracy: RAG systems can generate more accurate responses, as they have access to a knowledge base of relevant information.
• Efficiency: RAG systems are more efficient, as they do not need to generate responses from scratch each time.
• Flexibility: RAG systems can be used to answer a wider range of questions, including questions that the LLM has not been trained with.

Here’s an overview diagram of how a RAG system is working:

Vector Stores in AWS

As the generative AI space grows, so too does the number of databases that support embeddings. Many options are available, including pure vector databases and multi-model databases that support storing embeddings. Some well-known and popular options include PostgreSQL, MongoDB, and Redis. For a list of vector database engines and their trends, see this link. If you are unfamiliar with embeddings, please see our previous article by Alain for an explanation.

On AWS, there are multiple options for building a RAG system, including:

• Kendra
• OpenSearch (AWS fork of the open source ElasticSearch)
• Aurora Database - PostgresQL with pgvector plugin
• MongoDB
• Redis

In this article, we’ll have a deeper understanding of the first three solutions. Let’s dive in!

Amazon Kendra

AWS Kendra is an intelligent enterprise search engine that allows you to search across different content sources with built-in connectors. It uses Machine Learning to understand the semantic meaning of your content and return relevant results. Kendra is one of the easiest RAG source services to set up and to use. It takes care of most of the hurdles, as you don’t need to convert the query or the documents to embeddings. Additionally, you can directly index the source of documents and even web pages with a few clicks.

In our case, we created an S3 bucket containing all the PDF and text documents. We also used Kendra connectors to crawl the CSSF’s web page for additional sources of information. To crawl a web page, you simply need to provide a seed URL. You can even set up a recurring schedule to automatically recrawl the web pages.

Kendra can be used to search a wide range of content types, including:

• S3 object buckets
• Databases
• Documents (PDF, TXT, CSV)
• Web pages
• Email
• Code repositories
• Customer support tickets
• Messaging systems (like Slack)

For a complete list of connectors that you can use with Kendra, see the connector list.

To retrieve relevant documents from Kendra, you can use the Retrieve API. This will return the relevant passages from documents with a confidence score. You can also filter based on document fields and attributes.

Another cool thing about Kendra is that it has a built-in user access permission setup. You can control the access to documents based on the user’s context by using a user token, user ID, or a user attribute (more information here).

Kendra has very good features and puts forward the ease of development to retrieve documents from an index. If we look at the advantages:

• Ease of importing documents. You just need to create connectors and Kendra does the rest to ingest and index.
• No need to convert documents or the user input to embedding: This can save you a lot of time and effort.
• Ability to schedule connectors to scan through a document source: This can help you keep your RAG index up to date and also avoid recurring manual tasks

But all of this comes with a catch, the price of the service. Although you can test Kendra for free for a month, it quickly becomes expensive if you want to use it in a production environment. Kendra comes in two tiers: developer and enterprise edition, priced at $810 and $1008 respectively. Plus, you have to take into consideration the hourly rate of connector synchronizations.

Amazon OpenSearch

The second solution to store embeddings on AWS is OpenSearch. OpenSearch service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in AWS. This service is an Amazon version of the fully open-source and analytics engine Elasticsearch, used mostly for log analytics, application, and enterprise search.

OpenSearch 2.9 version also supports the storing of vector embeddings, which is perfect for our case.

To store and index vector in OpenSearch we are using a plugin called k-NN (short for k-nearest neighbors). The k-nn plugin enables searching for the k-nearest neighbors to a query point across an index of vectors. This plugin supports three different methods for vector search:

• Approximate k-NN: like its name, it takes an approximate approach on nearest neighbors by using several algorithms like nmslib and faiss. These algorithms generally trade-off the search accuracy in return for performance, latency, and smaller memory.
• Script Score k-NN: This one executes a brute force, exact k-NN search over “knn_vector” fields.
• Painless extensions: On top of Script Score, this one also supports pre-filtering using other fields.

To create indexes and mappings you can use the OpenSearch API, dashboard (provided by the service) and the dev tools on the dashboard

1. Create Index that supports KNN plugin

   1
   2
   3
   4
   5
   6
   7
   8
   9
   PUT /cssf-rag-poc
   {
     "settings": {
    "index": {
      "knn": true,
      "knn.space_type": "cosinesimil"
    }
     }
   }
   

2. Create Mapping for the index

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
    {
    "properties": {
        "embedding": {
            "type": "knn_vector",
            "dimension": 1536 //length of created embeddings
        },
        "document_id": {"type": "text"},
        "url": {"type": "text"},
        "title": {"type": "text"},
        "page": {"type": "integer"},
        "document_type": {"type": "text"},
        "language_code": {"type": "text"},
        "cssf_circular_num": {"type": "text"},
        "page_content": {"type": "text"}
    }
   }
   

3. Import embeddings

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   payload = {
        "embedding": embeddings,
        "document_id": title,
        "url": file_path,
        "title": title,
        "page": index,
        "document_type": doc_type,
        "cssf_circular_num": circular_num,
        "language_code": lang,
        "page_content": item.page_content
   }
   opensearch_client.index(
        index=index_name,
        body=payload
    )
   

4. Query

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
vector_query = {
    "size": 5,
    "query": {
        "knn": {
            "embedding": {
                "vector": query_embeddings,
                "k": 2
            }
        }
    },
    "_source": [
        "page_content",
        "document_id",
        "document_type",
        "cssf_circular_num",
        "lang",
        "url"
    ]
}
resp = aos_client.search(body=vector_query, index=AWS_AOS_INDEX_NAME)

Amazon Aurora Serverless

Relational Database Service (RDS) is AWS’s managed database service that supports a wide range of database engines, including MySQL, PostgreSQL, Oracle, SQL Server, MariaDB, and Amazon Aurora. In this demo, we used Aurora Serverless v2 with PostgreSQL compatibility. Aurora PostgreSQL is up to 5x faster than PostgreSQL databases running on traditional hardware and can scale horizontally.

Aurora Serverless does not provide an out-of-the-box solution for storing vector embeddings, but there is an open-source plugin called pgvector that can be used for this purpose. pgvector provides an exact nearest neighbor (NN) search by default, but it also supports approximate NN search for better performance - this has a tradeoff between speed and recall. The supported indexes are IVFFlat and HNSW. We are not going through them in this article, but if you are curious, the documentation can be found here.

1. Deploy Aurora: You actually have three options to deploy a PostgreSQL database: RDS, Aurora, and Aurora Serverless. For this use case, we chose Aurora Serverless PostgreSQL. For pgvector, you will need a PostgreSQL version 11 and above.

2. pgvector Plugin: You can easily activate the pgvector plugin on Postgresql using one of the PostgreSQL clients (been using DBeaver for a while).

   1
   CREATE EXTENSION vector;
   

3. Create table: Create a table to store the vector embeddings. The following table definition shows an example:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   CREATE TABLE embeddings (
    id SERIAL PRIMARY KEY,
    embedding vector(1536),
    cssf_circular_num varchar,
    document_type varchar,
    document_id varchar,
    language_code varchar,
    page integer,
    page_content varchar,
    title varchar,
    url varchar
   );
   

4. Index embeddings:

   1
   2
   INSERT INTO embeddings (embedding, cssf_circular_num, document_type, document_id,language_code, page, page_content, title, url) 
   VALUES ([1,2,3],”08/350”,”PDF”, , “EN”, 3, “Some Content”, “Circular 08/350”, “https://cssf.lu/08350en.pdf”)
   

5. Query Embeddings: To query the embeddings, you can use the following SQL statement:

   1
   2
   3
   SELECT *
   FROM embeddings 
   ORDER BY embedding <-> '{embedding}' LIMIT 5;
   

Improving the conversation

Searching for relevant documents in RAG sources for each query is unnecessary and resource-intensive, especially for small talk or irrelevant questions. In our case, we wanted to search RAG sources if necessary and also be able to filter documents with extracted information. Like the following example:

1
2
3
User Input: => "Can you summarize the CSSF Circular 08/350?"
User intent => “Circular” Get this specific document
Information to extract => circular 08/350

Above, the intent of the user is for a specific document, and we need to extract the circular number so that we can make a query on RAG and filter on a specific document.

We categorize four types of intents:

• GenericCSSF: Any generic question about the CSSF
• GenericCircular: Any generic question about circulars of the CSSF
• Circular: a question about a specific circular (just like the example above)
• SmallTalk or OT: small talk, off-topic

Here, small talk and off-topic questions would not need a RAG source retrieval, whereas the other would. Also, we can go even further with circular and even filter relevant documents by the circular number extracted from the user input.

To solve this problem, we first tried chaining LLM queries using the famous library LangChain (for more information, read our article here). By chaining LLM queries, we were able to refine questions and extract the intent of the user and the information in order to decide whether or not to make a RAG search.

We also wanted to try something new and test out the Amazon Lex service (read about our thoughts on it later here) and see if we can use it to guide the conversation between the user and our LLM. Lex is a fully managed service that makes it easy to build, test, and deploy conversational interfaces into any application using voice and text. It is based on the same Deep Learning technology that powers Amazon Alexa devices.

In this demonstration, we used Lex to understand the intent of the user and extract important information to retrieve relevant documents. We created intents (see screenshot a), such as for circular queries, generic CSSF questions, small talk, and also a fallback intent if no intent match was made. If the user’s intent is one of the first three, we try to find relevant documentation; otherwise, the LLM will directly generate a response without additional documents.

We configured these intents by giving Lex some sample utterances, which are example queries that users might input. Using NLP, Lex tries to match the intent from the user input and the sample utterances. You can see below (screenshot b), the sample utterances that we created for the “Circular” intent.

You can also see in the screenshot b, some template called “circular_num”. These are actually called “Slots” in Amazon Lex. Slots are used to extract specific information from user input (in our case, Circular Number). Just like the Intents, you can create Slots and enter some example values string or in regex and their variations, and Lex will try to match for you.

When you call the recognizeText API of Lex, it returns the possible matches of intents with their score ranking, as well as the slot types that were captured in the process. You can see below a screenshot of the test UI of Lex, where you can see the intents and slot types for a user input. So, in summary, without interacting with our foundation model, we can get the user’s intent and basically understand if we need to query our RAG source or not.

Benefits of using Lex as a conversation handler:

• Much faster response than an LLM response generation: If you just want to get the intent behind the user’s input, it’s great.
• Cost-effective: Let’s be honest, each LLM response generation costs us money, by the token! So it’s much cheaper to use Lex for this use.
• Easy to set up: For our use case, it was really easy to set up and start using Lex. We created a couple of Intents and Slots, and that was it. It might not be the case if you are trying to build a full chatbot with conditions and so on.

Disadvantages of using Lex:

• Requires a language code in the request: While interacting with Lex API, you also send a localID, basically a language code in the request. And you might need to switch between localID’s if you want to support multiple languages on your application. So the issue is that Lex does not automatically recognize the language of the text. You might need to use Amazon Translate service or a similar thing to capture the language. Luckily, I was able to find a small library in Python that did the job without any API calls.
• Requires retraining: Each time you need to add intents and slots or different language support, you will need to retrain your bot with example questions.
• Character limitation: Another limitation of Lex is that the recognizeText API is limited to only 1024 characters, and this applies to some of the LLM chatbot use cases.

Overall, Lex is a powerful tool for guiding the conversation between a user and an LLM. It offers several advantages over LLM response generation, such as faster response time, cost-effectiveness. However, Lex does require additional setup and maintenance, such as providing the language code in the request and retraining each time we change the structure (intents and slots) or add new languages.

Architecture

Let’s now have a look at the architecture of the solution. The solution is similar to the one used in our previous article (here). We just added RAG sources and also the part for converting documents into embeddings. The main parts of the solution are:

1. Web UI: A single-page application written in ReactJS, stored in an S3 bucket, and distributed with CloudFront CDN.
2. Backend: Lambda function exposed via the API Gateway.
3. Authentication and authorization: Provided by the Cognito service. A login page with Hosted UI on Cognito was created, and API calls are authorized using Cognito via an intermediary Lambda Authorizer function.
4. S3 Bucket: Used for storing the documents for our RAG sources. Newly uploaded documents are converted to embeddings and indexed to the different RAG sources
5. Lambda Function: Core component of the solution, responsible for the following:
   • Getting the user intent from Amazon Lex.
   • Converting the question into embeddings.
   • Searching the RAG source for the relevant documents.
   • Creating a prompt using the information from the previous steps.
   • Calling Claude model on Amazon Bedrock to generate a response.

While developing this solution, we encountered a limitation of API Gateway. Apparently, API Gateway has a request timeout duration of 29 seconds, and that limit is not configurable. To bypass this limitation, we found two choices: Expose the lambda function directly using a function URL. (documentation) Create a WebSocket API using API Gateway, but this created a need to store an external map of user-client connections. The second one being too complex for a demo, we opted for the function URL.

Solution Comparison

In terms of the choice of solution, both Amazon OpenSearch and Aurora PostgreSQL provide the same level of functionality. OpenSearch is a distributed search and analytics engine, while Aurora is a relational database. Therefore, the best choice for your team will depend on their experience and expertise.If your team is accustomed to using relational databases, then choose Aurora. If they have prior experience with OpenSearch or ElasticSearch, or if you need a solution that can scale to handle a large volume of data, then choose OpenSearch.

Both Aurora and OpenSearch require some level of development knowledge and document handling (splitting into chunks, converting to embeddings). However, Amazon Kendra excels in its ability to quickly develop and build a RAG system without the need to process the documents. Kendra eliminates the need to process the documents because it has a built-in document processor that can handle all of the necessary tasks. This makes Kendra a good choice for teams that want to build a RAG system quickly and easily, without having to invest in a lot of development resources.

Let’s now look at the prices of the solutions:

* With our demo usage, we were able to stay in the Lambda free tier
** We used OpenAI Embedding to convert the documents to embeddings using text-embeddings-ada model. The price on OpenAI is $0.0001 per 1000 tokens

Notes about the demo:

• We indexed around 500 documents in the RAG Sources. (400 PDF files and 100 webpages)
• The total cost is calculated for 1000 queries
• We split the documents into chunks of 3000 characters with an overlap of 200 characters. In total 5020 documents are stored in RAG sources. On Average a document contains 1000 tokens.

We’ll also compare the results and accuracy with our previous demo (previous article). Let’s have a look at the results on three different sources.

Let’s wrap this up

Choice of RAG Source

Building a RAG chatbot on AWS can be a complex task, but it can be done using a number of different tools and services. Amazon Kendra is a good option for those who want an easy-to-use solution, but it can be expensive. Other options, such as OpenSearch Service and Aurora Database, are more complex to set up but can be more cost-effective.

When we look at the responses, if Claude is provided with the relevant documentation, he is able to concisely answer questions, explain, and summarize about the CSSF. Despite Claude being able to communicate complex technical concepts in a clear and concise way, Lex is causing confusion by not providing complete information..

In terms of the RAG sources, Aurora and OpenSearch are working quite well with similar retrieval speeds. We noticed that all three sources are retrieving similar documents if the questions contain all the information. It’s not shocking for Aurora and OpenSearch because they have the same chunks and embeddings.

Conversation Handling with Amazon Lex

Amazon Lex is a good choice for building chatbots in a limited scope, in terms of intent capture and language support. For example if you have a small use case with a limited number of possible questions, you can use Lex to handle the conversation. However, it becomes more difficult to use Lex when you need to extract information from linked questions. For example, if you ask Lex the following questions:

1. When was the circular CPDI 23/34 published?
2. Could you summarize the circular for me?

Lex will lose the history of extracted information from the first question and will not be able to retrieve the relevant document (circular 23/34) for the second question.

One way to work around this limitation is to create a tree structure in Lex to continue the conversation. However, this approach can be complex and time-consuming, especially if you have a large number of possible questions. Also, what’s the point of Generative AI if we are building a conversation tree for each possible action of a chatbot.

Another option is to use a chained LLM query to answer linked questions (like we did in our previous article). We can use the LLM’s to refine the question and extract the intent and information. This approach is more accurate than using Lex, but it is also more expensive, as each LLM query costs money.

Limitations of Claude

Another limitation was the prompt structure and answers of Claude. After many iterations, we were able to come up with something manageable, but it still has its limitations, such as Claude not paying attention to its instructions. For example:

Use of XML tags just as suggested by the Anthropic prompt guide (link), but Claude kept using the same XML tags in its responses. From our iteration, we concluded that it’s better to mark specific sections like the following example. This reduces Claude’s usage of section separators significantly. But we can see sometimes that Claude still uses separators in its answers. Check out the question-response for “When was the circular CPDI 23/34 published?”

XML Section Separator example:

1
2
3
4
Human: 
<history>
${HISTORY}
</history>

Second type of separators:

1
2
3
4
5
6
7
8
The CSSF description :   
-------------------
${CSSF_DESCRIPTION}
-------------------
Relevant Documents:
-------------------
${DOCUMENTS}
-------------------

Also, the same goes for Claude proactively asking and answering questions by itself. Check out this question above:

1
"je cherche aussi les circulaires détaillant les risques ICT pour les sociétés réglementées par le cadre PSF. Peux-tu me les indiquer?"

Have a look at the highlighted passage. You’ll notice at the end that Claude proactively asks and answers questions on its own, even though Claude’s instructions specifically include:

1
"Do NOT proactively ask and answer questions on your own. Only respond directly to the user's questions."

The bad side is that Claude does this multiple times within a response.

Demo Screenshots