Seamless access to data with LLM based agents
Have you ever struggled with answering questions fast and accurately using data coming from multiple sources? We’ve implemented an AI powered assistant for you to engage in human-like conversations and get those immediate answers that you’re looking for.
Understanding the problem
By interviewing different stakeholders and potential users of the tool, we identified common types of questions and use cases in which the tool could provide a solution. We mapped common unanswered questions to which platforms in the company offered this information, and we reviewed their associated APIs. Based on this understanding we then assembled a team with all the necessary skills to implement the tool.
Getting answers from given data with Retrieval Augmented Generation
We used Retrieval Augmented Generation (RAG), which enables us to exploit generative AI to answer questions in the wild based on existing knowledge databases. Instead of training or fine tuning an LLM using this information (which is expensive, time consuming and might be prone to hallucinations), RAG uses semantic search based on embeddings to retrieve pieces of information from a database that are relevant for answering the asked question. Then, a Generative AI tool (e.g. OpenAI GPT 3.5, in our case) is prompted with instructions, the question and the contexts, to craft a human-like answer.
An Agent-based solution for more human-like interactions
Instead of using classic GenAI prompting, we implemented LLM-based Agents. Using language as a proxy for reasoning, these tools plan the actions to follow to collect the data to answer a given question, choosing them from a predefined set. We carefully implemented actions to retrieve data from our different sources, depending on the overall characteristics of the input question. Thus, the Agent is able to choose the right source of information, and follow the necessary steps to get data.
A Slack interface for seamless integration with the current way of working
As we use Slack for our regular conversations with teammates and is the prioritized channel of communication of the company, we decided to use it as the interface for communicating with our bot. We fully integrated our tool on Slack, including features like welcome messages and onboarding info, to learn how to use tool the first time you interact with the bot during the day, feedback buttons as reactions to indicate if the answer was good or not, and even push notifications every 2 weeks, to let users know about the new implemented features.
Our own regression testing framework to always ensure reliable answers
To make sure that any new implemented feature or change in our Agents does not affect the quality of previous functionalities, we needed to implement a regression testing framework. This is challenging, considering that answers are non-deterministic due to the intrinsic nature of Generative AI. We implemented a custom solution based on prompt engineering and LLMs, which uses a series of test sets of questions crafted in advance for every new feature, and automatically verifies which are correctly answered by the bot. Hence, when every new feature is deployed, we trigger the regression testing pipeline to ensure that performance is higher or remains at least the same.
A user-metrics dashboard to monitor usage of the tool
We built a dashboard in Grafana that allows the main stakeholders of the company to monitor the overall usage of the application, while providing the development team with relevant metrics to understand the behavior of the bot and keep improving it. These include number of questions and new users, percentage of feedback provided, positive and negative responses, and many others. This dashboard is intended to be dynamic, meaning that we will keep adding new features as we keep developing our bot.
Our ML team brought to life a fully functional product seamlessly integrating with our existing way of working, that enables all our internal teams to rapidly have answers to their questions while finding necessary sources of information.
Our Engineering team develop a rock solid architecture that enabled us to include each connection one by one, including an automated QA pipeline to make sure that every new feature was working correctly.
Our product team led the design of a transparent and user friendly experience that all our team members love, identifying opportunities to add value for the users.
Our tool can be rapidly implemented in other contexts to digest other knowledge databases, including organizational reports, minutes from past meetings, customer support info, etc.
AK Bot is a fully functional chatbot that connects users with answers to their questions and knowledge resources in a fraction of a second. With our experience in RAG and Agents, we’re ready to deliver these tools fast and accurately.