Generative AI for Text is Revolutionary, why so?

For high accuracy with any AI/ML Project on a down-stream task, finetuning is required. This requires a great amount of data, processing power and time. All of these are expensive, and especially a real-world scenario - when data changes frequently. Retraining the model on every update is infeasible.

Large Language Models have a solution to this, one can provide an example of what they want in the prompt. "Give me this data in the below format", or "Using this extract, answer the following questions".

This feature of LLMs allow us to connect them to huge databases of real-time information which are constantly updated. The traditional database consists of rows and columns, this novel Database consists of Dimensions and Vectors - Aptly, they're called Vector Stores.

We use Embedding Models to convert a String to a Vector, this vector encapsulates all the information in the sentence, with a highlight on Semantic Information.

Our source documents are converted to Vectors and inserted in the database, we take the question passed to the LLM and convert it to a vector as well. Using Dot-Product or Cosine Similarity formulas, we compare all vectors in the database to our query vector to get a metric of similarity.

The most similar documents to the query, meaning that the query and document contain similar semantic meaning, are given to the LLM. The LLM is then asked to generate a response using the data provided in the prompt. This enables us to the specific information from which the model is generating information, and the model gets relevant documents from which it can answer accurately.

Originally written June 10, 2023, Rewritten June 18, 2023.