Pipeline 2: The Vector Store Administrator
The Vector Store Administrator AI agent performs the tasks we implemented in Chapter 6, Scaling RAG Bank Customer Data with Pinecone. The novelty in this section relies on the fact that all the data we upsert for RAG is AI-generated. Let’s open Pipeline_2_The_Vector_Store_Administrator.ipynb in the GitHub repository. We will build the Vector Store Administrator on top of the Generator and the Commentator AI agents in four steps, as illustrated in the following figure:
Figure 10.7: Workflow of the Vector Store Administrator from processing to querying video frame comments
- Processing the video comments: The Vector Store Administrator will load and prepare the comments for chunking as in the Pipeline 2: Scaling a Pinecone Index (vector store) section of Chapter 6. Since we are processing one video at a time in a pipeline, the system deletes the files processed, which keeps disk space constant. You can enhance the functionality...
