Implementing the LLM Twin’s RAG feature pipeline
The last step is to review the LLM Twin’s RAG feature pipeline code to see how we applied everything we discussed in this chapter. We will walk you through the following:
- ZenML code
- Pydantic domain objects
- A custom object-vector mapping (OVM) implementation
- The cleaning, chunking, and embedding logic for all our data categories
We will take a top-down approach. Thus, let’s start with the Settings class and ZenML pipeline.
Settings
We use Pydantic Settings (https://docs.pydantic.dev/latest/concepts/pydantic_settings/) to define a global Settings class that loads sensitive or non-sensitive variables from a .env file. This approach also gives us all the benefits of Pydantic, such as type validation. For example, if we provide a string for the QDRANT_DATABASE_PORT variable instead of an integer, the program will crash. This behavior makes the whole application more deterministic...
