Completing Our Project
So, we are at the end of the project, and we now need to add meat to our work. We have built the scaffold of our project, but we don’t really have anything else at the moment. We still need to create code for all of our apps. When we have done this, we will deploy the code to the public PyPI servers. This will be critical because we are now going to pull our pipeline code from a code repository, which is the ideal scenario. We will also set up CI for our code, which will do the checking and scanning of our code. Given the limited space, we will not be covering deployment using the CI of pipeline code, but this is the next step in that process. We will also cover schema management and some limited data governance. The goal is to have a working example of a data pipeline that is in line with something you would see in production.
This chapter covers the following topics:
- Documentation
- Faking data with Mockaroo
- Managing our schemas with...
