Key Considerations for Management Best Practices
This chapter will address some niche focus areas in data engineering:
- Data profiling
- Gaps
- Restatements
- Correction
- Metadata
- Trends
- Data calendars
These topics are often hotly debated by systems developers and as a data engineer, you will want to have a voice in the decisions being made. The goal is data that remains relevant and clearly correct within its well-defined context.
The niche areas naturally fall into two types, data profiling and metadata, because creating a modern data factory will involve processing related to making data fit for purpose. Having data properly positioned for use and reuse in downstream processing is an essential need of the end user/consumer application. Today and going forward, automated intelligence, machine learning (ML), and knowledge engineering will be how data is leveraged and then later re-leveraged without the need to create new custom software. Data will...
