Understanding data format conversion in KDF
KDF allows the conversion of incoming data from JSON to either Apache Parquet (Parquet) or Apache ORC (ORC) format. Parquet and ORC are popular columnar formats as opposed to JSON or Comma Separated Values (CSV), which are row formats. Columnar formats provide several advantages for storage and faster querying compared to row formats, especially in big-data use cases. In row formats, data for all columns in a row is stored together, which means that when querying a subset of columns, the data for all columns needs to be read and the unneeded columns filtered out. In columnar formats, data is stored by columns. This provides the ability to only retrieve data for the columns specified. This results in less data scanned for returning query results, and more sequential reads, resulting in better performance. In addition, since data in a column tends to be similar, columnar formats allow for better compression as well. This results in space saving...
