Pointwise evaluations

Let’s start our deeper discussion with pointwise evaluations. Just to recap, a pointwise evaluation is when you compare the generated output (for a given input) with the expected output. First, let’s start with using existing benchmarks for evaluation.

Comparison on benchmarks

When solving enterprise problems, you typically work on already explored problem classes. Your goal is to choose the best approach from existing ones and adjust it to the specific business needs (and that’s one of the distinctions between research and industry). That means that typically, it’s a good idea to take a look at how this type of problem has been formulated in academic literature, what dataset has been collected, and how solutions have been evaluated.

There are dozens or hundreds of publicly available benchmarks and datasets (some of them are not intended for commercial use though, or are subject to an additional license). The industry is evolving...