Working with large result sets

When working with potentially large result sets, it may not always be feasible to load all data to memory and work on it. You may need to stream data elements in a controlled manner. This section shows how to deal with such situations using concurrency primitives.

Streaming results using a goroutine

In this use case, a goroutine sends the results of a query via a channel. A context can be used to cancel the streaming goroutine.

How to do it...

Create a data structure that holds the data elements and error information:

type Result struct {
  Err error
  // Other data elements
}

The StreamResults function runs the database query and creates a goroutine that iterates the query results. The goroutine sends each result via a channel:

func StreamResults(
    ctx context.Context,
    db *sql.DB,
    query string,
    args ...any,
) (<-chan...