As a geospatial analyst, you may frequently encounter the following general data types:

Each format contains its own challenges for access and processing. When you perform analysis on data, usually you have to do some form of preprocessing first. You might clip or subset a satellite image of a large area down to just your area of interest, or you might reduce the number of points in a collection to just the ones meeting certain criteria in your data model. A good example of this type of preprocessing is the SimpleGIS example at the end of Chapter 1, Learning Geospatial Analysis with Python. The state dataset included just the state...

Despite dozens of formats, geospatial data have common traits. Understanding these traits can help you approach and understand unfamiliar data formats by identifying the ingredients common to nearly all spatial data. The structure of a given data format is usually driven by its intended use. Some data is optimized for efficient storage or compression, some is optimized for efficient access, some is designed to be lightweight and readable (web formats), while other data formats seek to contain as many different data types as possible.

Interestingly, some of the most popular formats today are also some of the simplest and even lack features found in more capable and sophisticated formats. Ease of use is extremely important to geospatial analysts because so much time is spent integrating data into geographic information systems as well as exchanging data among analysts. Simple data formats facilitate these activities the best.

Geospatial datasets are often very large files easily reaching hundreds of megabytes or even several gigabytes in size. Geospatial software can be quite slow in trying to repeatedly access large files when performing analysis. As discussed briefly in Chapter 1, Learning Geospatial Analysis with Python, spatial indexing creates a guide, which allows software to quickly locate query results without examining every single feature in the dataset. Spatial indexes allow software to eliminate possibilities and perform more detailed searches or comparisons on a much smaller subset of the data.

Overview data is most commonly found in raster formats. Overviews are resampled, lower resolution versions of raster datasets that provide thumbnail views or simply faster loading image views at different map scales. They are also known as pyramids and the process of creating them is known as pyramiding an image. These overviews are usually preprocessed and stored with the full resolution data either embedded with the file or in a separate file. The compromise of this convenience is that the additional images add to the overall file size of the dataset; however, they speed up image viewers. Vector data also has a concept of overviews, usually to give a dataset geographic context in an overview map. However, because vector data is scalable, reduced size overviews are usually created on the fly by software using a generalization operation as mentioned in Chapter 1, Learning Geospatial Analysis with Python.

Occasionally, vector data is rasterized by converting it into a thumbnail image...

As discussed in Chapter 1, Learning Geospatial Analysis with Python, metadata is any data that describes the associated dataset. Common examples of metadata include basic elements such as the footprint of the dataset on the Earth as well as more detailed information such as spatial projection and information describing how the dataset was created. Most data formats contain the footprint or bounding box of the data on the Earth. Detailed metadata is typically stored in a separate location in a standard format such as the U.S. Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata (CSDGM), ISO, or the newer European Union initiative, which includes metadata requirements, called the Infrastructure for Spatial Information in the European Community (INSPIRE).

The preceding elements can be stored in a variety of ways in a single file, multiple files, or database depending on the format. Additionally, this geospatial information can be stored in a variety of formats, including embedded binary headers, XML, database tables, spreadsheets/CSV, separate text, or binary files.

Human readable formats such as XML files, spreadsheets, and structured text files require only a text editor to investigate. These files are also easily parsed and processed using Python's built-in modules, data types, and string manipulation functions. Binary-based formats are more complicated. It is thus typically easier to use a third-party library to deal with binary formats.

However, you don't have to use a third-party library, especially if you just want to investigate the data at a high level. Python's built-in struct module has everything that you need. The struct module lets you read and write binary data as strings. When using the struct module, you need...