Using a third-party tool—parse

While manually parsing data, as seen in the previous recipe, works very well for small strings, it can be very laborious to tweak the exact formula to work with a variety of input. What if the input has an extra dash sometimes? Or it has a variable length header depending on the size of one of the fields?

A more advanced option is to use regular expressions, as we'll see in the next recipe. But there's a great module in Python called parse (https://github.com/r1chardj0n3s/parse) that allows us to reverse format strings. It is a fantastic tool, that's powerful, easy to use, and greatly improves the readability of the code.

Getting ready

Add the parse module to the requirements.txt file in our virtual environment and reinstall the dependencies, as shown in the Creating a virtual environment recipe.

The requirements.txt file should look like this:

delorean==1.0.0
requests==2.18.3
parse==1.8.2

Then, reinstall the modules in the virtual environment:

$ pip install -r requirements.txt
...
Collecting parse==1.8.2 (from -r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/13/71/e0b5c968c552f75a938db18e88a4e64d97dc212907b4aca0ff71293b4c80/parse-1.8.2.tar.gz
...
Installing collected packages: parse
  Running setup.py install for parse ... done
Successfully installed parse-1.8.2

How to do it...

1. Import the parse function:

>>> from parse import parse

2. Define the log to parse, in the same format as in the Extracting data from structured strings recipe:

>>> LOG = '[2018-05-06T12:58:00.714611] - SALE - PRODUCT: 1345 - PRICE: $09.99'

3. Analyze it and describe it as you'll do when trying to print it, like this:

>>> FORMAT = '[{date}] - SALE - PRODUCT: {product} - PRICE: ${price}'

4. Run parse and check the results:

>>> result = parse(FORMAT, LOG)
>>> result
<Result () {'date': '2018-05-06T12:58:00.714611', 'product': '1345', 'price': '09.99'}>
>>> result['date']
'2018-05-06T12:58:00.714611'
>>> result['product']
'1345'
>>> result['price']
'09.99'

5. Note the results are all strings. Define the types to be parsed:

>>> FORMAT = '[{date:ti}] - SALE - PRODUCT: {product:d} - PRICE: ${price:05.2f}'

6. Parse once again:

>>> result = parse(FORMAT, LOG)
>>> result
<Result () {'date': datetime.datetime(2018, 5, 6, 12, 58, 0, 714611), 'product': 1345, 'price': 9.99}>
>>> result['date']
datetime.datetime(2018, 5, 6, 12, 58, 0, 714611)
>>> result['product']
1345
>>> result['price']
9.99

7. Define a custom type for the price to avoid issues with the float type:

>>> from decimal import Decimal
>>> def price(string):
...   return Decimal(string)
...
>>> FORMAT = '[{date:ti}] - SALE - PRODUCT: {product:d} - PRICE: ${price:price}'
>>> parse(FORMAT, LOG, {'price': price})
<Result () {'date': datetime.datetime(2018, 5, 6, 12, 58, 0, 714611), 'product': 1345, 'price': Decimal('9.99')}>

How it works...

The parse module allows us to define a format, such as string, that reverses the format method when parsing values. A lot of the concepts that we discussed when creating strings applies here—put values in brackets, define the type after a colon, and so on.

By default, as seen in step 4, the values are parsed as strings. This is a good starting point when analyzing text. The values can be parsed into more useful native types, as shown in steps 5 and 6 in the How to do it... section. Please note that while most of the parsing types are the same as the ones in the Python Format Specification mini-language, there are some others available, such as ti for timestamps in ISO format.

If native types are not enough, our own parsing can be defined, as demonstrated in step 7 in the How to do it... section. Note that the definition of the price function gets a string and returns the proper format, in this case a Decimal type.

All the issues about floats and price information described in the There's more section of the Extracting data from structured strings recipe apply here as well.

There's more...

The timestamp can also be translated into a delorean object for consistency. Also, delorean objects carry over timezone information. Adding the same structure as in the previous recipe gives the following object, which is capable of parsing logs:

class PriceLog(object):
  def __init__(self, timestamp, product_id, price):
    self.timestamp = timestamp
    self.product_id = product_id
    self.price = price
  def __repr__(self):
    return '<PriceLog ({}, {}, {})>'.format(self.timestamp,
                                            self.product_id,
                                            self.price)
  @classmethod
  def parse(cls, text_log):
    '''
    Parse from a text log with the format
    [<Timestamp>] - SALE - PRODUCT: <product id> - PRICE: $<price>
    to a PriceLog object
    '''
    def price(string):
      return Decimal(string)
    def isodate(string):
      return delorean.parse(string)
    FORMAT = ('[{timestamp:isodate}] - SALE - PRODUCT: {product:d} - '
              'PRICE: ${price:price}')
    formats = {'price': price, 'isodate': isodate}
    result = parse.parse(FORMAT, text_log, formats)
    return cls(timestamp=result['timestamp'],
               product_id=result['product'],
               price=result['price'])

So, parsing it returns similar results:

>>> log = '[2018-05-06T14:58:59.051545] - SALE - PRODUCT: 827 - PRICE: $22.25'
>>> PriceLog.parse(log)
<PriceLog (Delorean(datetime=datetime.datetime(2018, 6, 5, 14, 58, 59, 51545), timezone='UTC'), 827, 22.25)>

This code is contained in the GitHub file Chapter01/price_log.py.

All parse supported types can be found in the documentation at https://github.com/r1chardj0n3s/parse#format-specification.

See also

• The Extracting data from structured strings recipe
• The Introducing regular expressions recipe
• The Going deeper into regular expressions recipe