Chapter 2: Applications of Natural Language Processing

Activity 2: Building and training your own POS tagger

Solution:

1. The first thing to do is pick a corpus that we want to train our tagger on. Import the necessary Python packages. Here, we use the nltk treebank corpus to work on:

   import nltk

   nltk.download('treebank')

   tagged_sentences = nltk.corpus.treebank.tagged_sents()

   print(tagged_sentences[0])

   print("Tagged sentences: ", len(tagged_sentences))

   print ("Tagged words:", len(nltk.corpus.treebank.tagged_words()))

2. Next, we need to determine what features our tagger will take into consideration when determining what tag to assign to a word. These can include whether the word is all capitalized, is in lowercase, or has one capital letter:

   def features(sentence, index):

   """ sentence: [w1, w2, ...], index: the index of the word """

   return {

   'word': sentence[index],

   'is_first': index == 0,

   'is_last': index == len(sentence) - 1,

   'is_capitalized': sentence[index][0].upper() == sentence[index][0],

   'is_all_caps': sentence[index].upper() == sentence[index],

   'is_all_lower': sentence[index].lower() == sentence[index],

   'prefix-1': sentence[index][0],

   'prefix-2': sentence[index][:2],

   'prefix-3': sentence[index][:3],

   'suffix-1': sentence[index][-1],

   'suffix-2': sentence[index][-2:],

   'suffix-3': sentence[index][-3:],

   'prev_word': '' if index == 0 else sentence[index - 1],

   'next_word': '' if index == len(sentence) - 1 else sentence[index + 1],

   'has_hyphen': '-' in sentence[index],

   'is_numeric': sentence[index].isdigit(),

   'capitals_inside': sentence[index][1:].lower() != sentence[index][1:]

   }

   import pprint

   pprint.pprint(features(['This', 'is', 'a', 'sentence'], 2))

   {'capitals_inside': False,

   'has_hyphen': False,

   'is_all_caps': False,

   'is_all_lower': True,

   'is_capitalized': False,

   'is_first': False,

   'is_last': False,

   'is_numeric': False,

   'next_word': 'sentence',

   'prefix-1': 'a',

   'prefix-2': 'a',

   'prefix-3': 'a',

   'prev_word': 'is',

   'suffix-1': 'a',

   'suffix-2': 'a',

   'suffix-3': 'a',

   'word': 'a'}

3. Create a function to strip the tagged words of their tags so that we can feed them into our tagger:

   def untag(tagged_sentence):

   return [w for w, t in tagged_sentence]

4. Now we need to build our training set. Our tagger needs to take features individually for each word, but our corpus is actually in the form of sentences, so we need to do a little transforming. Split the data into training and testing sets. Apply this function on the training set.

   # Split the dataset for training and testing

   cutoff = int(.75 * len(tagged_sentences))

   training_sentences = tagged_sentences[:cutoff]

   test_sentences = tagged_sentences[cutoff:]

   print(len(training_sentences)) # 2935

   print(len(test_sentences)) # 979

   and create a function to assign the features to 'X' and append the POS tags to 'Y'.

   def transform_to_dataset(tagged_sentences):

   X, y = [], []

   for tagged in tagged_sentences:

   for index in range(len(tagged)):

   X.append(features(untag(tagged), index))

   y.append(tagged[index][1])

   return X, y

   X, y = transform_to_dataset(training_sentences)

   from sklearn.tree import DecisionTreeClassifier

   from sklearn.feature_extraction import DictVectorizer

   from sklearn.pipeline import Pipeline

5. Apply this function on the training set. Now we can train our tagger. It's basically a classifier since it's categorizing words into classes, so we can use a classification algorithm. You can use any that you like or try out a bunch of them to see which works best. Here, we'll use the decision tree classifier. Import the classifier, initialize it, and fit the model on the training data. Print the accuracy score.

   clf = Pipeline([

   ('vectorizer', DictVectorizer(sparse=False)),

   ('classifier', DecisionTreeClassifier(criterion='entropy'))

   ])

   clf.fit(X[:10000], y[:10000]) # Use only the first 10K samples if you're running it multiple times. It takes a fair bit :)

   print('Training completed')

   X_test, y_test = transform_to_dataset(test_sentences)

   print("Accuracy:", clf.score(X_test, y_test))

   The output is as follows:

Figure 2.19: Accuracy score

Activity 3: Performing NER on a Tagged Corpus

Solution:

1. Import the necessary Python packages and classes.

   import nltk

   nltk.download('treebank')

   nltk.download('maxent_ne_chunker')

   nltk.download('words')

2. Print the nltk.corpus.treebank.tagged_sents() to see the tagged corpus that you need extract named entities from.

   nltk.corpus.treebank.tagged_sents()

   sent = nltk.corpus.treebank.tagged_sents()[0]

   print(nltk.ne_chunk(sent, binary=True))

3. Store the first sentence of the tagged sentences in a variable.

   sent = nltk.corpus.treebank.tagged_sents()[1]

4. Use nltk.ne_chunk to perform NER on the sentence. Set binary to True and print the named entities.

   print(nltk.ne_chunk(sent, binary=False))

   sent = nltk.corpus.treebank.tagged_sents()[2]

   rint(nltk.ne_chunk(sent))

   The output is as follows:

   

Figure 2.20: NER on tagged corpus