Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Mastering Apache Solr 7.x

You're reading from   Mastering Apache Solr 7.x An expert guide to advancing, optimizing, and scaling your enterprise search

Arrow left icon
Product type Paperback
Published in Feb 2018
Publisher Packt
ISBN-13 9781788837385
Length 308 pages
Edition 1st Edition
Languages
Tools
Arrow right icon
Authors (3):
Arrow left icon
Dharmesh Vasoya Dharmesh Vasoya
Author Profile Icon Dharmesh Vasoya
Dharmesh Vasoya
Chintan Mehta Chintan Mehta
Author Profile Icon Chintan Mehta
Chintan Mehta
Sandeep Nair Sandeep Nair
Author Profile Icon Sandeep Nair
Sandeep Nair
Arrow right icon
View More author details
Toc

Table of Contents (10) Chapters Close

Preface 1. Introduction to Solr 7 2. Getting Started FREE CHAPTER 3. Designing Schemas 4. Mastering Text Analysis Methodologies 5. Data Indexing and Operations 6. Advanced Queries – Part I 7. Advanced Queries – Part II 8. Managing and Fine-Tuning Solr 9. Client APIs – An Overview

Understanding tokenizers

We have previously seen that an analyzer may be a single class or a set of defined tokenizer and filter classes.

The analyzer executes the analysis process in two steps:

  • Tokenization (parsing): Using configured tokenizer classes
  • Filtering (transformation): Using configured filter classes

We can also do preprocessing on a character stream before tokenization; we can do this with the help of CharFilters (we will see this later in the chapter). An analyzer knows its configured field, but a tokenizer doesn't have any idea about the field. The job of the tokenizer is only to read from a character stream, apply a tokenization mechanism based on its behavior, and produce a new sequence of a token stream. 

What is a tokenizer?

...
lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime
Banner background image