First, let's download Lucene. Apache Lucene can be downloaded from its official download page. As of now, the latest version of Lucene is 4.10. Here is the link to the official page of Lucene: http://lucene.apache.org/core/

The Download button will take you to all available Apache mirrors where you can download Lucene.

Lucene's index contains {lucene version}.zip or {lucene version}.tar.gz, including the Lucene core library, HTML documentation, and demo application. Meanwhile, {lucene version-src}.zip or {lucene version-src}.tar.gz contains the source code for that particular version.

The following is a sample of what the download page looks like:

Lucene is written entirely in Java. The prerequisite for running Lucene is Java Runtime Environment. Lucene runs on Java 6 or higher. If you use Java 7, make sure you install update 1 as well. Once your download is complete, you can extract the contents to a directory and you are good to go. In case you get some errors, the links to the FAQ and mailing list of Lucene users is as follows:

Mailing List:

http://lucene.apache.org/core/discussion.html

FAQ:

http://wiki.apache.org/lucene-java/LuceneFAQ

Java Runtime is required. If you have not installed Java yet, visit Oracle's website to download Java. Here is a link to the Java download page: http://www.oracle.com/technetwork/java/javase/downloads/index.html.

You may also want to use an IDE to work on a Lucene project. Many users prefer Eclipse IDE, but you are free to make your own choice, such as NetBeans, or whatever you prefer. If you want to give Eclipse IDE a try, you can refer to the following link: https://www.eclipse.org/downloads/.

Having set up a development environment, let's proceed to create our first Lucene project.

Thanks to the well-organized and efficient architecture of Lucene, the Lucene core JAR is only 2.4 MB in size. The core library provides the basic functionality to start a Lucene project. By adding it to your Java classpath, you can begin to build a powerful search engine. We will show you a couple ways to do this in Eclipse.

First, we will set up a normal Java project in Eclipse. Then, we will add Lucene libraries to the project. To do so, follow these steps:

Another way to include the Lucene library is via Apache Maven. Maven is a project management and build tool that provides facilities to manage project development lifecycles. A detailed explanation of Maven is beyond the scope of this book. If you want to know more about Maven, you can check out the following link: http://maven.apache.org/what-is-maven.html.

To set up Lucene in Maven, you need to insert its dependency information into your project's pom.xml. You can visit a Maven repository (http://mvnrepository.com/artifact/org.apache.lucene) where you can find Lucene.

After you have updated pom.xml, the Lucene library will show up in your project after your IDE sync up dependencies.

Once the JAR files are made available to the classpath, you can go ahead and start writing code. Both methods we described here would provide access to the Lucene library. The first method adds JARs directly. With Maven, when you add dependency to pom.xml, the JARs will be downloaded automatically.

We need to first define an analyzer to initialize IndexWriterConfig. Then, a Directory should be created to tell Lucene where to store the index. With these two objects defined, we are ready to instantiate an IndexWriter.

The following is a code snippet that shows you how to obtain an IndexWriter:

  Analyzer analyzer = new WhitespaceAnalyzer();
  Directory directory = new RAMDirectory();
  IndexWriterConfig config = new IndexWriterConfig(Version.LATEST, analyzer);
  IndexWriter indexWriter = new IndexWriter(directory, config);

First, we instantiate a WhitespaceAnalyzer to parse input text, and tokenize text into word tokens. Then, we create an in-memory Directory by instantiating RAMDirectory. We configure IndexWriterConfig with the WhitespaceAnalyzer we just created and finally, we pass both Directory and IndexWriterConfig to create an IndexWriter. The IndexWriter is now ready to update the index.

An IndexWriter consists of two major components, directory and analyzer. These are necessary so that Lucene knows where to persist indexing information and what treatment to apply to the documents before they are indexed. Analyzer's treatment is especially important because it maintains data consistency. If an index already exists in the specified directory, Lucene will update the existing index. Otherwise, a new index is created.

The Lucene-analyzers-common module contains all the major components we discussed in this section. Most commonly-used analyzers can be found in the org.apache.lucene.analysis.core package. For language-specific analysis, you can refer to the org.apache.lucene.analysis {language code} packages.

Many analyzers in Lucene-analyzers-common require little or no configuration, so instantiating them is almost effortless. For our current exercise, we will instantiate the WhitespaceAnalyzer by simply using new object:

Analyzer analyzer = new WhitespaceAnalyzer();

An analyzer is a wrapper of three major components:

The analysis phase includes pre- and post-tokenization functions, and this is where the character filter and token filter come into play. The character filter preprocesses text before tokenization to clean up data such as striping out HTML markups, removing user-defined patterns, and converting a special character or specific text. The token filter executes the post tokenization filtering; its operations involve various kinds of manipulations. For instance, stemming, stop word filtering, text normalization, and synonym expansion are all part of token filter. As described earlier, the tokenizer splits up text into tokens. The output of these analysis processes is TokenStream where the indexing process can consume and produce an index.

Lucene provides a number of standard analyzer implementations that should fit most of the search applications. Here are some additional analyzers, which we haven't talked about yet:

This code snippet shows you how to create a simple TextField:

  Document doc = new Document();
  String text = "Lucene is an Information Retrieval library written in Java.";
  doc.add(new TextField("fieldname", text, Field.Store.YES));

In this scenario, we create a document object, initialize a text, and add a field by creating a TextField object. We also configure the field to store a value so it can be retrieved during a search.

A Lucene document is a collection of field objects. A field is the name of the value pairs, which you may add to the document. A field is created by simply instantiating one of the Field classes. Field can be inserted into a document via the add method.

The following code sample shows you an example of adding a simple document to an index:

public class LuceneTest {
  public static void main(String[] args) throws IOException {
    Analyzer analyzer = new WhitespaceAnalyzer();
    Directory directory = new RAMDirectory();
    IndexWriterConfig config = new IndexWriterConfig(Version.LATEST, analyzer);
      IndexWriter indexWriter = new IndexWriter(directory,
        config);
      Document doc = new Document();
      String text = "Lucene is an Information Retrieval library written in Java";
      doc.add(new TextField("fieldname", text, Field.Store.YES));
      indexWriter.addDocument(doc);
      indexWriter.close();
    }
}

Note that the preceding code snippet combined all the sample codes we learned so far. It first initializes an analyzer, directory, IndexWriterConfig, and IndexWriter. Once the IndexWriter is obtained, a new Document is created with a custom TextField. The Document is then added to IndexWriter. Also, note that we call indexWriter.close() at the end. calling this method, will commit all changes and close the index.

The IndexWriter class exposes an addDocument(doc) method that allows you to add documents to an index. IndexWriter will write to an index specified by directory.

IndexWriter provides the interface to delete documents from an index. It takes either term or query as argument, and will delete all the documents matching these arguments:

Here is a code snippet on how deleteDocuments is called:

  indexWriter.deleteDocuments(new Term("id", "1"));"));
  indexWriter.close();

Assuming IndexWriter is already instantiated, this code will trigger IndexWriter to delete all the documents that contain the term id where the value equals 1. Then, we call close to commit changes and close the IndexWriting. Note that this is a match to a Field called id; it's not the same as DocId.

In fact, deletions do not happen at once. They are kept in the memory buffer and later flushed to the directory. The documents are initially marked as deleted on disk so subsequent searches will simply skip the deleted documents; however, to free the memory, you still need to wait. We will see the underlying process in detail in due course.

Here is a code snippet that shows you how to obtain an IndexSearcher:

Directory directory = getDirectory();
IndexReader indexReader = DirectoryReader.open(directory);
IndexSearcher indexSearcher = new IndexSearcher(indexReader);

The first line assumes we can gain access to a Directory object by calling getDirectory(). Then, we obtain an IndexReader by calling DirectoryReader.open(directory). The open method in DirectoryReader is a static method that opens an index to read, which is analogous to IndexWriter opening a directory to write. With an IndexReader initialized, we can instantiate an IndexSearcher with the reader.

Here is a code snippet:

QueryParser parser = new QueryParser("Content", analyzer);
Query query = parser.parse("Lucene");

Assuming an analyzer is already declared and available as a variable, we pass it into QueryParser to initialize the parser. The second parameter is the name of the field where we will perform a search. In this case, we are searching a field called Content. Then, we call parse(String) to interpret the search string Lucene into Query object. Note that, at this point, we only return a Query object. We have not actually executed a search yet.

Here is what we learned so far and put together into an executable program:

public class LuceneTest {
    public static void main(String[] args) throws IOException, ParseException {
        Analyzer analyzer = new StandardAnalyzer();
        Directory directory = new RAMDirectory();
        IndexWriterConfig config = new IndexWriterConfig(Version.LATEST, analyzer);
        IndexWriter indexWriter = new IndexWriter(directory, config);
        Document doc = new Document();
        String text = "Lucene is an Information Retrieval library written in Java";
        doc.add(new TextField("Content", text, Field.Store.YES));
        indexWriter.addDocument(doc);
        indexWriter.close();
        IndexReader indexReader = DirectoryReader.open(directory);
        IndexSearcher indexSearcher = new IndexSearcher(indexReader);
        QueryParser parser = new QueryParser( "Content", analyzer);
        Query query = parser.parse("Lucene");
        int hitsPerPage = 10;
        TopDocs docs = indexSearcher.search(query, hitsPerPage);
        ScoreDoc[] hits = docs.scoreDocs;
        int end = Math.min(docs.totalHits, hitsPerPage);
        System.out.print("Total Hits: " + docs.totalHits);
        System.out.print("Results: ");
        for (int i = 0; i < end; i++) {
            Document d = indexSearcher.doc(hits[i].doc);
            System.out.println("Content: " + d.get("Content");
        }
    }
}

The preceding code sets up a StandardAnalyzer to analyze text, uses a RAMDirectory for index store, configures IndexWriter to put a piece of content into the index, and uses QueryParser to generate Query object, in order to perform a search. It also has a sample code that shows how to retrieve search results from TopDocs by displaying total hits, and shows matching documents by DocId.

Here is a diagram showing how the search portion works between components:

A search string enters into QueryParser.parse(String). QueryParser then uses an analyzer to process the search string to produce a set of tokens. The tokens are then mapped to the Query object, and get sent to IndexSearcher to execute a search. The search result returned by IndexSearcher is a TopDocs object where it contains statistics of total matches and DocIds of the matching documents.

Note that it is preferable to use the same analyzer for both indexing and searching to get the best results.

Here is a sample implementation on pagination:

public List<Document> getPage(int from , int size){
  List<Document> documents = new ArraList<Document>();
  Query query = parser.parse(searchTerm);
  TopDocs hits = searcher.search(query, maxNumberOfResults);
  int end = Math.min(hits.totalHits, size);
  for (int i = from; i < end; i++) {
    int docId = hits.scoreDocs[i].doc;
    //load the document
    Document doc = searcher.doc(docId);
    documents.add(doc);
  }
  return documents;
}

When we perform search in Lucene, actual results are not preloaded immediately. In TopDocs, we only get back an array of ranked pointers. It's called ranked pointers because they are not actual documents, but a list of references (DocId). By default, results are scored by the scoring mechanism. We will see more about scoring in detail in Introduction section Chapter 7, Flexible Scoring. For paging, we can calculate position offset, apply pagination ourselves, and leverage something like what we showed in the sample code to return results by page. Developers at Lucene actually recommend re-executing a search on every page, instead of storing the initial search results (refer to http://wiki.apache.org/lucene-java/LuceneFAQ#How_do_I_implement_paging.2C_i.e._showing_result_from_1-10.2C_11-20_etc.3F). The reasoning is that people are usually only interested in top results and they are confident in Lucene's performance.