Xapian-lite

Xapian-lite is a minimal Emacs dynamic module for Xapian. It provides a simple interface that allow you to index files and search for phrases. I initially wrote it for my note app Xeft.

;; Querying my ~40MB worth of notes.
(benchmark-run 100 (xeft-query-term "common lisp" xeft-database 0 10))
;;=> (0.031512 0 0.0)

Usage

Because it’s so basic, the dynamic module is very easy to use and also very flexible. To index files, use

(dolist (file (directory-files "my-note-dir"))
  (xapian-lite-reindex-file file dbpath))

This indexes each file in my-note-dir, saving them to the database at dbpath. If the database doesn’t exist yet, it is created.

To search for a term, use

(xapian-lite-query-term "search term" dbpath 0 10)

This returns a list of paths of the files that contains search term, ranked by relevance. The 0 and 10 means “return 10 results starting from the 0th place”, it is essentially used for paging. If you want all the result, use 0 and 999999.

When a file is modified, call xeft-reindex-file again on that file. If a file is removed, you don’t need to remove it from the database, it will be automatically removed. If the file has been indexed and haven’t been modified, xeft-reindex-file is (kind of) a no-op (i.e. fast).

Both file path and database path must be absolute path.

Building

To build the module, you need to have Xapian installed. On Mac, it can be installed with macports by

sudo port install xapian-core

Then, build the module by

make PREFIX=/opt/local

Here /opt/local is the default prefix of macports, which is what I used to install Xapian. Homebrew and Linux users probably can leave it empty.

You can also build an standalone module, which doesn’t require xapian dynamic library when running. For that you need to build xapian as a static library. First get the source from https://xapian.org/docs/install.html.

tar -xf <xapian tarball>
./configure --disable-shared
make CPPFLAGS=-fPIC

Now find libxapian.a in .libs directory, copy it out into xapian-lite’s project root, and run

make standalone

Xapian-lite as a dependency

I put pre-built binary for GNU/Linux and macOS in GitHub Releases. If you want to use xapian-lite in your package, you can do something like this:

(defvar xeft--linux-module-url "https://github.com/casouri/xapian-lite/releases/download/v1.0/xapian-lite-amd64-linux.so"
  "URL for pre-built dynamic module for Linux.")

(defvar xeft--mac-module-url "https://github.com/casouri/xapian-lite/releases/download/v1.0/xapian-lite-amd64-mac.dylib"
  "URL for pre-built dynamic module for Mac.")

(defun xeft--download-module ()
  "Download pre-built module from GitHub. Return non-nil if success."
  (require 'url)
  (let ((module-path (expand-file-name
                      "xapian-lite.so"
                      (file-name-directory
                       (locate-library "xeft.el" t)))))
    (cond
     ((eq system-type 'gnu/linux)
      (url-copy-file xeft--linux-module-url module-path)
      t)
     ((eq system-type 'darwin)
      (url-copy-file xeft--mac-module-url module-path)
      t)
     (t (message "No pre-built module for this operating system. We only have them for GNU/Linux and macOS")
        nil))))