Elasticsearch for dev¶

By default Wagtail uses wagtail.search.backends.database for its search engine. This just uses native DB text matching so is fine for finding pages etc in the admin during normal dev work. The production site however uses a remote instance of Elasticsearch. Use of Elasticsearch is facilitated by a Wagtail interface which essentially allows the use of the standard Django query API while Elasticsearch API calls are made behind the scenes.

When working on search features it's therefore advisable to run a local instance of Elasticsearch (with an up-to-date index) and to ensure your local dev instance of Wagtail uses this local Elasticsearch instance for its search backend.

Not doing so risks adding search features not supported by the Wagtail ES interface and general confusion over anomalies and discrepancies between local and prod search behaviour.

You could of course run Elasticsearch in a Docker container but this is a simple guide to running it on your Docker host and enabling communication to that instance, from within the Wagtail Docker container.

Installing Elasticsearch on Mac/Linux¶

~~Nice and simple, just follow the instructions here: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/brew.html#brew~~

Update: Avoid the quick Homebrew install as this will pull the latest 8+ version. Wagtail only supports ES 7.x (we currently use 7.14.2) Using version 8 will fundametally scupper indexing due to deprecated use of 'boost' values on certain fields.

It's quicker therefore to install from the download and run globally as follows:

curl -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.2-darwin-x86_64.tar.gz
curl https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.2-darwin-x86_64.tar.gz.sha512 | shasum -a 512 -c -
tar -xzf elasticsearch-7.14.2-darwin-x86_64.tar.gz
# to run
cd elasticsearch-7.14.2/
./bin/elasticsearch

View http://localhost:9200/ and some json output as below will confirm Elasticsearch is working correctly

{
  "name" : "TGM-15776.local",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OE_iv7G0SYOyDVdzRpRfXg",
  "version" : {
    "number" : "7.14.2",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "6bc13727ce758c0e943c3c21653b3da82f627f75",
    "build_date" : "2021-09-15T10:18:09.722761972Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Ensure the Docker (for mac) Wagtail web container can communicate with the host¶

Pause the tate-wagtail-web container, then add the extra_hosts line shown below, under the web service config in docker-compose.yml

web:
    ...
    # Added to local ES connection from Docker
    extra_hosts:
      - 'host.docker.internal:host-gateway'

Next add the shm_size setting below to the db service config in docker-compose.yml

db:
    ...
    # Added to enable indexing
    shm_size: 512mb

Then start the tate-wagtail-web container and go into it using fab sh

Now add the following environmental variables (in the container still)

export ELASTICSEARCH_URL=http://host.docker.internal:9200
export ELASTICSEARCH_NUMBER_OF_SHARDS=1
export ELASTICSEARCH_NUMBER_OF_REPLICAS=0

The value of 'ELASTICSEARCH_URL' just tells Wagtail to use the IP of Docker's host plus the default port number of the local instance of Elasticsearch. The setting of ELASTICSEARCH_NUMBER_OF_SHARDS and ELASTICSEARCH_NUMBER_OF_REPLICAS may not strictly be necessary but are used to set values recomended for local work in commenting in settings.base.py

At this point, Wagtail can be started. Try a test search. This should reveal that while hopefully Wagtail can connect to the local instance of Elasticsearch the instance has no indices so won't actually be of any use. So, we need to create the indices

Indexing the site¶

Run dj update_index

This creates multiple indices from the Wagtail database data that are then used by Elasticsearch to provide results.

Once finished, indices then should be shown here: http://localhost:9200/_cat/indices

A search such as http://localhost:8000/search?q=turner should now provide results exactly like the production server

Finally, Elasticsearch can be stopped with control + c