CIS Replication

Collection items and various related data sets have equivalent model definitions in the Wagtail project, which are regularly re-populated with data from CIS.

Population is achieved by running a series of Django management commands. In production, these are triggered regularly by Azure functions, but you can also run the commands locally.

Before running the following commands, run the following to put all of the relevant structural pages in place first:

python manage.py create_structural_pages

Then run each of the commands below, in the order they are listed:

• CIS Replication
  • import_movements
    • Example:
  • import_catalogue_groups
    • Example:
  • import_artworks
    • Example:
  • import_archives
    • Example:
  • import_artists
    • Example:
  • import_artist_bios
    • Example:

import_movements

Approx run time: 30 seconds

Populates the art.Movement model, which is a taxonomy used to classify collection items. It runs first so that all other imports can identify and establish relationships with the relevant movements for each page.

Example:

python manage.py import_movements --verbosity=2

import_catalogue_groups

Initial run time: 10 mins Subsequent run time: 1-2 mins

Populates the groups.CatalogueGroupPage page model with approximately 1,230 entries from CIS.

Example:

python manage.py import_catalogue_groups --verbosity=2

import_artworks

Initial run time: 10-11 hours (approx. 8 mins per 1000 items) Subsequent run time: 35-45 mins (approx. 30 seconds per 1000 items)

Populates the artworks.ArtworkPage page model with approximately 78,000 entries from CIS, as well adding/updating any CustomImage objects with images relating to each artwork (available for editors to choose from the "Artworks" collection in Wagtail).

Example:

python manage.py import_artworks --verbosity=2 --start-page=1 --stop-page=2 --resilient

import_archives

Initial run time: 3-4 hours (approx. 8 mins per 1000 items) Subsequent run time: 10-15 mins (approx. 30 seconds per 1000 items)

Populates the archives.ArchivePage and archives.ArchiveItemPage page models with approximately 23,500 entries from CIS, as well adding/updating any CustomImage objects with images relating to each item (available for editors to choose from the "Archives" collection in Wagtail).

Example:

python manage.py import_archives --verbosity=2 --start-page=1 --stop-page=2 --resilient

import_artists

Initial run time: 30-40 mins (approx. 8 mins per 1000 items) Subsequent run time: 2-3 mins (approx. 30 seconds per 1000 items)

Populates the artists.ArtistPage page model with approximately 4,400 entries from CIS. It's important for this command to run after import_artworks, because artists have a representative_work field that needs to reference an existing artworks.ArworkPage object.

Example:

python manage.py import_artists --verbosity=2 --start-page=1 --stop-page=2 --resilient

import_artist_bios

Approx run time: 3-4 mins

Uses wikipedia APIs to populate the bio and wikipedia_url field values for all ArtistPages.

Example:

python manage.py import_artist_bios --verbosity=2