========================
Delphi API Elasticsearch
========================

This repo contains the Delphi Elasticsearch Utils CLI, index configurations, and various request
examples related to the Delphi API's Elasticsearch indices.

:Release:
   .. image:: https://img.shields.io/badge/release-v1.0-blue.svg?style=flat-square

:Maintained by:
    Delphi Team – Sony Music Data & Analytics


Glossary
--------

`index mapping`_:
    Mapping is the process of defining how a document, and the fields it contains, are stored and indexed.

.. _`index mapping`: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html


CLI Utility
-----------

The CLI utility can be run via the convenience shell script:

   .. code-block:: bash

     ./bin/es-tools.sh --help


Bulk Indexing
^^^^^^^^^^^^^

Bulk indexing jobs can be run via:

   .. code-block:: bash

     ./bin/es-tools.sh bulk-index --name linkfire-link


Create & Reindex
----------------

Using Manual HTTP requests
^^^^^^^^^^^^^^^^^^^^^^^^^^

#. Use the `create index API <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html>`_
   to create a new index (ex: ``artist``\ ) with new mapping configuration.

   * See commands example: ``1-create-index.http``

#. Use the `reindex API <https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html>`_
   to copy documents from the old index to the new one.

   * See commands example: ``2-reindex.http``

#. Use the `task management API <https://www.elastic.co/guide/en/elasticsearch/reference/current/tasks.html>`_
   to retrieve the status of the reindexing task.

   * See commands example: ``3-get-task-status.http``

#. Use the `bulk index alias API <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html>`_
   to alias the new index (ex: ``artist``\ ) so that it's used for reading and writing.

   * See commands example: ``4-update-aliases.http``

#. Use the `Open Distro ISM API <https://opendistro.github.io/for-elasticsearch-docs/docs/im/ism/api/#add-policy>`_
   to set indexes policy by alias.

   * See commands example: ``5-add-policy.http``


Related Documentation
^^^^^^^^^^^^^^^^^^^^^

* `Change the mapping of an existing field <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings>`_
* `ASCII folding token filter <https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-asciifolding-tokenfilter.html>`_
* `Lowercase token filter <https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lowercase-tokenfilter.html>`_


Generating Documentation
------------------------

This project's documentation from code can be automatically generated via Sphinx_.

The below assumes you have already followed the steps above in "Getting Set Up Locally".

.. _Sphinx: http://www.sphinx-doc.org/en/stable/index.html

#. Generate the stub ``.rst`` files and the HTML browser files with ``make`` from the project root:

   .. code-block:: bash

    make docs


Linting Your Code
-----------------

Run before pushing:

  .. code-block:: bash

    make sort-imports


Checking Code Quality
---------------------

Use the pylint command to check quality of code updates:

  .. code-block:: bash

    make lint


Creating Releases
-----------------

Use bump2version_ to create a new Delphi API release.

  .. code-block:: bash

    # dry-run to see what will happen
    bump2version --verbose patch --allow-dirty --dry-run

    # update patch version
    bump2version --verbose patch


.. _bump2version: https://github.com/c4urself/bump2version