Skip to content

Data retention

While collecting data can be fun, some housekeeping and cleanup might be required, too. OpenSearch provides multiple mechanisms for this through Index State Management (ISM).

Configuring a policy

The first step in ISM is to set up a policy. Navigate from the left hand menu to Index management → State management policies. Create a new policy, and give it an appropriate name and a useful description.

ISM templates

Make sure newly created indexes gets the correct ISM policy by adding an index pattern, e.g. accesslog-*.

States

ISM works with states, one of which is initial, which means that indexes are set to the initial state when created. The names of the various state is set by the administrator. The names are only descriptive but it makes sense that they're logical. When working with logs, a common naming standard is hot / warm / cold / deleted, in which indexes in a hot state (the initial state) are configured for fast read/write operations while indexes in warm and cold states may be moved to slower storage, get lower redundancy, etc. For other scenarios with shorter lifespan, a setup with two states (e.g. active and deleted) might be enough.

In every state, an index can be subject to actions and transitions.

Actions

An index can be configured to perform various actions when reaching certain tresholds, e.g. age or size. This often happens in the initial state. When an index rolls over, its alias is updated. For instance, when the index named accesslog-000143 grows to 2 GB, a new index named accesslog-000144 will be created and the alias accesslog will point to the new index.

Transitions

A transition is when an index moves from one state to another.

Example policy

Each state will hold instructions for what to do with an index when transitioned to and/or staying in the given state, and what decides when and where to transition next. Below is an example of a set of states with actions and transitions:

  • Hot state (initial)
    • Action: Roll over whenever the index size reaches 2 GB. The alias is updated automatically.
    • Transition: Move to the warm state after 10 days
  • Warm state
    • Action: Shed the replicas
    • Transition: Move to the cold state after 20 days
  • Cold state
    • Action: Set index read-only
    • Transition: Move to the deleted state after 60 days
  • Deleted
    • Action: Delete the index

Applying a policy to existing indexes

If there are indexes created before the policy was configured, the policy can be retroactively applied by navigating to Index management → Indexes, selecting the appropriate index(es), clicking the Actions button (upper right) → Apply policy.

Listing policy managed indexes

Indexes managed by policies can be reviewed by navigating to Index management → Policy managed indexes. This list will show the state of every index, whether a policy action or transitioning has been run on the index, and whether it was successful.