Configuring log shippers

Log shippers are used to feed content into your OpenSearch instance. This article shows a simple example on how to use Vector for log shipping, including client-side data structuring. More examples can be found in the Vector documentation.

Note: While Vector can be used for all sorts of messaging and events, this article will show how to set up reading, parsing, and shipping logs from a log file.

The basics

Vector use three consepts for its message flow:

• Sources: defines which logs are fed into Vector
• Transforms: parses, structures, and reformats the logs
• Sinks: defines where the logs should be forwarded

A simple pipeline

The source

The following source definition (YAML format) makes Vector read from NGINX access log files. In this case, the source is a file type. Other source types are listed in Vector's documentation.

---
sources:
  source_nginx_access_log:
    type: file
    include:
      - /var/log/nginx/*-access_log
    read_from: beginning

The transform

The following transform (YAML format) parses and structures incoming NGINX logs with the powerful remap transform, using the parse_nginx_log built-in function. Afterwards, it renames some of the fields so they map to the Elastic Common Scheme (ECS) naming standard.

---
transforms:
  transform_nginx_access_log:
    type: remap
    inputs:
      - source_nginx_access_log
    source: |-
      .log.file.path = del(.file)

      tmp,err = parse_nginx_log(.message, "combined", "%d/%b/%Y:%T %z")
      if err == null {
        . = merge(., tmp)
        del(.message)
      } else {
        log("parse_nginx_log failed", level: "error")
      }

      # Mapping to ECS
      .client.ip = del(.client)
      .user_agent = parse_user_agent!(del(.agent))
      .user.name = del(.user)
      .http.response.status_code = del(.status)
      .http.request.referrer = del(.referer)
      .nginx.request = del(.request)

Other transforms are listed in Vector's documentation.

The sink

The following sink definition sends the logs to an OpenSearch instance, using the sink type elasticsearch. Other sink types are listed in Vector's documentation.

---
sinks:
  sink_opensearch:
    type: elasticsearch
    inputs:
      - transform_nginx_access_log
    api_version: v8
    compression: none
    doc_type: _doc
    endpoints:
      - https://failover-abcd1234.elastic-aas.h.bitbit.net:9200
    mode: bulk
    opensearch_service_type: managed
    auth:
      strategy: "basic"
      user: logpusher
      password: YourSecretPassword
    distribution:
      retry_max_duration_secs: 300
    bulk:
      index: "accesslog"

A note on Vector's configuration files and directory structure

Sources, transforms, and sinks can be defined as shown above in one or more configuration files.

Vector also supports a structured configuration directory approach, where a directory will have the subdirectories sources/, transforms/, and sinks/. Within these subdirectories, the various components can be defined using either json, yaml, or toml, based on the file suffix. The filename (without the suffix) sets the component name.

Based on this, the definitions above can be saved into individual files as shown below. Note that the two first lines of each definition will then not be necessary, since the path and filename will decide the function and name.

/
`- etc/
   `- vector/
      `- configs/
         |- sources/
         |  `- source_nginx_access_log.yaml
         |- transforms/
         |  `- transform_nginx_access_log.yaml
         `- sinks/
            `- sink_opensearch.yaml

Running Vector

Vector is rather strict when it comes to configuration and typecasting, and incomplete and erroneous configurations will make it fail. Testing the pipelines is always recommended before deploying new configurations.

By default, Vector will also check whether each sink can be considered healthy before starting to submit data to it. If you haven't added your log source to the ISaaS instance's ACL yet, this would be a good time to do so.

Debugging Vector

If Vector is configured with API access, the pipelines can be monitored with the command

# vector tap

See this article on how to enable and use this powerful debugging and monitoring tool.

Lift-off

Assuming everything works flawlessly, you should now be seeing data flow into your ISaaS instance. The logical next move will be configuring index templates, aliases, and data retention.