Send data from Tremor to Axiom

This step-by-step guide helps you configure Tremor connectors and events components to interact with your databases, APIs, and ingest data from these sources into Axiom.

Axiom provides a unique way of ingesting Tremor logs into Axiom. With your connector definitions, you can configure Tremor connectors and events components to interact with your external systems, such as databases, message queues, or APIs, and eventually ingest data from these sources into Axiom.

Installation

Install the latest package from the runtime releases tag on your local machine.

Configuration using HTTP

To send logs via Tremor to Axiom, you need to create a configuration file. For example, create axiom-http.troy with the following content (using a file as example data source):

define flow client_sink_only
flow
  use std::time::nanos;
  use tremor::pipelines;

  define connector input from file
  args
    file = "in.json"  # Default input file is 'in.json' in current working directory
  with
    codec = "json",   # Data is JSON encoded
    preprocessors = ["separate"],   # Data is newline separated
    config = {
        "path": args.file,
        "mode": "read"
    },
  end;
  create connector input;

  define connector http_client from http_client
  args
    dataset,
    token
  with
    config = {
      "url": "https://AXIOM_DOMAIN/v1/ingest/#{args.dataset}",
      "tls": true,
      "method": "POST",
      "headers": {
        "Authorization": "Bearer #{args.token}"
      },
      "timeout": nanos::from_seconds(10),
      "mime_mapping": {
        "*/*": {"name": "json"},
      }
    }
  end;
  create connector http_client
  with
    dataset = "DATASET_NAME",
    token = "API_TOKEN"
  end;

  create pipeline passthrough from pipelines::passthrough;

  connect /connector/input to /pipeline/passthrough;
  connect /pipeline/passthrough to /connector/http_client;

end;

deploy flow client_sink_only;

This assumes you have set TREMOR_PATH in your environment pointing to tremor-runtime/tremor-script/lib if you are using a src clone then you can execute it as follows tremor server run axiom-http.troy

Replace API_TOKEN with the Axiom API token you have generated. For added security, store the API token in an environment variable.

Replace DATASET_NAME with the name of the Axiom dataset where you send your data.

Replace AXIOM_DOMAIN with the base domain of your edge deployment. For more information, see Edge deployments.

Configuration using Syslog

You can also send logs via Tremor to the Syslog endpoint using a file as an example data source.

1. Click Settings > Endpoints.
2. Click New endpoint.
3. Click .
4. Name the endpoint.
5. Select the dataset where you want to send data.
6. Copy the URL displayed for the newly created endpoint. This is the target URL where you send the data.

In the code below, replace url with the URL of your Syslog endpoint.

define flow client_sink_only
flow
  use std::time::nanos;
  use tremor::pipelines;

  define connector input from file
  args
    file = "in.json"  # Default input file is 'in.json' in current working directory
  with
    codec = "json",    # Data is JSON encoded
    preprocessors = ["separate"],    # Data is newline separated
    config = {
        "path": args.file,
        "mode": "read"
    },
  end;
  create connector input;

define connector syslog_forwarder from tcp_client
args
  endpoint_hostport,
with
    tls = true,
    codec = "syslog",
    config = {
      "url": "#{args.endpoint_hostport}",
      "no_delay": false,
      "buf_size": 1024,
    },
    reconnect = {
      "retry": {
        "interval_ms": 100,
        "growth_rate": 2,
        "max_retries": 3,
      }
    }
 end;
  create connector syslog_forwarder
  with
    endpoint_hostport = "tcp+tls://testsyslog.syslog.axiom.co:6514"
  end;

  create pipeline passthrough from pipelines::passthrough;

  connect /connector/input to /pipeline/passthrough;
  connect /pipeline/passthrough to /connector/syslog_forwarder;

end;

deploy flow client_sink_only;