Mutualized Inputs

Objective

This guide will help you understand how to use the mutualized inputs of Logs Data Platform to ingest your logs to the platform.

Inputs are the components from Logs Data Platform that you connect to ingest your logs in the platform. In this guide, we focus specifically on the mutualized inputs that are available to everyone by default.

Prerequisites

Before reading this documentation, you should:

• Have read the introduction to Logs Data Platform
• Have read the quick start to Logs Data Platform
• Have created and configured a Logs Data Platform account
• Have created a Logs Data Platform stream

Instructions

Why different inputs?

Logs Data Platform imposes a few constraints as to how your logs are structured to guarantee an efficient and optimal indexation of the logs you are sending to us. The different inputs are responsible for enforcing those constraints regardless of the compatible format that you give to your logs and converting them to a common format before storing them in the platform.

The log formats that are accepted by Logs Data Platform are the following:

• GELF: This is the native format of logs used by Graylog. This JSON format allows you to send logs really easily. See the GELF Payload Specification. The GELF input only accepts a null (\0) delimiter.
• LTSV: This simple format is very efficient and human readable. You can learn more about it here. LTSV has two inputs that accept a line delimiter or a null delimiter.
• RFC 5424: This format is commonly used by logs utilities such as syslog. It is extensible enough to allow you to send all your data. More information about it can be found here.
• Cap'n'Proto: The most efficient log format. This is a binary format that allows you to maintain a low footprint and high speed performance. For more information, check out the official website: Cap'n'Proto.
• Beats: A secure and reliable protocol used by the beats family in the Elasticsearch ecosystem (Ex: Filebeat, Metricbeat, Winlogbeat).

Mutualized vs Dedicated inputs

This guide describes the mutualized inputs that are by default included in the platform work. Before going on with the guide, here is a reminder of the few characteristics of dedicated inputs that differentiate them from the mutualized ones:

• They are optional and charged on a per-input basis.
• You can choose which port they listen to.
• You can filter IP addresses that are allowed to send logs.
• You can choose to run Logstash or Flowgger. You can configure their sources as well as transform the logs they ingest before storing them in Logs Data Platform.

As you can see, mutualized inputs allow you more flexibility in how you handle your logs at ingestion as well as more security features. If you need any of these features, you should look at the documentation for dedicated inputs.

Ingesting Logs

There are two main ways to ingest the logs generated by your systems or applications in Logs Data Platform:

• Use logs exporter softwares that parse log files that are locally stored on your filesystem, format the logs if necessary and connect to Logs Data Platform's inputs to send the logs. You will find separate documentations for such widely-used softwares: Filebeat, syslog-ng, NXLog, and the following documentation should allow you to configure any other similar software.
• Use libraries in your own software that can directly send logs to Logs Data Platform. We have some documentation that can help you through this process if you use Python 2,Python 3 or Rust though any other library that is compatible with the OpenSearch API can be used.

Whatever the choice you make, you will have to properly configure your software or libraries to send your logs to Logs Data Platform. The following section will help you find the information relevant to that purpose.

Configuring your software

To configure your software, you need the following information:

• Input endpoint URL: the endpoint URL has the form XXX.logs.ovh.com, XXX corresponding to the cluster you are assigned to. It can be found in your OVHcloud Control Panel on the home page of your Logs Data Platform account.

• Input endpoint port: the port to which your software must connect to depends on the format of your logs, and whether or not you use a secured transport layer. The table below describes the matching between port and log format, but you will also find this information in your OVHcloud Control Panel.

• Certificate: if you use a secured transport layer, the certificates can also be found in the OVHcloud Control Panel.

• X-OVH-TOKEN: the X-OVH-token is used to dispatch the logs you ingest into Logs Data Platform to the correct log stream. You can find the X-OVH-TOKEN corresponding to your stream in the in the Data stream section of the OVHcloud Control Panel.

If your software interacts directly with the OpenSearch API, please follow this documentation.

You should now have all the information you need to configure your software to ingest logs in Logs Data Platform.

Go Further

Ingest your logs with Filebeat
Ingest your logs with syslog-ng
Ingest your logs with NXLog
Push your logs with the OpenSearch API
Python 2 SDK
Python 3 SDK
Rust SDK
Ingest the logs from your Kubernetes cluster