Pushing logs with a SDK - Python 3.x - logging-ldp

Knowledge Base

KB0055767

Pushing logs with a SDK - Python 3.x - logging-ldp


Icons/System/eye-open Created with Sketch. 233 Views 16.01.2023 Cloud / Logs Data Platform

Objective

This guide will show you how to push your logs to Logs Data Platform using Python 3.x.

logging-ldp is intended to be a high performance logging formatter and handler to send log entries into Logs Data Platform.

This package includes:

  • a TCP/TLS handler to send log entries over TCP with TLS support.
  • a formatter to convert logging record into GELF(1.1).
  • a facility to ensure fields suits the LDP naming conventions.

Requirements

To complete this guide you will need:

Instructions

Install

Using pip

You can use pip to install logging-ldp, make sure you have the latest version:

$ pip3 install --upgrade pip
[...]
Successfully installed pip-<version>
$ pip3 install --upgrade logging-ldp
[...]
Successfully installed logging-ldp-<version> setuptools-18.3.1

Using sources

logging-ldp is available on the OVH github repository and can be installed manually:

$ git clone git@github.com:ovh/python-logging-ldp.git
Cloning into 'python-logging-ldp'...
remote: Counting objects: 58, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 58 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (58/58), 9.62 KiB | 0 bytes/s, done.
Resolving deltas: 100% (26/26), done.
Checking connectivity... done.

$ cd python-logging-ldp
$ python3 setup.py install
[...]
Using /usr/lib/python3.x/site-packages
Finished processing dependencies for logging-ldp==<version>

How to send logs

The following example shows how to send log in Graylog TCP input:

import logging
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

Send additional static meta data

To automatically append meta data on all your logs, you can implement an alternate Schema:

import logging
from marshmallow import fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

def setup_logging():
    # Load you config there
    config = dict(name="myapp", version="0.0.1")

    # Define a custom Schema
    class MyApp(LDPSchema):
        app_name = fields.Constant(config['name'])
        app_version = fields.Constant(config['version'])

    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", schema=MyApp))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    logging.info("Test !")

The log entry sent to Graylog will be something like:

{
  "_app_name": "myapp",
  "_app_version": "0.0.1",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 36,
  "short_message": "Test !",
  "timestamp": 1556036745.6493497,
  "version": "1.1"
}

Note: The result is "pretty printed" only for the documentation.

Send additional intermittent meta data

To define occasional meta data, you can define a Schema with Nested sub-items:

import logging
from marshmallow import Schema, fields
from logging_ldp.formatters import LDPGELFFormatter
from logging_ldp.handlers import LDPGELFTCPSocketHandler
from logging_ldp.schemas import LDPSchema

# Define a sub-schema
class User(Schema):
    name = fields.String(required=True)
    age = fields.Integer()

# Define a custom schema to apply on log entries
class AccountInfo(LDPSchema):
    user = fields.Nested(User, required=True)
    manager = fields.Nested(User)

def setup_logging():
    handler = LDPGELFTCPSocketHandler(hostname="gra1.logs.ovh.com")
    handler.setFormatter(LDPGELFFormatter(
        token="XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        schema=AccountInfo
    ))
    logging.getLogger().addHandler(handler)
    logging.getLogger().setLevel(logging.INFO)

if __name__ == '__main__':
    setup_logging()

    current_user = dict(name="John Doe")
    boss = dict(name="Roger Smith", age=51)
    logging.info("User has logged", extra=dict(user=current_user, manager=boss))

The log entry sent will be:

{
  "_manager_age_int": 51,
  "_manager_name": "Roger Smith",
  "_user_name": "John Doe",
  "_X-OVH-TOKEN": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "file": "test_thread.py",
  "host": "cdumay-desk",
  "level": 6,
  "line": 40,
  "short_message": "User has logged",
  "timestamp": 1556037587.4444475,
  "version": "1.1"
}

As we can see:

  • Objects are transformed to flatten dictionaries: manager.name is renamed manager_name.
  • Fields are types using the LDP naming convention: manager.age is renamed manager_age_int
  • Null values are not sent: user.age.
  • You can set required=True to force meta data value or default=xxx to add data automatically.

Go further

Related articles