Logging configuration in OpenContrail

We know that all software components and services generate log files. These log files are vital in troubleshooting and debugging problems. If the log files are not managed properly then it can  be extremely  difficult to get a good look into them.

Although system administrators cannot control the generation of logs, they can achieve  some level of log management by

• having log rotators to get rid of the old log files.
• using syslog to catch alerts.
• archiving logs etc.

OpenContrail has several components, many of which can generate logs as well as store them in the log files. OpenContrail also provides the mechanism to configure the logging, so that the system administrators / DevOps can define the logging parameters to suite their own requirements.

In this blog post we will see logging support in OpenContrail components and what are the logging configuration mechanisms supported by it.

OpenContrail uses Sandesh protocol which provides the mechanism to exchange messages between various OpenContrail components. It also provides the functionality of logging those messages and the logs into the log files. You can read more about Sandesh in this great article

Logging can be configured by :

• choosing the log file
• selecting the log file size
• defining custom formatters/loggers
• using syslog etc.

OpenContrail has mainly Python components and C++ components.

Python components of OpenContrail are :

• contrail API server
• schema transformer
• SVC monitor
• discovery server
• analytics Op server

C++ components of OpenContrail are :

•  contrail vrouter
•  contrail controller
•  Query engine
•  contrail analytics server
•  contrail DNS

C++ components of OpenContrail use log4cplus for logging and python components use python logging.

OpenContrail versions

The configuration mechanisms defined in this post are supported by the master version of OpenContrail.

You need to cherry pick the below patches if you are using R2.2 or R2.1 version as these patches are still not merged yet.

OpenContrail R2.2

https://review.opencontrail.org/#/c/11106/

OpenContrail R2.1

https://review.opencontrail.org/#/c/11116/

https://review.opencontrail.org/#/c/11105/

Logging in OpenContrail python modules

First we will talk about logging in python components of OpenContrail. OpenContrail supports logging configuration for python components in three ways:

1. Use the default logging provided by OpenContrail.
2. Define your own log configuration file based on the python logging
3. Define new logging mechanism by implementing a new logger or using other logging libraries like oslo.log

The configuration files of python components support the below logging parameters:

• log_file
• log_level
• log_local
• logging_conf
• logger_class

In order to define custom logging configuration, we need to use the ‘logging_conf’ and ‘logger_class’ parameters. When these two parameters are defined, the other ones are ignored.

1. Use the default logging provided by OpenContrail.

You don’t have to do anything here. If you are not particular about logging configuration, then this is good enough.

2. Define your own log configuration file based on the python logging

You can define your own log configuration file. Please refer to the logging file format for more information on how to define the log config file for pythin logging.

Define the ‘logger_class’ and ‘logging_conf’ configuration parameters in the OpenContrail python component configuration files.

logger_class = pysandesh.sandesh_logger.SandeshConfigLogger
logging_conf = PATH TO THE LOG CONFIG FILE

Eg.

contrail-api.conf

[DEFAULT]
...
logger_class = pysandesh.sandesh_logger.SandeshConfigLogger
logging_conf = /etc/contrail/contrail-api-logger.conf

Format of the log configuration file

As mentioned above this has all the details about defining the log configuration file.

Log configuration file should have three main sections defined – [loggers],[handlers] and [formatters].

Below is a sample log configuration file format. This sample file can be used for all the OpenContrail python components. You can define one configuration file per module as well.

[loggers]
keys=root,contrail_api,contrail_svc_monitor,contrail_schema,contrail_discovery_handler,contrail_analytics_api

[handlers]
keys=root_handler,contrail_api_handler,contrail_syslog_handler,contrail_svc_handler,contrail_schema_handler,contrail_analytics_handler

[formatters]
keys=contrail_formatter,contrail_syslog_formatter,svc_formatter

[logger_root]
level=NOTSET
handlers=root_handler

[logger_contrail_api]
level=NOTSET
handlers=contrail_api_handler,contrail_syslog_handler
qualname=contrail-api
propagate=0

[logger_contrail_svc_monitor]
level=NOTSET
handlers=contrail_svc_handler,contrail_syslog_handler
qualname=contrail-svc-monitor
propagate=0

[logger_contrail_schema]
level=NOTSET
handlers=contrail_schema_handler,contrail_syslog_handler
qualname=contrail-schema
propagate=0

[logger_contrail_discovery]
level=NOTSET
handlers=contrail_discovery_handler,contrail_syslog_handler
qualname=contrail-discovery
propagate=1

[handler_root_handler]
class=StreamHandler
level=NOTSET
formatter=contrail_formatter
args=()

[handler_contrail_api_handler]
class=handlers.RotatingFileHandler
args=('/var/log/contrail/contrail-api.log', 'a', 3000000, 10)
formatter=contrail_formatter

[handler_contrail_svc_handler]
class=handlers.RotatingFileHandler
args=('/var/log/contrail/svc-monitor.log', 'a', 3000000, 8)
formatter=svc_formatter

[handler_contrail_schema_handler]
class=handlers.RotatingFileHandler
args=('/var/log/contrail/contrail-schema.log', 'a', 2000000, 7)
formatter=contrail_formatter

[handler_contrail_discovery_handler]
class=handlers.RotatingFileHandler
args=('/var/log/contrail/contrail-discovery-conf.log', 'a', 3000000, 0)
formatter=contrail_formatter

[handler_contrail_analytics_handler]
class=handlers.RotatingFileHandler
args=('/var/log/contrail/contrail-analytics.log', 'a', 3000000, 0)
formatter=contrail_formatter


[handler_contrail_syslog_handler]
class=handlers.SysLogHandler
level=ERROR
formatter=contrail_syslog_formatter
args=('/dev/log', handlers.SysLogHandler.LOG_USER)

[formatter_contrail_formatter]
format= %(asctime)s [%(name)s]: %(message)s
datefmt=%m/%d/%Y %I:%M:%S %p
class=logging.Formatter

[formatter_contrail_syslog_formatter]
format=contrail : %(asctime)s [%(name)s]: %(message)s
datefmt=%m/%d/%Y %I:%M:%S %p
class=logging.Formatter

[formatter_svc_formatter]
format=SVC MON %(asctime)s [%(name)s]: %(message)s
datefmt=%m/%d/%Y %I:%M:%S %p
class=logging.Formatter

As you can see above, a logger is defined for each of the OpenContrail components.

[logger_contrail_api]
level=NOTSET
handlers=contrail_api_handler,contrail_syslog_handler
qualname=contrail-api
propagate=0

‘qualname’ should match the OpenContrail component name, otherwise the logger defined for the OpenContrail component would not get reflected.

Below are the ‘qualname’ for each of the OpenContrail components.

Defining your own logging configuration file gives you the flexibility to choose the logging parameters as per your requirements.
You can choose the logging handlers supported by the python logging like RotatingFileHandler, TimedRotatingFileHandler, WatchedFileHandler, MemoryHandler etc.

You can also choose a simple handler like FileHander and use logrotate or other external log rotaters to rotate the log files.

3. Define your own custom logging mechanism or use existing logging libraries.

If you’r someone who likes to define your new logging mechanism, this can also be done.

In order to do this you need to first:

• write your custom logging class
• define the custom logging class in the ‘logger_class’ configuration parameter.

Make sure that your custom python class is loadable. Your custom python class should be derived from ‘sandesh_base_logger.SandeshBaseLogger’.

Contrail Oslo Logger

You can find one custom logger – Contrail Oslo Logger here. Contrail Oslo logger uses the oslo.log and oslo.config modules of OpenStack.

You can define the log configuration options supported by oslo.log in a configuration file and provide the name of the file in the ‘logging_conf’ configuration parameter.

You can find the logging options supported by oslo.log here and here.

If you would like to have your own logging mechanism please see the code of contrail oslo logger as reference.

Logging in OpenContrail C++ components

OpenContrail C++ components use log4cplus  for logging.

OpenContrail supports the below logging parameters in the component configuration files :

• log_disable : Disable logging
• log_file    : Name of the log file
• log_property_file : Path of the log property file.
• log_files_count : Maximum log file roll over index
• log_file_size  : Maximum size of the log file
• log_level  : Severity level for local logging of sandesh messages

Similar to the python logging configuration file, you can define a log configuration file for the C++ components and give the path of the configuration file in the ‘log_property_file’ configuration parameter. When ‘log_property_file’ is defined, other logging parameters are ignored by the OpenContrail C++ components. log4cplus uses the term property file for the log configuration file.

The log property file should be defined in the format described here.

You can refer to this link to understand the format of the log4cplus log property file.

Define ‘log_property_file’ in the DEFAULT section of the C++ component configuration files to use the log property file defined by you.

Eg. contrail-control.conf

[DEFAULT]
log_property_file=/etc/contrail/control-log.properties

Sample lop property file

log4cplus.rootLogger = DEBUG, logfile, syslog

log4cplus.appender.logfile = log4cplus::FileAppender
log4cplus.appender.logfile.File = /var/log/contrail/contrail-collector.log
log4cplus.appender.logfile.Append = true
log4cplus.appender.logfile.ImmediateFlush = true

log4cplus.appender.logfile.layout = log4cplus::PatternLayout

log4cplus.appender.logfile.layout.ConversionPattern = %D{%Y-%m-%d %a %H:%M:%S:%Q %Z} %h [Thread %t, Pid %i]: %m%n

log4cplus.appender.syslog = log4cplus::SysLogAppender
log4cplus.appender.syslog.Threshold=ERROR
log4cplus.appender.syslog.FACILITY=USER
log4cplus.appender.syslog.layout = log4cplus::PatternLayout
log4cplus.appender.syslog.layout.ConversionPattern = %D{%Y-%m-%d %a %H:%M:%S:%Q %Z} %h [Thread %t, Pid %i]: %m%n

You can refer to the Appenders supported by log4cplus here.

Conclusion

You’ve now hopefully seen how logging is supported in OpenContrail and how you can define your own custom logging configuration files. With this knowledge, it should be possible for system admins/DevOps to manage the log files properly and help them quickly and efficiently troubleshoot problems.