How to redirect JBoss / Wildfly logs to Logstash using syslog

Logstash is a fantastic product to parse and process logs and events from other systems. You can store the resulting data in ElasticSearch and later visualize and analyze the results with Kibana. The combination of the three is called ELK and it produces a really powerful way of working with data, from processing and storing it to its visualization in many forms.

If you work with a web application deployed in JBoss or Wildfly (I’ll cover in this article Wildfly 8.2.0) you can send your all your logs or only the ones coming from your application to Logstash, for further processing. You can use it for example to visualize in Kibana how many errors your application threw, or how many times a specific event was triggered.

• Logstash configuration
• Wildfly / JBoss configuration

There are several ways of connecting this web application server to logstash but I’ll focus in one of the simplest: through syslog channel. Syslog is a standard to format log messages and send them through a TCP port. Both logstash and JBoss can handle this way of logging so it’s pretty easy to connect them.

Logstash configuration

I’ll assume at this point that you have your ElasticSearch and Logstash installations already configured (in my case in localhost). After that, the only thing you need to do to make logstash listen for syslog messages is to run:

./bin/logstash -e 'input { syslog { port => 615 } } output { elasticsearch { host => localhost } }'

As you can see it’s as easy as configuring the syslog input. In my case I’ve chosen a custom port to avoid colliding with any other syslog channel in the system. You can configure the output to do whatever you want, in my case it’s simply storing the messages in ElasticSearch.

Wildfly / JBoss configuration

Now you need to configure the other side, the generator of events which is in this case the application server. In order to do that you need to configure your standalone.xml file, create a handler for syslog and add this handler to your logger.

First open the file [wildfly_root]/standalone/configuration/standalone.xml and go to the specific section for logging:

<profile>
  <subsystem xmlns="urn:jboss:domain:logging:2.0">

    <console-handler name="CONSOLE">
      <level name="INFO"/>
      <formatter>
        <named-formatter name="COLOR-PATTERN"/>
      </formatter>
    </console-handler>

    <periodic-rotating-file-handler name="FILE" autoflush="true">
      <formatter>
        <named-formatter name="PATTERN"/>
      </formatter>
      <file relative-to="jboss.server.log.dir" path="server.log"/>
      <suffix value=".yyyy-MM-dd"/>
      <append value="true"/>
    </periodic-rotating-file-handler>

  <syslog-handler name="SYSLOG" enabled="true">
    <level name="INFO"/>
    <hostname value="127.0.0.1"/>
    <port value="615"/>
    <app-name value="my-app-name"/>
    <formatter><syslog-format syslog-type="RFC3164"/></formatter>
  </syslog-handler>

  <logger category="com.arjuna">
    <level name="WARN"/>
  </logger>
  <logger category="org.apache.tomcat.util.modeler">
    <level name="WARN"/>
  </logger>
  <logger category="org.jboss.as.config">
    <level name="DEBUG"/>
  </logger>
  <logger category="sun.rmi">
    <level name="WARN"/>
  </logger>
  <logger category="jacorb">
    <level name="WARN"/>
  </logger>
  <logger category="jacorb.config">
    <level name="ERROR"/>
  </logger>

  <logger category="my.app.package">
    <level name="INFO"/>
    <handlers>
      <handler name="CONSOLE"/>
      <handler name="FILE"/>
      <handler name="SYSLOG"/>
    </handlers>
  </logger>

  <root-logger>
    <level name="INFO"/>
    <handlers>
      <handler name="CONSOLE"/>
      <handler name="FILE"/>
    </handlers>
  </root-logger>
  ...
</subsystem>

As you can see in the highlighted sections all you need is to configure a syslog-handler pointing to the same host and port where you have logstash listening, and later configure a logger that uses the handler. I’m assuming here that you are using the JBoss logging implementation to handle your application logs, if you’re overriding it with log4j or any other implementation that won’t work.

You need to set up the logger category to point to your application root package (category="my.app.package"), or feel free to configure as many loggers you want with the severity options you prefer.

That’s all! Start your application server with your application deployed and your log messages will go to ElasticSearch. I recommend you to play a little with logstash filters if you want to select the messages you want to store.