Tuesday, November 25, 2014

Using ELK's JMX Plugin to gather JBoss EAP metrics

In this short blog post I outline how to use the ELK (ElasticSearch/LogStash/Kibana) stack to gather data via JMX from JBoss EAP 6.

Ingredients:

- ElasticSearch 1.4.0
- Logstash 1.4.2
- Kibana 3.1.2
- JBoss EAP 6.2

After having gone throught the downloading and installation process first we need to extract some libs out of the modules for JBoss EAP 6.2 that are needed for getting remote JMX calls to work.

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib create a file jboss_remoting.rb

module JMX
  module JBoss
    java_import java.lang.System

    class JBossRemoting

      sep = System.get_property 'file.separator'
      home = System.get_property 'jboss.home'

      if home != nil
        modulebase = [home, 'modules','system','layers','base','org','jboss'].join(sep)

        libs = [
          ['remoting-jmx','main'],
          ['remoting3','main'],
          ['logging','main'],
          ['xnio','main'],
          ['xnio','nio','main'],
          ['sasl','main'],
          ['marshalling','main'],
          ['marshalling','river','main'],
          ['as','cli','main'],
          ['staxmapper','main'],
          ['as','protocol','main'],
          ['dmr','main'],
          ['as','controller-client','main'],
          ['threads','main']
        ]

        for index in 0 ... libs.size
          Dir[modulebase + sep + libs[index].join(sep) + sep + "*.jar"].each {|file| require file }
        end
      end
    end
  end
end

In elastic-search-1.4.0/vendor/bundle/jruby/1.9/gems/jmx4r-0.1.4/lib/jmx4r.rb after

    require 'jruby'

            add
    require 'jboss_remoting'
    In logstash-1.4.2/config/logstash.conf I created the following configuration

    input {
    jmx{
        path => "logstash-1.4.2/jmx"
        polling_frequency => 30
        type => "jmx"
        nb_thread => 4
      }
    }

    output {
      elasticsearch { host => localhost }
    }

    and in logstash-1.4.2/jmx/jmx.conf I configured

    {
      // JBoss Remoting JMX URL
      "url" : "service:jmx:remoting-jmx://localhost:9999",
      //username to connect to jmx
      "username" : "<your-user>",
      //password to connect to jmx
      "password": "<your-password>",
      "alias" : "jmx.instance1.elasticsearch",
      //List of JMX metrics to retrieve
      "queries" : [
        {
          "object_name" : "java.lang:type=Memory",
          "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
          "object_alias" : "Memory"
        }, {
          "object_name" : "java.lang:type=Runtime",
          "attributes" : [ "Uptime", "StartTime" ],
          "object_alias" : "Runtime"
        }, {
          "object_name" : "java.lang:type=Threading",
          "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
          "object_alias" : "Threading"
        }, {
          "object_name" : "java.lang:type=OperatingSystem",
          "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
          "object_alias" : "OperatingSystem"
        } ]
    }

    As the original JMX input plugin only supports configuration via host and port you need to apply the pull request 141 (https://github.com/elasticsearch/logstash-contrib/pull/141) to your local installation, i.e. by downloading https://raw.githubusercontent.com/jcordes73/logstash-contrib/master/lib/logstash/inputs/jmx.rb and save it to logstash-1.4.2/lib/logstash/inputs

    Now you need to setup a JBoss management user via jboss-eap-6.2/bin/add-user.sh to reflect the username and password defined in jmx.conf above.

    Because of a change in behaviour in regards to security you need to add the following configuration to elasticsearch-1.4.0/config/elasticsearch.yml

    http.cors.enabled: true
    http.cors.allow-origin: "*"
    and also change http.enabled to true.

    We need to set some environment variables so that the JBoss Remoting libraries can be found from an existing JBoss EAP installation:

    JBOSS_HOME=/opt/jboss-eap-6.2
    export JBOSS_HOME
    JAVA_OPTS="-Djboss.home=$JBOSS_HOME"
    export JAVA_OPTS
    Now you can start ElasticSearch via elasticsearch-1.4.0/bin/elasticsearch and logstash via logstash-1.4.2/bin/logstash -f logstash-1.4.2/config/logstash.conf.

    To finally see your JMX data via Kibana, open kibana-3.1.2/index.html with your favourite web-browser.

    Tuesday, November 11, 2014

    Hawtio authentication with LDAP on JBoss Fuse

    Finally here is the second part on Hawtio authentication with LDAP, this time on JBoss Fuse / A-MQ 6.1

    • Create a file named ldap-auth.xml and copy it into the deploy folder (you need to adjust the LDAP settings according to your structure)
    <?xml version="1.0" encoding="UTF-8"?>
    <blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:jaas="http://karaf.apache.org/xmlns/jaas/v1.0.0" xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0" xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0">
    <ext:property-placeholder placeholder-prefix="${" placeholder-suffix="}"/>
    <jaas:config name="karaf" rank="2">
    <jaas:module className="org.apache.karaf.jaas.modules.ldap.LDAPLoginModule" flags="required">
    connection.url=ldap://<LDAP-IP>:389
    connection.username=cn=Manager,dc=redhat,dc=com
    connection.password=redhat
    user.base.dn=ou=User,ou=ActiveMQ,dc=activemq,dc=redhat,dc=com
    user.filter=(uid=%u)
    user.search.subtree=true
    role.base.dn=ou=User,ou=ActiveMQ,dc=activemq,dc=redhat,dc=com
    role.filter=(uid=%u)
    role.name.attribute=uid
    role.search.subtree=true
    authentication=simple
    </jaas:module>
    </jaas:config>
    </blueprint> 
     (see also https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.1/html-single/Security_Guide/index.html#JAASAuth-LDAPLoginModule)

    •  In etc/system.properties add
    hawtio.authenticationEnabled=true
    hawtio.realm=karaf
    hawtio.role=admin
    hawtio.rolePrincipalClasses=org.apache.karaf.jaas.boot.principal.RolePrincipal,org.apache.karaf.jaas.modules.RolePrincipal,org.apache.karaf.jaas.boot.principal.GroupPrincipal

    (see https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Fuse/6.1/html-single/Security_Guide/#WebConsole))

    HornetQ-ActiveMQ Bridge

    In case you need a messaging bridge between HornetQ on JBoss EAP and ActiveMQ on JBoss Fuse / A-MQ then the projects on Github at https://github.com/jcordes73/messaging-bridges maybe what you are after.

    There I'm featuring

    • JBoss Fuse / A-MQ 6.1
    • JBoss EAP 6.x (at least 6.1.1 required)
    • ActiveMQ on JBoss Fuse / A-MQ
    • HornetQ on JBoss EAP 6
    • ActiveMQ RAR Adapter on JBoss EAP 6
    • SSL for JBoss Fuse / A-MQ <--> JBoss EAP communication
    • XA
    The bridge comes in two flavours

    • Running on JBoss EAP
    • Running on JBoss Fuse / A-MQ