Tuesday, 12 May 2015

Elasticsearch error - Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

After adding the new elasticsearch node, initially, I was struggling with error :

WARN: org.elasticsearch.discovery: [logstash-id.xxx.in-25379-6424] waited for 30s and no initial state was set by the discovery

and I corrected the situation by adding iptables rules.

Thereafter, things appeared to be smooth but could not find new elasticsearch index using Elasticsearch head plugin even after few minutes. So, I started searching through debug logs and spotted an error :

log4j, [2015-05-12T11:00:02.003] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
    at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
    at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
log4j, [2015-05-12T11:00:06.507] DEBUG: org.elasticsearch.discovery.zen: [logstash-id.xxx.in-26328-4264] filtered ping responses: (filter_client[true], filter_data[false]) {none}
^CInterrupt received. Shutting down the pipeline. {:level=>:warn, :file=>"logstash/agent.rb", :line=>"119"}

Since I was using latest version of logstash and elasticsearch, I was bit puzzled as all the google solutions(references) were pointing to the old versions of them:

[root@id admin]# /opt/logstash/bin/logstash --version
logstash 1.4.2-modified

[root@es2 ~]# rpm -qa|grep elastic -i
elasticsearch-1.4.4-1.noarch

Finally, after reading good documentation of logstash, I decided to add 'protocol' option and vola! - it worked!!

output {
        #stdout { codec => rubydebug }
        if [type] == "netflow" {
                elasticsearch {
                        cluster => "elk-cluster"
                        index => "netflow-%{+YYYY.MM.dd}"
                        host => "10.4.0.47"
                        protocol => "http"
                        workers => 2
                }
        }


No comments:

Post a Comment