Fluentd: Could not push logs to Elasticsearch -
i have deployed elasticsearch 2.2.0 now, i'm sending logs using td-agent 2.3.0-0.
the final on tag chain was..
<match extra.geoip.processed5.**> type copy <store> type file path /var/log/td-agent/sp_l5 time_slice_format %y%m%d time_slice_wait 10m time_format %y%m%dt%h%m%s%z compress gzip utc </store> <store> type elasticsearch host 11.0.0.174 port 9200 logstash_format true logstash_prefix logstash_business logstash_dateformat %y.%m flush_interval 5s </store> </match> now, added time out within elasticsearch type block
request_timeout 45s this td-agent.log debug enabled.
2016-02-08 15:58:07 +0100 [info]: plugin/in_syslog.rb:176:listen: listening syslog socket on 0.0.0.0:5514 udp 2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 15:59:03 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: not push logs elasticsearch, resetting connection , trying again. read timeout reached 2016-02-08 16:03:33 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: not push logs elasticsearch, resetting connection , trying again. read timeout reached 2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 16:03:35 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: not push logs elasticsearch, resetting connection , trying again. read timeout reached 2016-02-08 16:08:05 +0100 [warn]: plugin/out_elasticsearch.rb:200:rescue in send: not push logs elasticsearch, resetting connection , trying again. read timeout reached 2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 16:08:09 +0100 [info]: plugin/out_elasticsearch.rb:77:client: connection opened elasticsearch cluster => {:host=>"11.0.0.174", :port=>9200, :scheme=>"http"} 2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed flush buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="fluent::elasticsearchoutput::connectionfailure" error="could not push logs elasticsearch after 2 retries. read timeout reached" plugin_id="object:fd9738" 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run' 2016-02-08 16:12:40 +0100 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed flush buffer. next_retry=2016-02-08 15:59:04 +0100 error_class="fluent::elasticsearchoutput::connectionfailure" error="could not push logs elasticsearch after 2 retries. read timeout reached" plugin_id="object:1034980" 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:204:in `rescue in send' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:194:in `send' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.3.0/lib/fluent/plugin/out_elasticsearch.rb:188:in `write' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:345:in `write_chunk' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/buffer.rb:324:in `pop' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:321:in `try_flush' 2016-02-08 16:12:40 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.19/lib/fluent/output.rb:140:in `run' i'm running on aws on ubuntu 14.04 c3.large. have tested access td-agent machine creating index, adding documents , deleting index using curl without problem. (to sure, opened communications in security groups)
another test td-agent machine...
root@bilbo:~# telnet 11.0.0.174 9200 trying 11.0.0.174... connected 11.0.0.174. escape character '^]'. / http/1.0 http/1.0 200 ok content-type: application/json; charset=utf-8 content-length: 320 { "name" : "gandalf-gandalf", "cluster_name" : "aaaa_dev", "version" : { "number" : "2.2.0", "build_hash" : "8ff36d139e16f8720f2947ef62c8167a888992fe", "build_timestamp" : "2016-01-27t13:32:39z", "build_snapshot" : false, "lucene_version" : "5.4.1" }, "tagline" : "you know, search" } connection closed foreign host. root@bilbo:~# with strace can see this
[pid 10774] connect(23, {sa_family=af_inet, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = -1 einprogress (operation in progress) [pid 10774] clock_gettime(clock_monotonic, {4876, 531526283}) = 0 [pid 10774] select(24, null, [23], null, {45, 0}) = 1 (out [23], left {44, 999925}) [pid 10774] fcntl(23, f_getfl) = 0x802 (flags o_rdwr|o_nonblock) [pid 10774] connect(23, {sa_family=af_inet, sin_port=htons(9200), sin_addr=inet_addr("11.0.0.174")}, 16) = 0 [pid 10774] fcntl(23, f_getfl) = 0x802 (flags o_rdwr|o_nonblock) [pid 10774] write(23, "post /_bulk http/1.1\r\nuser-agent: faraday v0.9.2\r\nhost: 11.0.0.174:9200\r\ncontent-length: 4798\r\n\r\n", 97) = 97 [pid 10774] fcntl(23, f_getfl) = 0x802 (flags o_rdwr|o_nonblock) [pid 10774] write(23, "{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"feb 8 16:18:47 bilbo ::apache::pre::access: - 10.0.0.15 - control [08/feb/2016:16:18:47 +0100] \\\"get /status/memcached.php http/1.1\\\" 200 1785 \\\"-\\\" \\\"check_http/v2.0 (monitoring-plugins 2.0)\\\" control-pre.fluzo.com:443 0\",\"n\":\"bilbo\",\"s\":\"info\",\"f\":\"local3\",\"t\":\"feb 8 16:18:47\",\"h\":\"bilbo\",\"a\":\"apache\",\"e\":\"pre\",\"o\":\"access\",\"ip\":\"-\",\"ip2\":\"10.0.0.15\",\"rl\":\"-\",\"ru\":\"control\",\"rt\":\"[08/feb/2016:16:18:47 +0100]\",\"met\":\"get\",\"pqf\":\"status/memcached.php\",\"hv\":\"http/1.1\",\"st\":\"200\",\"bs\":\"1785\",\"ref\":\"-\",\"ua\":\"check_http/v2.0 (monitoring-plugins 2.0)\",\"vh\":\"control-pre.aaaa.com\",\"p\":\"443\",\"rpt\":\"0\",\"co\":null,\"ci\":null,\"la\":null,\"lo\":null,\"ar\":null,\"dm\":null,\"re\":null,\"@timestamp\":\"2016-02-08t16:19:47+01:00\"}\n{\"index\":{\"_index\":\"logstash_apache-2016.02.08\",\"_type\":\"fluentd\"}}\n{\"message\":\"feb 8 16:18:47 bilbo ::apache::pre::access: - 10.0.0.15 - control [08/feb/2016:16:18:47 +0100] \\\"get /status/core.php http/1.1\\\" 200 1898 \\\"-\\\" \\\"c"..., 4798) = 4798 seems connection open.
to avoid confusion, have installed td-agent on elasticsearch machine same output.
this elasticsearch configuration...
### managed puppet ### --- bootstrap: mlockall: true cluster: name: aaaa0_dev discovery: zen: minimum_master_nodes: 1 ping: multicast: enabled: false unicast: hosts: - 11.0.0.174 gateway: expected_nodes: 1 recover_after_nodes: 1 recover_after_time: 5m hostname: gandalf http: compression: true index: store: compress: stored: true type: niofs network: bind_host: 11.0.0.174 publish_host: 11.0.0.174 node: name: gandalf-gandalf path: data: /var/lib/elasticsearch-gandalf logs: /var/log/elasticsearch/gandalf transport: tcp: compress: true any idea?
thanks.
update
against real elasticsearch cluster works (3 nodes)
this configuration.
### managed puppet ### --- bootstrap: mlockall: true cluster: name: aaaa discovery: zen: minimum_master_nodes: 2 ping: multicast: enabled: false unicast: hosts: - el0 - el1 - el2 gateway: expected_nodes: 3 recover_after_nodes: 2 recover_after_time: 5m hostname: kili http: compression: true index: store: compress: stored: true type: niofs network: bind_host: 11.0.0.253 publish_host: 11.0.0.253 node: name: kili-kili path: data: - /var/lib/elasticsearch0 - /var/lib/elasticsearch1 logs: /var/log/elasticsearch/kili transport: tcp: compress: true however, kibana did create .kibana index elasticsearch 1 node. also, y test node curl.
Comments
Post a Comment