https://zhangdapeng.dev/tags/logstash/Recent content in Logstash on Dapeng ZhangHugo -- gohugo.ioen-usMon, 21 Jul 2025 18:16:35 +0800https://zhangdapeng.dev/p/lek-logstash-elasticsearch-kibana-gu-zhang-yi/Mon, 21 Jul 2025 18:16:35 +0800https://zhangdapeng.dev/p/lek-logstash-elasticsearch-kibana-gu-zhang-yi/<p>今早遇到如下几个问题：</p> <blockquote> <p>&ldquo;Courier Fetch: 30 of 60 shards failed.&rdquo;</p> <p>&ldquo;Data too large, data for [time] would be larger than limit&rdquo;</p> </blockquote> <p>经过Stackoverflow查看到如下几个方案</p> <p>针对第一个问题，执行如下的请求，大概的意思就是说，清理缓存的意思吧，试了一下果然有效了。</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl --user es_admin:qeeniao1234 http://10.132.14.58:9200/logstash-*/_cache/clear </span></span></code></pre></div><p>但是这个问题并没有解决我如下的问题，查看elasticsearch.log仍然会有第二个问题的提示，</p> <p>根据Stackoverflow的查找，做了下面的请求操作。打开Kibana就没有对应类似的提示了。</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl --user es_admin:qeeniao1234 -XPUT http://10.132.14.58:9200/_cluster/settings -d <span style="color:#e6db74">&#39;{ </span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> &#34;persistent&#34; : { </span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> &#34;indices.breaker.fielddata.limit&#34; : &#34;75%&#34; </span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> } </span></span></span><span style="display:flex;"><span><span style="color:#e6db74">}&#39;</span> </span></span></code></pre></div><p>原因就是elasticsearch默认启动是给了indices.breaker.fielddata.limit的值为60%，需要设置大一点的值，</p> <p>于是我就给了75%。</p> <p>经过我的再次查看，发现在使用kabana的时候尽量的选择时间区间范围小一点，这个应该是跟服务器的硬件设备有很大关系的，</p> <p>一旦你查看的数据太大了，那么对应的Fielddata的数据量就对应的增加。</p> <p>像我这里：rt</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#f92672">[</span>FIELDDATA<span style="color:#f92672">]</span> Data too large, data <span style="color:#66d9ef">for</span> <span style="color:#f92672">[</span>rt<span style="color:#f92672">]</span> would be larger than limit of <span style="color:#f92672">[</span>422523699/402.9mb<span style="color:#f92672">]</span> </span></span></code></pre></div><p>在查看过去24小时的时候，它的数据量达到了402M,可想而之其他fielddata放在一起会有多大。</p>