What seems to be happening is that i’m not getting the same number of parameters (165 with old script, 210 with new).
Pastebin here:
https://pastebin.com/r5bR1rcH
Extras outputting from the script should be fine. The application poller should just be picking out the keys that it wants from an array…it likely didn’t use all 165 from the original script either.
Do you get the error if you narrow the timeframe down to very recent data? Like the last couple hours?
Can you remove the mysql app rrd files from one of the hosts to see if it works when they build fresh with all of the fields/ds’s?
Sadly narrowing the data won’t help, it’s like if all the new data is ‘corrupted’. And of course I can do that, but it’s the first time I attempt it. I’m relatively new to LNMS, but a sysadmin, so if you wouldn’t mind just telling me how, I certainly can
Actually, i found the rrd folder, so, just delete all RRDs related to app-mysql?
Moved the files away, re-discovered, and re-polled. We should see how that goes in a few.
Sadly, still ‘error drawing graphs’.
I really wonder what i’m doing wrong…
Okay, i might have an idea, but very much unsure why. You saw the raw output of the mysql script already, goes down to ‘qp’ as for values. But when I poll it through smtpwalk, i only get until ‘cr’.
See this pastebin:
https://pastebin.com/NCJVFYnn
Furthermore, the output seems to be fluctuating. Sometimes stops at ‘e2’. Sometimes ‘dh’. Sometimes ‘dy’. This makes no sense to me.
I just also tested the snmpwalk locally on the server itself, same thing. When I poll, I get wildly variable results as far as the number of variables shown through snmp-extend. However, the results of ./mysql output seems consistent, variables goes up to ‘qp’.
Just to do a blind fool test, I reverted to the original script as provided. No more inconsistencies. snmpwalk returns consistent results, and graphs are appearing (albeit all NAN). Let me test again that first script you sent originally.
With the original script you made? Consistent output through snmp-extend. Always ends up by ‘ek’, just like when it’s ran manually on the target server.
EDIT: talked too soon. it’s also variable. i’m at loss here.
That’s pretty weird. Both my 10.x and 5.5.x servers snmpget through to qp consistently. If I set a really low snmpget timeout, it’s all or nothing, either I get everything or I get a timeout error. I can’t seem to duplicate a cut off.
i’m as perplex as you are. let me sleep on that this weekend, on monday i’ll redo the whole checklist.
cheers
T
Just a little update, I left your first corrected script run for all weekend, all stats were fine, apart of course Inodb insert buffer. I now just put back the second corrected script, and I will leave it running for 48h. I get ‘error drawing graphs’ now, but we will see with time.
T
Mine have continued working as expected over the weekend. Hm.
Do you see anything that leads you believe snmp/udp packets could be getting dropped between your LibreNMS and MySQL server?
I guess an easy test would be to configure agentaddress udp:161,tcp:161 in your MySQL server’s snmpd.conf, restarting snmpd, then running that same snmpget with tcp to see if you get the full output.
Alternately, I think the file below will only output keys that the LibreNMS app currently uses.
So you may have better luck with that but it’s kind of bummer to lose the rest of the keys that someone may use to create more graphs later on.
I really don’t think snmp/udp packets are being dropped, after all i have around 80 servers monitored without a glitch on the same system. As the previous script works fine, something is very differently handled. After 12 something hours, the graphs are still completely broken (error drawing graph)
RRDTool Command
rrdtool graph /tmp/Qm1HvDoHXPyPSdtW --alt-autoscale-max --rigid -E --start 1543243200 --end 1543329600 --width 1681.2 --height 308 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal --alt-autoscale-max --rigid -E --start 1543243200 --end 1543329600 --width 1681.2 --height 308 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:‘Connections Now Min Max Avg\l’ DEF:ds0=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:AVERAGE DEF:ds0min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:MIN DEF:ds0max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:MAX DEF:ds1=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:AVERAGE DEF:ds1min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:MIN DEF:ds1max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:MAX DEF:ds2=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:AVERAGE DEF:ds2min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:MIN DEF:ds2max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:MAX DEF:ds3=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:AVERAGE DEF:ds3min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:MIN DEF:ds3max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:MAX DEF:ds4=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:AVERAGE DEF:ds4min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:MIN DEF:ds4max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:MAX DEF:ds5=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:AVERAGE DEF:ds5min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:MIN DEF:ds5max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:MAX LINE1.25:ds0#CC0000:‘Max Connection ’ GPRINT:ds0:LAST:%5.2lf%s GPRINT:ds0min:MIN:%5.2lf%s GPRINT:ds0max:MAX:%5.2lf%s GPRINT:ds0:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds1#008C00:‘Max Used Conne ’ GPRINT:ds1:LAST:%5.2lf%s GPRINT:ds1min:MIN:%5.2lf%s GPRINT:ds1max:MAX:%5.2lf%s GPRINT:ds1:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds2#4096EE:‘Aborted Client ’ GPRINT:ds2:LAST:%5.2lf%s GPRINT:ds2min:MIN:%5.2lf%s GPRINT:ds2max:MAX:%5.2lf%s GPRINT:ds2:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds3#73880A:‘Aborted Connec ’ GPRINT:ds3:LAST:%5.2lf%s GPRINT:ds3min:MIN:%5.2lf%s GPRINT:ds3max:MAX:%5.2lf%s GPRINT:ds3:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds4#D01F3C:‘Threads Connec ’ GPRINT:ds4:LAST:%5.2lf%s GPRINT:ds4min:MIN:%5.2lf%s GPRINT:ds4max:MAX:%5.2lf%s GPRINT:ds4:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds5#36393D:‘New Connection ’ GPRINT:ds5:LAST:%5.2lf%s GPRINT:ds5min:MIN:%5.2lf%s GPRINT:ds5max:MAX:%5.2lf%s GPRINT:ds5:AVERAGE:’%5.2lf%s\n’ HRULE:0#555555
RRDTool Output
ERROR: rrdc_fetch: Got 92 lines, expected 151
Note that this affects the entierty of the MySQL plugin. I remember you mentioned mk_agent, while i’m using snmp extend. Could that be the issue?
I can try your new iteration, and see how goes.
It’s been running for at least half an hour. I removed the previous rrd files, re-discovered, re-polled. Error drawing graph. However this time I always get the same output from the script through snmp, but there’s 2 missing variables from the rrdtool command:
RRDTool Command
rrdtool graph /tmp/INz404HVwUyjBggi --alt-autoscale-max --rigid -E --start 1543246200 --end 1543332600 --width 1681.2 --height 308 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal --alt-autoscale-max --rigid -E --start 1543246200 --end 1543332600 --width 1681.2 --height 308 -c BACK#EEEEEE00 -c SHADEA#EEEEEE00 -c SHADEB#EEEEEE00 -c FONT#000000 -c CANVAS#FFFFFF00 -c GRID#a5a5a5 -c MGRID#FF9999 -c FRAME#5e5e5e -c ARROW#5e5e5e -R normal --font LEGEND:8:DejaVuSansMono --font AXIS:7:DejaVuSansMono --font-render-mode normal COMMENT:‘Connections Now Min Max Avg\l’ DEF:ds0=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:AVERAGE DEF:ds0min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:MIN DEF:ds0max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MaCs:MAX DEF:ds1=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:AVERAGE DEF:ds1min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:MIN DEF:ds1max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:MUCs:MAX DEF:ds2=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:AVERAGE DEF:ds2min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:MIN DEF:ds2max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:ACs:MAX DEF:ds3=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:AVERAGE DEF:ds3min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:MIN DEF:ds3max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:AdCs:MAX DEF:ds4=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:AVERAGE DEF:ds4min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:MIN DEF:ds4max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:TCd:MAX DEF:ds5=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:AVERAGE DEF:ds5min=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:MIN DEF:ds5max=/opt/librenms/rrd/www100.votresite.ca/app-mysql-919.rrd:Cs:MAX LINE1.25:ds0#CC0000:‘Max Connection ’ GPRINT:ds0:LAST:%5.2lf%s GPRINT:ds0min:MIN:%5.2lf%s GPRINT:ds0max:MAX:%5.2lf%s GPRINT:ds0:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds1#008C00:‘Max Used Conne ’ GPRINT:ds1:LAST:%5.2lf%s GPRINT:ds1min:MIN:%5.2lf%s GPRINT:ds1max:MAX:%5.2lf%s GPRINT:ds1:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds2#4096EE:‘Aborted Client ’ GPRINT:ds2:LAST:%5.2lf%s GPRINT:ds2min:MIN:%5.2lf%s GPRINT:ds2max:MAX:%5.2lf%s GPRINT:ds2:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds3#73880A:‘Aborted Connec ’ GPRINT:ds3:LAST:%5.2lf%s GPRINT:ds3min:MIN:%5.2lf%s GPRINT:ds3max:MAX:%5.2lf%s GPRINT:ds3:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds4#D01F3C:‘Threads Connec ’ GPRINT:ds4:LAST:%5.2lf%s GPRINT:ds4min:MIN:%5.2lf%s GPRINT:ds4max:MAX:%5.2lf%s GPRINT:ds4:AVERAGE:’%5.2lf%s\n’ LINE1.25:ds5#36393D:‘New Connection ’ GPRINT:ds5:LAST:%5.2lf%s GPRINT:ds5min:MIN:%5.2lf%s GPRINT:ds5max:MAX:%5.2lf%s GPRINT:ds5:AVERAGE:’%5.2lf%s\n’ HRULE:0#555555
RRDTool Output
ERROR: rrdc_fetch: Got 149 lines, expected 151
Little update, been running for a few days, still no dice. So I guess for now, I will revert to the first corrected script you provided, and i’ll just live with the missing metrics. The rest worked fine, so no huge deal.
Thanks again