Poller crashing on some printers

Help
1

Hi All,

For a while now, a few days at least, I’ve been getting the alert that some devices have not polled successfully. When I look into this in “unpolled devices” it is always printers - some desktop printers and some photocopiers, and it seems that the poller process is crashing towards the end.

The problems seem to be a little sporadic and don’t show for all printers or even the same printer every time. Out of 130 printers it seems to affect about 6 printers in any given polling session.

Here are some logs as requested:

librenms@ubuntu-srv:~$ ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS  | 21.3.0-44-g422d1195c
DB Schema | 2021_02_09_122930_migrate_to_utf8mb4 (205)
PHP       | 7.4.3
Python    | 3.8.5
MySQL     | 10.3.25-MariaDB-0ubuntu0.20.04.1
RRDTool   | 1.7.2
SNMP      | NET-SNMP 5.8
====================================

[OK]    Composer Version: 2.0.11
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct

Here are the logs for a Toshiba photo copier/printer/scanner:

Discovery log: https://p.libren.ms/view/a49b48e6

Poller log with error: https://p.libren.ms/view/a4162c16

The interesting bit is right at the end:

Checking toner Drawer 2... 0 %
Checking toner Drawer 3... 19 %
 
In Grammar.php line 136:
                                                                                                                                                                                                                         
  Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886

Here is a Lexmark desktop printer exhibiting the same issue:

Discovery log: https://p.libren.ms/view/4acfb469

Poller log with error: https://p.libren.ms/view/96717b41

Seems to be the same error:

Checking toner Maintenance Kit... 100 %
Checking toner Tray 1... 0 %
 
In Grammar.php line 136:
                                                                                                                                                                                                                         
  Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886

No firmware changes have happened with any of the printers that I’m aware of and I don’t recall seeing this problem for long - maybe a few days to a week or so.

The server is Ubuntu Server 20.04.2 LTS and gets Ubuntu updates applied automatically (unattended updates) and also LibreNMS updates automatically.

1 Like
2

Hi,
I noticed the same here on one of my printer yesterday. The polling did not finish. Was always in the “unpolled device” list. I manually discovered/polled then so it worked again. I have no clue what was causing this.

3

Manually polling the device sometimes helps for me too, and removes it from the unpolled devices list - but only temporarily, as the problems soon come back and a few printers end up in the unpolled devices list again.

This is because the error is somewhat intermittent - sometimes I can poll a given printer and get the error, then 20 minutes later poll the same printer and the poll might go through without crashing.

I suspect something about the specific data being returned causes a parsing error - so if a particular SNMP field returns a specific value an error occurs, if it returns a slightly different value it goes through. (Although the data is probably still being processed incorrectly despite not crashing in those cases)

We have ~130 printers of the types that are generating these errors, approx 5 will be in the unpolled devices list at any given time however it is not always the same devices - they rotate in and out of the list as the intermittent/random failures occur.

If you only have a handful of printers you might not see the problem very often.

4

@DBMandrake could you please run this and then upload the content of /tmp/printer.snmprec ?

./scripts/collect-snmp-data.php -h 164 -f /tmp/printer.snmprec
5

Hi,

That script doesn’t seem to be happy with the arguments provided:

librenms@ubuntu-srv:~$ pwd
/opt/librenms
librenms@ubuntu-srv:~$ ./scripts/collect-snmp-data.php -h 164 -f /tmp/printer.snmprec
Script to collect snmp data from devices to be used for testing.
Snmp data is saved in tests/snmpsim.

Usage:
  You must specify an existing device to collect data from.

Required:
  -h, --hostname     ID, IP, or hostname of the device to collect data from
  -v, --variant      The variant of the OS to use, usually the device model

Optional:
  -m, --modules      The discovery/poller module(s) to collect data for, comma delimited
  -n, --prefer-new   Prefer new snmprec data over existing data
  -o, --os           Name of the OS to save test data for (only used if device is generic)
  -f, --file         Save data to file instead of the standard location
  -d, --debug        Enable debug output
      --snmpsim      Run snmpsimd.py using the collected data for manual testing.

Example:
  ./collect-snmp-data.php -h 192.168.0.1 -v 2960x
  ./collect-snmp-data.php -h 127.0.0.1 -v freeradius -m freeradius

I tried substituting the device IP address instead of the device ID but I get the same result. Needs some argument for -v as well?

6

I tried the following command, with the -v option added.

./scripts/collect-snmp-data.php -h 164 -v "Toshiba Printer" -f /tmp/printer.snmprec

This appeared to work however there are two problems - one is that the file -f is trying to create does not get created - the output seems to only go to stdout despite what the inbuilt help says about -f.

The other problem is that that printer is not currently exhibiting the issue that crashes polling - in fact there are no devices currently listed in unpolled devices and when I run poller.php manually again I do not see the error.

So I’ll need to wait until some printers are not polling correctly again. It might be related to something like the printer being out of paper for example which obviously will change through the day as people use printers.

Any thoughts on why -f isn’t working though ? It shows data from the printer to stdout but the file doesn’t even get created. Doesn’t seem to be a permissions issue.

Edit: worked out why it wasn’t saving with -f. Waiting for a printer to misbehave now.

7

Hi,

Hopefully these logs will help. I found another printer playing up and used:

./scripts/collect-snmp-data.php -h 10.0.5.6 -f /tmp/printer.snmprec -v "Lexmark Printer"

Here is the poller error:

https://p.libren.ms/view/c4ba26ee

And here is the log from collect-snmp-data.php:

https://p.libren.ms/view/a85583f8

This problem is hard to reproduce - quite often poller will fail the first time but succeed when re-run.

8

The poller output is only showing part of the error, can you look in logs/librenms.log and post the entire stack trace?

9

Hi,

I think this is what you are looking for ? This is the most recent one in the log so may not be the same specific case as previous log files:

https://p.libren.ms/view/98c032de

10

Does the issue resolve with https://github.com/librenms/librenms/pull/12681?

11

Unfortunately no. As far as I can see my system updated at midnight and is now running this merged patch.

Here is another log entry from this morning:

[2021-03-31 09:27:58] production.ERROR: Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 {"exception":"[object] (TypeError(code: 0): Argument 1 passed to Illuminate\\Database\\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Grammar.php:136)
[stacktrace]
#0 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(886): Illuminate\\Database\\Grammar->parameterize()
#1 [internal function]: Illuminate\\Database\\Query\\Grammars\\Grammar->Illuminate\\Database\\Query\\Grammars\\{closure}()
#2 /opt/librenms/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(630): array_map()
#3 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(887): Illuminate\\Support\\Collection->map()
#4 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/MySqlGrammar.php(134): Illuminate\\Database\\Query\\Grammars\\Grammar->compileInsert()
#5 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(916): Illuminate\\Database\\Query\\Grammars\\MySqlGrammar->compileInsert()
#6 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2878): Illuminate\\Database\\Query\\Grammars\\Grammar->compileInsertGetId()
#7 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1530): Illuminate\\Database\\Query\\Builder->insertGetId()
#8 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1051): Illuminate\\Database\\Eloquent\\Builder->__call()
#9 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1016): Illuminate\\Database\\Eloquent\\Model->insertAndSetId()
#10 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(857): Illuminate\\Database\\Eloquent\\Model->performInsert()
#11 /opt/librenms/app/Facades/LogManager.php(51): Illuminate\\Database\\Eloquent\\Model->save()
#12 /opt/librenms/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(261): App\\Facades\\LogManager->event()
#13 /opt/librenms/LibreNMS/Modules/PrinterSupplies.php(91): Illuminate\\Support\\Facades\\Facade::__callStatic()
#14 /opt/librenms/includes/polling/printer-supplies.inc.php(26): LibreNMS\\Modules\\PrinterSupplies->poll()
#15 /opt/librenms/includes/polling/functions.inc.php(317): include('/opt/librenms/i...')
#16 /opt/librenms/LibreNMS/Util/ModuleTestHelper.php(187): poll_device()
#17 /opt/librenms/LibreNMS/Util/ModuleTestHelper.php(145): LibreNMS\\Util\\ModuleTestHelper->collectOids()
#18 /opt/librenms/scripts/collect-snmp-data.php(131): LibreNMS\\Util\\ModuleTestHelper->captureFromDevice()
#19 {main}
"}

I tried to capture the SNMP data again using collect-snmp-data.php, however it seems that when I try to do a capture from an affected device it simply will not create the capture file - and the log above suggests that it uses the same code and hence may be being hit by the same bug…

So you would need to provide me with an snmpget command to capture the same data without going through collect-snmp-data.php to see the actual returned data from an affected printer. I think the previous capture I posted would not have exhibited the problem or the capture would have failed…

By the way is there an easy way to set up an alert that triggers when there are unpolled devices or is checking in the Poller log “unpolled devices” manually the only way to know?

12

I’m seeing this poller failure too. (also on a printer, when querying toner level or something.)
The fix PR #12681 has been merged (confirmed the change is in my includes/functions.php) but the problem persists.

The stacktrace logged to librenms.log is here:

[2021-04-01 00:16:00] production.ERROR: Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 {"exception":"[object] (TypeError(code: 0): Argument 1 passed to Illuminate\\Database\\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Grammar.php:136)
[stacktrace]
#0 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(886): Illuminate\\Database\\Grammar->parameterize()
#1 [internal function]: Illuminate\\Database\\Query\\Grammars\\Grammar->Illuminate\\Database\\Query\\Grammars\\{closure}()
#2 /opt/librenms/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(630): array_map()
#3 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(887): Illuminate\\Support\\Collection->map()
#4 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/MySqlGrammar.php(134): Illuminate\\Database\\Query\\Grammars\\Grammar->compileInsert()
#5 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php(916): Illuminate\\Database\\Query\\Grammars\\MySqlGrammar->compileInsert()
#6 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2878): Illuminate\\Database\\Query\\Grammars\\Grammar->compileInsertGetId()
#7 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Builder.php(1530): Illuminate\\Database\\Query\\Builder->insertGetId()
#8 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1051): Illuminate\\Database\\Eloquent\\Builder->__call()
#9 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(1016): Illuminate\\Database\\Eloquent\\Model->insertAndSetId()
#10 /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Model.php(857): Illuminate\\Database\\Eloquent\\Model->performInsert()
#11 /opt/librenms/app/Facades/LogManager.php(51): Illuminate\\Database\\Eloquent\\Model->save()
#12 /opt/librenms/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(261): App\\Facades\\LogManager->event()
#13 /opt/librenms/LibreNMS/Modules/PrinterSupplies.php(86): Illuminate\\Support\\Facades\\Facade::__callStatic()
#14 /opt/librenms/includes/polling/printer-supplies.inc.php(26): LibreNMS\\Modules\\PrinterSupplies->poll()
#15 /opt/librenms/includes/polling/functions.inc.php(317): include('/opt/librenms/i...')
#16 /opt/librenms/poller.php(140): poll_device()
#17 {main}
"}

As someone above mentioned, scripts/collect-snmp-data.php fails before producing output, but if you can help me make some sense out of the parameters I can run an snmpwalk and include the output.

Unlike DBMandrake, this is 100% reproducible for me - it fails every single time that device is polled, so it should be quite easy to test against.

The device in question is:

Brother NC-6800h, Firmware Ver.1.01 (08.12.12),MID 84UB05
System Name hl5370dw
Resolved IP 10.0.0.146
Hardware HL-5370DW [001ba90ba752]
Operating System Brother Printer 1.01
Serial B9J154306
Object ID .1.3.6.1.4.1.2435.2.3.9.1

Here is the output of validate.php:

====================================
Component | Version
--------- | -------
LibreNMS  | 21.3.0-57-g6616727a9
DB Schema | 2021_02_09_122930_migrate_to_utf8mb4 (205)
PHP       | 7.4.16
Python    | 3.6.9
MySQL     | 10.5.9-MariaDB-1:10.5.9+maria~bionic
RRDTool   | 1.7.0
SNMP      | NET-SNMP 5.7.3
====================================

Even better, I have another instance of librenms (running the docker image, so I don’t know its git history) which does not have that hex2bin digits cleanup patch, which does poll that device successfully every time.

13

I think it is related to whether the printer is in an error state or not - such as tray open or out of paper.

On the Lexmark printer in my office I was able to poll it without error, I then removed the paper to cause an out of paper situation and was able to reproduce the problem. I was going to post about this finding in a previous post but I was then unable to reliably reproduce it when I put the printer in and out of an error condition multiple times.

For me the problem started well before the patch (since the patch didn’t come until after I reported the problem…) but the patch had no effect - it seems no better or worse than before.

14

Just one more ‘me too’ comment - this is impacting two HP M806s in my environment, in the intermittent category. I’d believe the theory that it has an error then polling fails and re-running manually brings it back into a happy state again.

All I know is that the Capture->Poller thing ends with this each time:

Checking toner Tray 2... 40 %
Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 {"exception":"[object] (TypeError(code: 0): Argument 1 passed to Illuminate\\Database\\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 at /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Grammar.php:136)"} 

In Grammar.php line 136:
Argument 1 passed to Illuminate\Database\Grammar::parameterize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Query/Grammars/Grammar.php on line 886
15

For what it’s worth, I don’t believe this printer is in an error state; at least, the poller fails the same way whether or not the paper tray is installed, and the non-malfunctioning librenms instance doesn’t think it’s in an error state. But I wonder if it could be related to something like number of paper trays present at all.

From the logs on my git-tracking instance, the first instance of this failure appeared after the daily update on Mar 28 (California):

[2021-03-28 00:15:14] production.ERROR: Argument 1 passed to Illuminate\Database\Grammar::parame
terize() must be of the type array, int given, called in /opt/librenms/vendor/laravel/framework/
src/Illuminate/Database/Query/Grammars/Grammar.php on line 886 {"exception":"[object] (TypeError
(code: 0): Argument 1 passed to Illuminate\\Database\\Grammar::parameterize() must be of the typ
e array, int given, called in /opt/librenms/vendor/laravel/framework/src/Illuminate/Database/Que
ry/Grammars/Grammar.php on line 886 at /opt/librenms/vendor/laravel/framework/src/Illuminate/Dat
abase/Grammar.php:136)'

Unfortunately it looks like the commits that went in that particular day changed a lot of stuff but the most likely culprit I guess is:

commit e00eba327d1ed138211218380365efa0141918d5
Author: Jellyfrog <[email protected]>
Date:   Sat Mar 27 16:09:14 2021 +0100

    Re-implement Printer as a class based module (#12605)
16

Seeing the same issue on at least 10 Ricoh copiers.

17

Hi please run

snmprec.py --agent-udpv4-endpoint=device-ip --output-file=test.snmprec

Snmprec.py should be in your path already.

18

For others who might do this, snmprec can be found in the “snmpsim” package, on debian-like OSes anyway. Also note it will reveal some information about your local network such as IP and MAC addresses, if you care.

Here is the output from my printer that’s breaking the poller.

hl5370dw snmprec dump

19

I ended up just disabling printer-supplies entirely (Edit, Modules, left-hand side “Poller Modules”, printer-supplies, OFF)… problem ‘solved’

Sorry I couldn’t help more, but snmprec or snmpsim aren’t things on my distro of choice (RHEL) and I actually need the thing to poll devices to ensure they’re online more than I need to know the toner level is at 17%.

20

snmpsim is a python package which you can install with:
pip3 install snmpsim
if you don’t have pip3, you can do:
yum install python3-pip

1 Like