Very Strange traffic on two device

[Matteo_Anderson]

Hi all,
we have a very strange traffic spikes on two device.

We searched the forum and FAQ, we activate RRDTune for both the devices and all the ports.
Then we run the scripts with the following parameters:
./scripts/tune_port.php -h DEVICEIP -p all

The script cycle all the port but nothing changed.

Did you have any idea?

====================================

Component | Version

--------- | -------

LibreNMS | 21.10.0

DB Schema | 2021_25_01_0129_isis_adjacencies_nullable (221)

PHP | 7.4.24

Python | 3.9.5

MySQL | 10.5.12-MariaDB-1:10.5.12+maria~focal

RRDTool | 1.7.2

SNMP | NET-SNMP 5.9

====================================

[OK] Installed from the official Docker image; no Composer required

[OK] Database connection successful

[FAIL] You have lower_case_table_names set to 1 or true in mysql config.

[FIX]:

Set lower_case_table_names=0 in your mysql config file in the [mysqld] section.

[FAIL] Database: missing table (bgpPeers)

[FAIL] Database: missing table (bgpPeers_cbgp)

[FAIL] Database: missing table (ciscoASA)

[FAIL] Database: missing table (dbSchema)

[FAIL] Database: missing table (entityState)

[FAIL] Database: missing table (entPhysical)

[FAIL] Database: missing table (entPhysical_state)

[FAIL] Database: missing table (hrDevice)

[FAIL] Database: missing table (hrSystem)

[FAIL] Database: missing table (juniAtmVp)

[FAIL] Database: extra table (bgppeers)

[FAIL] Database: extra table (bgppeers_cbgp)

[FAIL] Database: extra table (ciscoasa)

[FAIL] Database: extra table (dbschema)

[FAIL] Database: extra table (entitystate)

[FAIL] Database: extra table (entphysical)

[FAIL] Database: extra table (entphysical_state)

[FAIL] Database: extra table (hrdevice)

[FAIL] Database: extra table (hrsystem)

[FAIL] Database: extra table (juniatmvp)

[FAIL] We have detected that your database schema may be wrong

[FIX]:

Run the following SQL statements to fix it

SQL Statements:

CREATE TABLE bgpPeers (bgpPeer_id int unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , vrf_id int unsigned NULL , astext varchar(255) NOT NULL , bgpPeerIdentifier text NOT NULL , bgpPeerRemoteAs bigint NOT NULL , bgpPeerState text NOT NULL , bgpPeerAdminStatus text NOT NULL , bgpPeerLastErrorCode int NULL , bgpPeerLastErrorSubCode int NULL , bgpPeerLastErrorText varchar(254) NULL , bgpLocalAddr text NOT NULL , bgpPeerRemoteAddr text NOT NULL , bgpPeerDescr varchar(255) NOT NULL DEFAULT ‘’ , bgpPeerInUpdates int NOT NULL , bgpPeerOutUpdates int NOT NULL , bgpPeerInTotalMessages int NOT NULL , bgpPeerOutTotalMessages int NOT NULL , bgpPeerFsmEstablishedTime int NOT NULL , bgpPeerInUpdateElapsedTime int NOT NULL , context_name varchar(128) NULL , PRIMARY KEY (bgpPeer_id), INDEX bgppeers_device_id_context_name_index (device_id,context_name));

CREATE TABLE bgpPeers_cbgp (device_id int unsigned NOT NULL , bgpPeerIdentifier varchar(64) NOT NULL , afi varchar(16) NOT NULL , safi varchar(16) NOT NULL , AcceptedPrefixes int NOT NULL , DeniedPrefixes int NOT NULL , PrefixAdminLimit int NOT NULL , PrefixThreshold int NOT NULL , PrefixClearThreshold int NOT NULL , AdvertisedPrefixes int NOT NULL , SuppressedPrefixes int NOT NULL , WithdrawnPrefixes int NOT NULL , AcceptedPrefixes_delta int NOT NULL , AcceptedPrefixes_prev int NOT NULL , DeniedPrefixes_delta int NOT NULL , DeniedPrefixes_prev int NOT NULL , AdvertisedPrefixes_delta int NOT NULL , AdvertisedPrefixes_prev int NOT NULL , SuppressedPrefixes_delta int NOT NULL , SuppressedPrefixes_prev int NOT NULL , WithdrawnPrefixes_delta int NOT NULL , WithdrawnPrefixes_prev int NOT NULL , context_name varchar(128) NULL , UNIQUE bgppeers_cbgp_device_id_bgppeeridentifier_afi_safi_unique (device_id,bgpPeerIdentifier,afi,safi), INDEX bgppeers_cbgp_device_id_bgppeeridentifier_context_name_index (device_id,bgpPeerIdentifier,context_name));

CREATE TABLE ciscoASA (ciscoASA_id int unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , oid varchar(255) NOT NULL , data bigint NOT NULL , high_alert bigint NOT NULL , low_alert bigint NOT NULL , disabled tinyint NOT NULL DEFAULT ‘0’ , PRIMARY KEY (ciscoASA_id), INDEX ciscoasa_device_id_index (device_id));

CREATE TABLE dbSchema (version int NOT NULL DEFAULT ‘0’ , PRIMARY KEY (version));

CREATE TABLE entityState (entity_state_id int unsigned NOT NULL auto_increment, device_id int unsigned NULL , entPhysical_id int unsigned NULL , entStateLastChanged datetime NULL , entStateAdmin int NULL , entStateOper int NULL , entStateUsage int NULL , entStateAlarm text NULL , entStateStandby int NULL , PRIMARY KEY (entity_state_id), INDEX entitystate_device_id_index (device_id));

CREATE TABLE entPhysical (entPhysical_id int unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , entPhysicalIndex int NOT NULL , entPhysicalDescr text NOT NULL , entPhysicalClass text NOT NULL , entPhysicalName text NOT NULL , entPhysicalHardwareRev varchar(64) NULL , entPhysicalFirmwareRev varchar(64) NULL , entPhysicalSoftwareRev varchar(64) NULL , entPhysicalAlias varchar(32) NULL , entPhysicalAssetID varchar(32) NULL , entPhysicalIsFRU varchar(8) NULL , entPhysicalModelName text NOT NULL , entPhysicalVendorType text NULL , entPhysicalSerialNum text NOT NULL , entPhysicalContainedIn int NOT NULL , entPhysicalParentRelPos int NOT NULL , entPhysicalMfgName text NOT NULL , ifIndex int NULL , PRIMARY KEY (entPhysical_id), INDEX entphysical_device_id_index (device_id));

CREATE TABLE entPhysical_state (id bigint unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , entPhysicalIndex varchar(64) NOT NULL , subindex varchar(64) NULL , group varchar(64) NOT NULL , key varchar(64) NOT NULL , value varchar(255) NOT NULL , PRIMARY KEY (id), INDEX device_id_index (device_id,entPhysicalIndex));

CREATE TABLE hrDevice (hrDevice_id int unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , hrDeviceIndex int NOT NULL , hrDeviceDescr text NOT NULL , hrDeviceType text NOT NULL , hrDeviceErrors int NOT NULL DEFAULT ‘0’ , hrDeviceStatus text NOT NULL , hrProcessorLoad tinyint NULL , PRIMARY KEY (hrDevice_id), INDEX hrdevice_device_id_index (device_id));

CREATE TABLE hrSystem (hrSystem_id int unsigned NOT NULL auto_increment, device_id int unsigned NOT NULL , hrSystemNumUsers int NULL , hrSystemProcesses int NULL , hrSystemMaxProcesses int NULL , PRIMARY KEY (hrSystem_id), INDEX hrsystem_device_id_index (device_id));

CREATE TABLE juniAtmVp (id bigint unsigned NOT NULL auto_increment, juniAtmVp_id int unsigned NOT NULL , port_id int unsigned NOT NULL , vp_id int unsigned NOT NULL , vp_descr varchar(32) NOT NULL , PRIMARY KEY (id), INDEX juniatmvp_port_id_index (port_id));

DROP TABLE bgppeers;

DROP TABLE bgppeers_cbgp;

DROP TABLE ciscoasa;

DROP TABLE dbschema;

DROP TABLE entitystate;

and 5 more…

[WARN] IPv6 is disabled on your server, you will not be able to add IPv6 devices.

[WARN] Updates are managed through the official Docker image

[Matteo_Anderson]

in the meanwhile we fixed every Database Warning, still see spikes in the overall traffic but not in the single interface detail.
Can someone help us?

[DBMandrake]

Hi,

Please read through these two threads, I suspect you are having the same issue - occasional partial failure of SNMP polling from the affected devices causing 0 bytes to be stored in the interface counters.

On subsequent polling with a proper value this will look like an enormous traffic spike for one polling session. I encountered it when testing port traffic utilisation alerts and actually had to add a workaround in my alert rules otherwise I’d get at least one or two per day.

Graph spikes in Cisco Nexus 5672UP

Hello, I’m newbie on librenms and I have a new installation with some devices configured. My problem is that some of them show spikes of Tb on their overall traffic graphs and this values aren’t correct. Specifically, this occurs with devices Cisco Nexus 5672UP, Cisco NX-OS and a large amount of ports enabled. This is a graph of one of them: [image] The other devices show their graphs correctly without spikes. This is validate.php output: ==================================== Component | Ve…

Incorrect port utilisation alerting with failed SNMP sessions?

I recently set up ethernet port utilisation alerting for switch ports, and after a bit of tuning of the alert and template it works really well, except for one thing. We have one switch which from time to time stops responding to SNMP (and web interface) for periods of time around 5-30 seconds so that there are the occasional 5 minute polling periods (a few per day) where LibreNMS is not able to poll SNMP on the device, and then in the following polling period it can again. This switch frequen…

One way to detect this would be to set up a high traffic utilisation alert and put the byte counters in your template so you can see the exact byte counters when the alert sets.

Include these variables in your alert template and they will help you pinpoint whether this is your issue or not:

ifInOctets
ifInOctets_prev
ifInOctets_delta
ifInOctets_rate
ifOutOctets
ifOutOctets_prev
ifOutOctets_delta
ifOutOctets_rate

Most likely you will see an ifInOctets_prev or ifOutOctets_prev value of 0 when the alert happens.

Here is an example of an alert you could use:

This will trigger when a port is over 80% utilisation. You could try increasing the comparison to check for >= 100 which should never normally occur but it should still trigger when you see your abnormal spikes.

If you find that this is indeed your issue then it is caused by the SNMP polling session to your device hanging, timing out or disconnecting part way through. If this happens at just the right place it can cause incorrect data to be saved.