Moving OS definitions to json / yaml files?

laf · 13 December 2016 23:52

Hi Folks,

So I’m toying with the idea of moving all OS definitions from includes/definitions.inc.php into individual json files such as includes/definitions/ios.json. This would:

Abstract config away from the codebase (making it less dependant on php).
Reduce the size of definitions.inc.php and thus memory usage slightly.
Help organise OS settings into one file, we can store test unit info, mibs + more.

I don’t expect it to be a big job initially as we can create the json files by just looping through $config['os'] and json_encode() each one.

A few example files (based on the same os):

This would run includes/discovery/os/ios.inc.php and includes/poller/os/ios.inc.php still as it’s quite bespoke.

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "custom_discovery": true
    },
    "polling": {
      "customer_poller": true
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

This one would support detection directly in the MIB using OIDs

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "sysObjectId": {
        "1.2.3.4.5.6.7",
        "1.2.3.4.5.6.8",
      }
    },
    "polling": {
      "customer_poller": true
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

This one would use sysName for OS detection and has a custom mib dir.

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "sysDescr": {
        "Cisco IOS",
        "IOS Device"
      }
    },
    "polling": {
      "customer_poller": true
    },
    "mib_dir": {
      "cisco"
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

What do people think about this? Does it look like it might make adding basic detection a bit easier? What is it missing?

Gorian · 14 December 2016 00:18

I love it. It definitely makes a lot of sense to me to seperate things like definitions, configuration, etc. from the actual code.itself.

murrant · 14 December 2016 00:44

Some really good documentation would be needed. Some sort of pre-commit validation would be good too.

What do you think of using YAML?

Gorian · 14 December 2016 03:16

I personally like json over yaml, easier to read and write in my opinion, as it’s not white-space delimated like YAML. Any particular reason for suggesting it?

murrant · 14 December 2016 06:50

Yes, YAML is designed for configuration, json is for serialization.

YAML is a superset if json, so any json file should be valid YAML.

I am not aware of any special use of spacing in YAML.

laf · 14 December 2016 08:46

I’m totally agnostic to the format we pick. I just chose json because I’ve been more used to using it. php has support for yaml files.

I use home assistant which is yaml based, that does have issues with spacing - not sure if that’s a yaml limitation or HA.

laf · 14 December 2016 12:32

Certainly looks cleaner / easier:

group: cisco
text: 'Cisco IOS'
type: network
ifXmcbc: 1
over:
    - { graph: device_bits, text: 'Device Traffic' }
    - { graph: device_processor, text: 'CPU Usage' }
    - { graph: device_mempool, text: 'Memory Usage' }
icon: cisco
bad_ifXEntry:
    - cisco1941
    - cisco886Va
poller_modules:
    cisco-ace-serverfarms: 1
    cisco-ace-loadbalancer: 1
    cisco-cbqos: 1
    cisco-cef: 1
    cisco-mac-accounting: 1
    cisco-voice: 1
    cisco-remote-access-monitor: 1
    cisco-sla: 1
    cisco-ipsec-flow-monitor: 1
    cipsec-tunnels: 1
    cisco-otv: 1
    ipmi: 0
    toner: 0
discovery_modules:
    toner: 0
    cisco-cef: 1
    cisco-sla: 1
    cisco-mac-accounting: 1
    cisco-otv: 1
    cisco-pw: 1
    cisco-vrf: 1
    cisco-vrf-lite: 1

laf · 14 December 2016 20:56

Test branch: https://github.com/librenms/librenms/compare/master...laf:yaml?expand=1

Definitions done as yaml files. Unit testing added. Polling and Disco still use the current includes/*/os/$os.inc.php files but this removes all the config options from includes/definitions.inc.php

Polling still works for me

murrant · 16 December 2016 22:07

Looks like a good start.

I’d like to see more validation of yaml files. For example check that all files contain valid type declarations.
I think you need walk through the files for discovery, but you can unload them if it isn’t an os match.
I think we need to move some of our dependencies to just use composer.json and /vendor, but you don’t have to do that in this PR. (We will still include the files in the repo for ease of install)

Thanks laf

laf · 16 December 2016 23:01

More tests are easy.

I’ve left using the files for discovery at the moment as we have more OS definitions than discovery files as some are bundled in into one file (linux.inc.php). It will need a fair bit more work to shift some of the os discovery to inside the yaml files.

Problem with using composer is that not everyone has it installed so unless we go down the path of installing it via daily.sh then we can’t make as much use of it unless you know of another way?

murrant · 16 December 2016 23:09

Yeah, don’t require the end user to run composer, add /vendor to the repo.

laf · 17 December 2016 00:11

I’ve just pushed an update now.

rebased.
added some additional tests to check we have text and type.
loaded all yaml files into $config for webui
updated discovery to load the os.

I’ll submit this as a pr over the weekend.

laf · 17 December 2016 00:12

I also need to find a yaml parser we can include. This isn’t compatible back to 5.3

http://uk.php.net/yaml also needs installing.

laf · 17 December 2016 00:16

https://github.com/mustangostang/spyc maybe

murrant · 17 December 2016 00:37

Use the Symphony v2 version.

laf · 19 December 2016 21:32

I’ve submitted a pull request with the proposed changes / format.