Moving OS definitions to json / yaml files?


#1

Hi Folks,

So I’m toying with the idea of moving all OS definitions from includes/definitions.inc.php into individual json files such as includes/definitions/ios.json. This would:

  • Abstract config away from the codebase (making it less dependant on php).
  • Reduce the size of definitions.inc.php and thus memory usage slightly.
  • Help organise OS settings into one file, we can store test unit info, mibs + more.

I don’t expect it to be a big job initially as we can create the json files by just looping through $config['os'] and json_encode() each one.

A few example files (based on the same os):

This would run includes/discovery/os/ios.inc.php and includes/poller/os/ios.inc.php still as it’s quite bespoke.

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "custom_discovery": true
    },
    "polling": {
      "customer_poller": true
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

This one would support detection directly in the MIB using OIDs

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "sysObjectId": {
        "1.2.3.4.5.6.7",
        "1.2.3.4.5.6.8",
      }
    },
    "polling": {
      "customer_poller": true
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

This one would use sysName for OS detection and has a custom mib dir.

  {
    "os": "ios",
    "group": "cisco",
    "text": "Cisco IOS",
    "type": "network",
    "icon": "cisco",
    "ifXmcbc": 1,
    "over": [
      {
        "graph": "device_bits",
        "text": "Device Traffic"
      },
      {
        "graph": "device_processor",
        "text": "CPU Usage"
      },
      {
        "graph": "device_mempool",
        "text": "Memory Usage"
      }
    ],
    "bad_ifXEntry": [
      "cisco1941",
      "cisco886Va"
    ],
    "poller_modules":
    {
      "cisco-ace-serverfarms": 1,
      "cisco-ace-loadbalancer": 1,
      "cisco-cbqos": 1,
      "cisco-cef": 1,
      "cisco-mac-accounting": 1,
      "cisco-voice": 1,
      "cisco-remote-access-monitor": 1,
      "cisco-sla": 1,
      "cisco-ipsec-flow-monitor": 1,
      "cipsec-tunnels": 1,
      "cisco-otv": 1,
      "ipmi": 0,
      "toner": 0
    },
    "discovery_modules":
    {
      "toner": 0,
      "cisco-cef": 1,
      "cisco-sla": 1,
      "cisco-mac-accounting": 1,
      "cisco-otv": 1,
      "cisco-pw": 1,
      "cisco-vrf": 1,
      "cisco-vrf-lite": 1
    },
    "discovery": {
      "sysDescr": {
        "Cisco IOS",
        "IOS Device"
      }
    },
    "polling": {
      "customer_poller": true
    },
    "mib_dir": {
      "cisco"
    },
    "register_mibs": {
	    "ciscoAAASessionMIB": "CISCO-AAA-SESSION-MIB"
    }
  }

What do people think about this? Does it look like it might make adding basic detection a bit easier? What is it missing?


#2

I love it. It definitely makes a lot of sense to me to seperate things like definitions, configuration, etc. from the actual code.itself.


#3

Some really good documentation would be needed. Some sort of pre-commit validation would be good too.

What do you think of using YAML?


#4

I personally like json over yaml, easier to read and write in my opinion, as it’s not white-space delimated like YAML. Any particular reason for suggesting it?


#5

Yes, YAML is designed for configuration, json is for serialization.

YAML is a superset if json, so any json file should be valid YAML.

I am not aware of any special use of spacing in YAML.


#6

I’m totally agnostic to the format we pick. I just chose json because I’ve been more used to using it. php has support for yaml files.

I use home assistant which is yaml based, that does have issues with spacing - not sure if that’s a yaml limitation or HA.


#7

Certainly looks cleaner / easier:

group: cisco
text: 'Cisco IOS'
type: network
ifXmcbc: 1
over:
    - { graph: device_bits, text: 'Device Traffic' }
    - { graph: device_processor, text: 'CPU Usage' }
    - { graph: device_mempool, text: 'Memory Usage' }
icon: cisco
bad_ifXEntry:
    - cisco1941
    - cisco886Va
poller_modules:
    cisco-ace-serverfarms: 1
    cisco-ace-loadbalancer: 1
    cisco-cbqos: 1
    cisco-cef: 1
    cisco-mac-accounting: 1
    cisco-voice: 1
    cisco-remote-access-monitor: 1
    cisco-sla: 1
    cisco-ipsec-flow-monitor: 1
    cipsec-tunnels: 1
    cisco-otv: 1
    ipmi: 0
    toner: 0
discovery_modules:
    toner: 0
    cisco-cef: 1
    cisco-sla: 1
    cisco-mac-accounting: 1
    cisco-otv: 1
    cisco-pw: 1
    cisco-vrf: 1
    cisco-vrf-lite: 1

#8

Test branch: https://github.com/librenms/librenms/compare/master...laf:yaml?expand=1

Definitions done as yaml files. Unit testing added. Polling and Disco still use the current includes/*/os/$os.inc.php files but this removes all the config options from includes/definitions.inc.php

Polling still works for me :slight_smile:


#9

Looks like a good start.

I’d like to see more validation of yaml files. For example check that all files contain valid type declarations.
I think you need walk through the files for discovery, but you can unload them if it isn’t an os match.
I think we need to move some of our dependencies to just use composer.json and /vendor, but you don’t have to do that in this PR. (We will still include the files in the repo for ease of install)

Thanks laf :slight_smile:


#10

More tests are easy.

I’ve left using the files for discovery at the moment as we have more OS definitions than discovery files as some are bundled in into one file (linux.inc.php). It will need a fair bit more work to shift some of the os discovery to inside the yaml files.

Problem with using composer is that not everyone has it installed so unless we go down the path of installing it via daily.sh then we can’t make as much use of it unless you know of another way?


#11

Yeah, don’t require the end user to run composer, add /vendor to the repo.


#12

I’ve just pushed an update now.

  • rebased.
  • added some additional tests to check we have text and type.
  • loaded all yaml files into $config for webui :frowning:
  • updated discovery to load the os.

I’ll submit this as a pr over the weekend.


#13

I also need to find a yaml parser we can include. This isn’t compatible back to 5.3 :frowning:

http://uk.php.net/yaml also needs installing.


#14

https://github.com/mustangostang/spyc maybe


#15

Use the Symphony v2 version.


#16

I’ve submitted a pull request with the proposed changes / format.