Extract LibreNMS data from filesystem on dead/dying SD card

Help
1

I’ve been running LibreNMS on a Raspberry Pi to monitor my home network and it’s been flawless until a couple of days ago when my SD card suddenly died :disappointed:

The Pi will no longer boot from the card and Linux doesn’t like it either. I tried running fsck but the card has gone into read-only mode so fsck can’t fix any of the problems it finds.

However, I do seem to be able to read data from the card using ext2explore on Windows.

Setting up a brand new instance of LibreNMS wouldn’t be the end of the world, but I’d like to salvage the data (going back several years) from the broken card if possible. Is there any way I can do that?

The normal backup/migration instructions involve dumping the SQL database, but obviously I can’t run that command - I only have access the contents of the filesystem.

Any help would be greatly appreciated :+1:

2

Copy the data to new SD card and try to boot?

3

I did try to create an image of the card using USBIT, but it failed fairly early on (with a CRC error IIRC). It seems that copying the entire card is not an option unfortunately.

I guess I could try dd instead.

4

I would use one of:

and would try to merge the files with a pristine install on a new SD card.

Good luck

5

This tool is hitting loads of CRC errors e.g.:

12/10/2022 12:05:38   Read Error occurred at offset 10,909,319,168;  LBA 21,307,264   (Data error (cyclic redundancy check))
12/10/2022 12:05:41   Read Error occurred at offset 10,911,416,320;  LBA 21,311,360   (Data error (cyclic redundancy check))
12/10/2022 12:05:42   Read Error occurred at offset 10,913,513,472;  LBA 21,315,456   (Data error (cyclic redundancy check))
12/10/2022 12:05:48   Read Error occurred at offset 10,934,484,992;  LBA 21,356,416   (Data error (cyclic redundancy check))
12/10/2022 12:05:52   Read Error occurred at offset 10,936,582,144;  LBA 21,360,512   (Data error (cyclic redundancy check))
12/10/2022 12:05:53   Read Error occurred at offset 10,938,679,296;  LBA 21,364,608   (Data error (cyclic redundancy check))
12/10/2022 12:05:59   Read Error occurred at offset 10,963,845,120;  LBA 21,413,760   (Data error (cyclic redundancy check))
12/10/2022 12:06:03   Read Error occurred at offset 10,965,942,272;  LBA 21,417,856   (Data error (cyclic redundancy check))
12/10/2022 12:06:04   Read Error occurred at offset 10,968,039,424;  LBA 21,421,952   (Data error (cyclic redundancy check))
12/10/2022 12:07:06   Read Error occurred at offset 11,387,469,824;  LBA 22,241,152   (Data error (cyclic redundancy check))
12/10/2022 12:07:10   Read Error occurred at offset 11,389,566,976;  LBA 22,245,248   (Data error (cyclic redundancy check))

But it’s copying the data anyway. I suspect the resulting image will be useless but I’ll try writing it to a new SD card and running fsck on it and see what happens. If that fails I’ll admit defeat I think.

6

I would concentrate only on the data files and not on the OS files as writing the OS onto a new SD card is quick as is installing additional programs. Still attempting to get a consistent file system via fsck is worth a try as maybe somethings can be restored from the journal.

7

Maybe you can mount the img rw via loop and running fsck is possible without writing to SD card.
Would try on a copy not on the original img thou.

8

I would concentrate only on the data files and not on the OS files

This was my original idea, but I don’t know which files I need.

Maybe you can mount the img rw via loop and running fsck is possible without writing to SD card.
Would try on a copy not on the original img thou.

Yeah, I was thinking the same thing.

9

I don’t know how you installed LibreNMS but in my case I would try to get /opt & /var/lib/mysql.

10

Ok, I’ve done pretty well in the end :+1:

I copied the corrupted SD card to an image file, mounted it as a loopback device and ran fsck on it, which found and “fixed” loads of errors.

Then I wrote that fixed image back to the SD card but it wouldn’t boot. Connecting the Pi up to a monitor and keyboard revealed that fundamental libraries in the OS were corrupted, so even very basic things didn’t work :frowning:

So I packaged up /opt/librenms and /var/lib/mysql into tar files and pulled them off the SD card using ext2explore (I couldn’t SFTP them off because even SFTP wouldn’t run due to some library being corrupted).

So then I went back to an old (working) backup image of the SD card from two years ago, copied that to the SD card, booted that up, then tried replacing /opt/librenms and /var/lib/mysql with the versions taken from the bad SD. This seemed to break all sorts of things, so I reinstated the original directories and just copied /opt/librenms/rrd from the bad SD card.

And hey presto, nearly all of my old data is back. I just had to rediscover a couple of devices that had been updated since 2020, and add a couple of devices that I had added in the meantime.

11

I wouldn’t trust the old card but hey, that’s just me :sweat_smile:

12

Sorry, I probably wasn’t clear enough in my last post - everything is now on a brand new SD card, and the old one is in the bin :slightly_smiling_face:

i.e. this part:

So then I went back to an old (working) backup image of the SD card from two years ago, copied that to the SD card…

Should have said:

So then I went back to an old (working) backup image of the SD card from two years ago, copied that to a new SD card…

I’ve spent a couple of quid extra on a “high endurance” SD card. Let’s see if that lasts longer than a couple of years…

13

If you can spare the RAM you could mount a ramdisk to /tmp and also disable local logging and send the events to a logserver. I don’t know which LibreNMS folders could also be on a ramdisk but I guess all that isn’t needed after a reboot of the server.

14

Yes, good idea. I’ve installed log2ram and moved /var/log and /opt/librenms/logs to ram disks :+1: