Another HTTP 500 - Issue

Hello guys,
My LibreNMS web interface has suddenly stopped working. I realized about this yesterday and looked for a solution online. I’ve followed some indications that were given to some other users with the same symptoms than me, so I validated my installation:

root@lnms:/opt/librenms(LAB)# ./validate.php

Component Version
LibreNMS 1.43-81-g57b16c77b
DB Schema ?
PHP 7.0.30-0+deb9u1
MySQL ?
RRDTool 1.6.0
SNMP NET-SNMP 5.7.3
==============================
[FAIL]  Missing dependencies! [FIX] ./scripts/composer_wrapper.php install --no-dev
         fideloper/proxy
[FAIL]  Outdated dependencies [FIX] ./scripts/composer_wrapper.php install --no-dev
         psy/psysh
[OK]    Composer Version: 1.7.2
[FAIL]  Missing dependencies!
        [FIX] composer install --no-dev
        Dependencies:
         fideloper/proxy
root@lnms:/opt/librenms(LAB)#

Then I thought the installation got corrupted during an update, so I ran ./scripts/composer_wrapper.php install --no-dev, as indicated.

root@lnms:/opt/librenms(LAB)# ./scripts/composer_wrapper.php install --no-dev                             
Do not run Composer as root/super user! See https://getcomposer.org/root for details
> LibreNMS\ComposerHelper::preInstall
Loading composer repositories with package information
Installing dependencies from lock file
Package operations: 1 install, 1 update, 28 removals
  - Removing ulrichsg/getopt-php (2.4.3)
  - Removing symfony/class-loader (v3.4.14)
  - Removing squizlabs/php_codesniffer (2.9.1)
  - Removing sebastian/version (2.0.1)
  - Removing sebastian/resource-operations (1.0.0)
  - Removing sebastian/recursion-context (2.0.0)
  - Removing sebastian/object-enumerator (2.0.1)
  - Removing sebastian/global-state (1.1.1)
  - Removing sebastian/exporter (2.0.0)
  - Removing sebastian/environment (2.0.0)
  - Removing sebastian/diff (1.4.3)
  - Removing sebastian/comparator (1.2.4)
  - Removing sebastian/code-unit-reverse-lookup (1.0.1)
  - Removing phpunit/phpunit-mock-objects (3.4.4)
  - Removing phpunit/phpunit (5.7.27)
  - Removing phpunit/php-timer (1.0.9)
  - Removing phpunit/php-text-template (1.2.1)
  - Removing phpunit/php-file-iterator (1.4.5)
  - Removing phpunit/php-code-coverage (4.0.8)
  - Removing phpspec/prophecy (1.8.0)
  - Removing maximebf/debugbar (1.13.1)
  - Removing justinrainbow/json-schema (5.2.7)
  - Removing jakub-onderka/php-parallel-lint (v1.0.0)
  - Removing fzaninotto/faker (v1.8.0)
  - Removing fojuth/readmegen (1.1.2)
  - Removing barryvdh/reflection-docblock (v2.0.4)
  - Removing barryvdh/laravel-ide-helper (v2.4.3)
  - Removing barryvdh/laravel-debugbar (v2.4.3)
  - Updating psy/psysh (v0.9.6 => v0.9.7): Downloading (100%)         
  - Installing fideloper/proxy (4.0.0): Downloading (100%)         
Generating autoload files
> LibreNMS\ComposerHelper::postInstall
> Illuminate\Foundation\ComposerScripts::postInstall
> php artisan optimize
Generating optimized class loader
The compiled services file has been removed.

Then set the permissions again…

root@lnms:/opt/librenms(LAB)# chown -R librenms:librenms /opt/librenms

And ran validate again…

root@lnms:/opt/librenms(LAB)# ./validate.php

Component Version
LibreNMS 1.43-81-g57b16c77b
DB Schema 267
PHP 7.0.30-0+deb9u1
MySQL 10.1.26-MariaDB-0+deb9u1
RRDTool 1.6.0
SNMP NET-SNMP 5.7.3
====================================
[OK]    Composer Version: 1.7.2
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database schema correct
[WARN]  IPv6 is disabled on your server, you will not be able to add IPv6 devices.

At this point, the HTTP Error 500 got fixed and I was able to get to the Login page, but after authentication I received a funny pacman ghost hiding an exception.

This is the chunk of logs being written everytime I press F5.

[2018-09-16 16:22:10] production.ERROR: ErrorException: file_put_contents(/opt/librenms/storage/framework/cache/data/a2/e3/a2e33a1054e69b8e67fdec79e8e92f7ef1097c07): failed to open stream: Permission denied in /opt/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:122
Stack trace:
#0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'file_put_conten...', '/opt/librenms/v...', 122, Array)
#1 /opt/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php(122): file_put_contents('/opt/librenms/s...', '1537126030b:1;', 2)
#2 /opt/librenms/vendor/laravel/framework/src/Illuminate/Cache/FileStore.php(66): Illuminate\Filesystem\Filesystem->put('/opt/librenms/s...', '1537126030b:1;', true)
#3 /opt/librenms/vendor/laravel/framework/src/Illuminate/Cache/Repository.php(175): Illuminate\Cache\FileStore->put('checks_popup_ti...', true, 5)
#4 /opt/librenms/vendor/laravel/framework/src/Illuminate/Cache/CacheManager.php(305): Illuminate\Cache\Repository->put('checks_popup_ti...', true, 5)
#5 /opt/librenms/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(221): Illuminate\Cache\CacheManager->__call('put', Array)
#6 /opt/librenms/app/Checks.php(159): Illuminate\Support\Facades\Facade::__callStatic('put', Array)
#7 /opt/librenms/app/Http/Controllers/LegacyController.php(11): App\Checks::postAuth()
#8 [internal function]: App\Http\Controllers\LegacyController->index('')
#9 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(55): call_user_func_array(Array, Array)
#10 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(44): Illuminate\Routing\Controller->callAction('index', Array)
#11 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Route.php(203): Illuminate\Routing\ControllerDispatcher->dispatch(Object(Illuminate\Routing\Route), Object(App\Http\Controllers\LegacyController), 'index')
#12 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Route.php(160): Illuminate\Routing\Route->runController()
#13 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Router.php(572): Illuminate\Routing\Route->run()
#14 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): Illuminate\Routing\Router->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#15 /opt/librenms/app/Http/Middleware/VerifyTwoFactor.php(38): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#16 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): App\Http\Middleware\VerifyTwoFactor->handle(Object(Illuminate\Http\Request), Object(Closure))
#17 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#18 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(41): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#19 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Routing\Middleware\SubstituteBindings->handle(Object(Illuminate\Http\Request), Object(Closure))
#20 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#21 /opt/librenms/app/Http/Middleware/LegacySession.php(44): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#22 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): App\Http\Middleware\LegacySession->handle(Object(Illuminate\Http\Request), Object(Closure))
#23 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#24 /opt/librenms/app/Http/Middleware/LegacyExternalAuth.php(38): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#25 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): App\Http\Middleware\LegacyExternalAuth->handle(Object(Illuminate\Http\Request), Object(Closure))
#26 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#27 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(65): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#28 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle(Object(Illuminate\Http\Request), Object(Closure))
#29 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#30 /opt/librenms/vendor/laravel/framework/src/Illuminate/Session/Middleware/AuthenticateSession.php(54): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#31 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Session\Middleware\AuthenticateSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#32 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#33 /opt/librenms/vendor/laravel/framework/src/Illuminate/Auth/Middleware/Authenticate.php(43): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#34 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Auth\Middleware\Authenticate->handle(Object(Illuminate\Http\Request), Object(Closure))
#35 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#36 /opt/librenms/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#37 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\View\Middleware\ShareErrorsFromSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#38 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#39 /opt/librenms/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#40 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Session\Middleware\StartSession->handle(Object(Illuminate\Http\Request), Object(Closure))
#41 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#42 /opt/librenms/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#43 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle(Object(Illuminate\Http\Request), Object(Closure))
#44 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#45 /opt/librenms/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(59): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#46 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Cookie\Middleware\EncryptCookies->handle(Object(Illuminate\Http\Request), Object(Closure))
#47 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#48 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#49 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Router.php(574): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#50 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Router.php(533): Illuminate\Routing\Router->runRouteWithinStack(Object(Illuminate\Routing\Route), Object(Illuminate\Http\Request))
#51 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Router.php(511): Illuminate\Routing\Router->dispatchToRoute(Object(Illuminate\Http\Request))
#52 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(176): Illuminate\Routing\Router->dispatch(Object(Illuminate\Http\Request))
#53 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(30): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}(Object(Illuminate\Http\Request))
#54 /opt/librenms/vendor/fideloper/proxy/src/TrustProxies.php(57): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#55 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Fideloper\Proxy\TrustProxies->handle(Object(Illuminate\Http\Request), Object(Closure))
#56 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#57 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(30): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#58 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle(Object(Illuminate\Http\Request), Object(Closure))
#59 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#60 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(30): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#61 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle(Object(Illuminate\Http\Request), Object(Closure))
#62 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#63 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#64 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle(Object(Illuminate\Http\Request), Object(Closure))
#65 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#66 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/CheckForMaintenanceMode.php(46): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#67 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(148): Illuminate\Foundation\Http\Middleware\CheckForMaintenanceMode->handle(Object(Illuminate\Http\Request), Object(Closure))
#68 /opt/librenms/vendor/laravel/framework/src/Illuminate/Routing/Pipeline.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Http\Request))
#69 /opt/librenms/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(102): Illuminate\Routing\Pipeline->Illuminate\Routing\{closure}(Object(Illuminate\Http\Request))
#70 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(151): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#71 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(116): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))
#72 /opt/librenms/html/index.php(53): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))
#73 {main}  
[2018-09-16 16:22:10] production.ERROR: ErrorException: file_put_contents(/opt/librenms/storage/framework/sessions/2AzGrF9fVNuvfU9SbXXUcrCj7tEJeyaN4JsWQhgK): failed to open stream: Permission denied in /opt/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:122
Stack trace:
#0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'file_put_conten...', '/opt/librenms/v...', 122, Array)
#1 /opt/librenms/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php(122): file_put_contents('/opt/librenms/s...', 'a:6:{s:6:"_toke...', 2)
#2 /opt/librenms/vendor/laravel/framework/src/Illuminate/Session/FileSessionHandler.php(83): Illuminate\Filesystem\Filesystem->put('/opt/librenms/s...', 'a:6:{s:6:"_toke...', true)
#3 /opt/librenms/vendor/laravel/framework/src/Illuminate/Session/Store.php(128): Illuminate\Session\FileSessionHandler->write('2AzGrF9fVNuvfU9...', 'a:6:{s:6:"_toke...')
#4 /opt/librenms/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(88): Illuminate\Session\Store->save()
#5 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(218): Illuminate\Session\Middleware\StartSession->terminate(Object(Illuminate\Http\Request), Object(Illuminate\Http\Response))
#6 /opt/librenms/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(189): Illuminate\Foundation\Http\Kernel->terminateMiddleware(Object(Illuminate\Http\Request), Object(Illuminate\Http\Response))
#7 /opt/librenms/html/index.php(58): Illuminate\Foundation\Http\Kernel->terminate(Object(Illuminate\Http\Request), Object(Illuminate\Http\Response))
#8 {main}

I tried to navigate to “/devices/”, tried adding “/debug=yes/” at the end of the URL, but always got the same screen.

I went through the search button looking for “http error 500”. Found some threads but non of them seemed to fit into my situation.

What can I do from here?

Max

chown -R librenms:librenms /opt/librenms
setfacl -d -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
setfacl -R -m g::rwx /opt/librenms/rrd /opt/librenms/logs /opt/librenms/bootstrap/cache/ /opt/librenms/storage/
1 Like

Thanks laf! It’s working again.