How can fix the problem

Test to mail failed Could not instantiate mail function
how can fix this?

Where do you see that error? did you try SMTP aswell?

Post your current mail options in the GUI and output of /opt/librenms/validate.php