Troubleshoot Azure Log Analytics Linux Agent - Azure Monitor (2023)

This article provides help to troubleshoot errors you may encounter with the Log Analytics agent for Linux in Azure Monitor and suggests possible solutions to resolve them.

The Log Analytics Agent Troubleshooter for Linux is a script designed to find and diagnose problems with the Log Analytics agent. It is automatically supplied with the agent during installation. Running the tool should be the first step in diagnosing a problem.

Use the troubleshooting tool

To run the troubleshooter, paste the following command into a terminal window on a computer running the Log Analytics agent:

sudo /opt/microsoft/omsagent/bin/troubleshooter

manual installation

The troubleshooting tool is automatically included when the Log Analytics agent is installed. If the installation somehow fails, you can also install the tool manually:

1. Make sure, thatGNU-Project-Debugger (GDB)is installed on the computer because troubleshooting depends on it.
2. Copy the fix package to your computer:wget https://raw.github.com/microsoft/OMS-Agent-for-Linux/master/source/code/troubleshooter/omsagent_tst.tar.gz
3. Unpack the package:tar -xzvf omsagent_tst.tar.gz
4. Run the manual installation:sudo ./install_tst

covered scenarios

The troubleshooter tool looks for the following scenarios:

• The agent is unhealthy; The heartbeat is not working properly.
• The agent does not start or cannot connect to Log Analytics.
• Agent syslog not working.
• The agent has high CPU or memory usage.
• The agent is having installation problems.
• Custom agent logs do not work.
• Agent logs cannot be collected.

For more information, seeTroubleshooting tool documentation on GitHub.

Clean and reinstall the Linux agent

A clean reinstall of the agent fixes most issues. This task could be the first suggestion from our support team to bring the agent to an undamaged state. Running the Troubleshooter Tool and the Log Collector Tool and attempting a clean reinstall can fix problems faster.

1. Download the cleanup script:

   $ wget https://raw.githubusercontent.com/microsoft/OMS-Agent-for-Linux/master/tools/purge_omsagent.sh

2. Run the cleanup script (with sudo privileges):

   $ sudo sh purge_omsagent.sh

   (Video) Sending Logs From A Linux Server to Log Analytics Workspace in Azure

We recommend that you use the Log Collector tool to get important logs for troubleshooting or before reporting a GitHub issue. For more information about the tool and how to run it, seeOMS Linux Agent Log Collector.

Important configuration files

Installation error codes

Integration Error Codes

Enable debug logging

Debugging the OMS output plugin

FluentD allows plugin-specific logging levels, allowing you to set different logging levels for inputs and outputs. To specify a different logging level for the OMS output, edit the agent's general configuration under/etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsagent.conf.

In the OMS output plugin, before the end of the configuration file, change thelog levelOwnership ofInformationto thedebugging:

<match oms.** docker.**> type out_oms log_level debug num_threads 5 buffer_chunk_limit 5m buffer_type file buffer_path /var/opt/microsoft/omsagent/<workspace id>/state/out_oms*.buffer buffer_queue_limit 10 flush_interval 20s retry_limit 10 retry_wait 30s </match>

With debug logging, you can view batch uploads in Azure Monitor separated by type, number of data items, and upload time.

Here is an example of debuggable logging:

Successful upload of oms.nagios x 1 in 0.14 s Successful upload of oms.omi x 4 in 0.52 s Successful upload of oms.syslog.authpriv.info x 1 in 0.91 s

verbose output

Instead of using the OMS output plugin, you can send data items directly tostdout, which is visible in the Log Analytics agent for Linux log file.

In the general Log Analytics agent configuration file at/etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsagent.conf, comment out the OMS output plugin by adding a#before each line:

#<match oms.** docker.**># type out_oms# log_level info# num_threads 5# buffer_chunk_limit 5m# buffer_type file# buffer_path /var/opt/microsoft/omsagent/<workspace id>/state/out_oms*.buffer# buffer_queue_limit 10# flush_interval 20s# retry_limit 10# retry_wait 30s#</match>

Uncomment the following section under the output plugin by removing the#before each line:

<match **> digite stdout</match>

Issue: Unable to connect to Azure Monitor through proxy

Probable Causes

• The proxy specified during integration was incorrect.
• Azure Monitor and Azure Automation service endpoints are not whitelisted in your data center.

resolution

1. Re-integrate into Azure Monitor with the Log Analytics agent for Linux using the following command with the option-vactivated. It allows verbose output from the agent connecting to Azure Monitor through the proxy:/opt/microsoft/omsagent/bin/omsadmin.sh -w <workspace id> -s <workspace key> -p <proxy conf> -v

2. check the sectionUpdate proxy settingsto verify that you have correctly configured the agent to communicate through a proxy server.

3. Double check the endpoints described in Azure MonitorNetwork Firewall Requirementslist to be correctly added to a whitelist. If you are using Azure Automation, the required network configuration steps are also linked above.

Problem: You get a 403 error when trying to integrate

Probable Causes

• The date and time on the Linux server are incorrect.
• The workspace ID and workspace key are incorrect.

resolution

1. Check the time on your Linux server using the date command. If the time is +/- 15 minutes from the current time, the integration will fail. To fix this issue, update your Linux server's date and/or time zone.
2. Make sure you have the latest version of the Log Analytics agent for Linux installed. The latest version now notifies you when the time difference causes the integration to fail.
3. Re-integrate using the correct workspace ID and workspace key in the installation instructions earlier in this article.

Issue: You see a 500 and 404 error in the log file right after the integration

This is a known issue that occurs when Linux data is first loaded into a Log Analytics workspace. This issue does not affect your data delivery or service experience.

Problem: You see that omiagent is using 100% CPU

Probable Causes

A regression to the nss-pem packagev1.0.3-5.el7caused a serious performance issue. We have seen this problem frequently in Redhat/Centos 7.x distributions. For more information about this issue, see1667121 Performance regression in libcurl.

Performance-related bugs are infrequent and difficult to reproduce. If you have this problem with omiagent, use the scriptomiHighCPUDiagnostics.sh, which captures the omiagent stack trace when it crosses a certain threshold.

1. Download the script:
   wget https://raw.githubusercontent.com/microsoft/OMS-Agent-for-Linux/master/tools/LogCollector/source/omiHighCPUDiagnostics.sh

2. Run the diagnostics for 24 hours with 30% CPU limit:
   bash omiHighCPUDiagnostics.sh --runtime-in-min 1440 --cpu-threshold 30

3. The call stack is stored in omiagent_trace. If you notice a lot of curl and NSS function calls, follow these troubleshooting steps.

resolution

1. Update the nss-pem packagev1.0.3-5.el7_6.1:
   sudo yum update nss-pem

2. If nss-pem isn't available for an upgrade, which Centos most often is, downgrade curl to 7.29.0-46. If you accidentally run "yum update" curl will be updated to 7.29.0-51 and the problem will appear again:
   sudo yum downgrade curl libcurl

3. IMO new start:
   sudo scxadmin -restart

Problem: You don't see forwarded syslog messages

Probable Causes

• The configuration applied to the Linux server does not allow the capture of sent facilities or log levels.
• Syslog is not correctly forwarded to the Linux server.
• The number of messages forwarded per second is too large for the basic configuration of the Log Analytics agent for Linux.

resolution

• Ensure that the configuration in the Log Analytics workspace for syslog has all features and the correct log levels. analysisConfigure the syslog collection in the Azure portal.
• Check the native syslog message daemons (rsyslog,syslog-from) can receive forwarded messages.
• Check the firewall settings on the syslog server to ensure messages are not being blocked.
• Simulate a syslog message for Log Analytics with aRecorderCommand:
  logger -p local0.err "This is my test message"

Problem: You get an Errno address that is already used in the omsagent log file

Do you see[error]: Unexpected error error_class=Errno::EADDRINUSE error=#<Errno::EADDRINUSE: Address is already in use - bind(2) to "127.0.0.1" port 25224>em omsagent.log.

Probable Causes

This error indicates that the Linux diagnostics extension (LAD) is installed alongside the Log Analytics Linux VM extension. It uses the same port for collecting syslog data as omsagent.

resolution

1. Run the following commands as root. Note that 25224 is an example and it is possible that your environment shows a different port number used by LAD.

   /opt/microsoft/omsagent/bin/configure_syslog.sh configure LAD 25229sed -i -e 's/25224/25229/' /etc/opt/microsoft/omsagent/LAD/conf/omsagent.d/syslog.conf

   You must then edit the correct onersyslogdorsyslog_ofconfig and change the LAD related setting to write on port 25229.

2. When the VM is runningrsyslogd, which is the file to change/etc/rsyslog.d/95-omsagent.conf(if available, else/etc/rsyslog🇧🇷 When the VM is runningsyslog_of, which is the file to change/etc/syslog-ng/syslog-ng.conf.

3. restart omagentRestart sudo /opt/microsoft/omsagent/bin/service_control.

4. Restart the syslog service.

Issue: You cannot uninstall omsagent using the clean option

Probable Causes

• The Linux diagnostics extension is installed.
• Installed and uninstalled the linux diagnostics extension but still getting an error saying omsagent is in use by mdsd and cannot be removed.

resolution

1. Uninstall the Linux diagnostic extension.
2. Remove the Linux diagnostic extension files from the machine if they are in the following location:/var/lib/waagent/Microsoft.Azure.Diagnostics.LinuxDiagnostic-<Version>/e/var/opt/microsoft/omsagent/LAD/.

Problem: You cannot see any Nagios data

Probable Causes

• The omsagent user does not have permission to read the Nagios log file.
• The Nagios source and filter were not commented out in the omsagent.conf file.

resolution

1. Add the omsagent user to read the nagios file by following theminstructions.

2. In the general configuration file for the Log Analytics agent for Linux at/etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsagent.conf, making sureBoththe Nagios source and the filter are not commented out.

   <translation> type end path /var/log/nagios/nagios.log format no tag oms.nagios</source><filter oms.nagios> type filter_nagios_log</filter>

Problem: You don't see any Linux data

Probable Causes

• Integration with Azure Monitor failed.
• The connection to Azure Monitor is blocked.
• The virtual machine has restarted.
• The OMI package was manually updated to a newer version than the one installed by the Log Analytics agent package for Linux.
• OMI is frozen and blocking the OMS agent.
• DSC resource recordsClass not foundmistake inomsconfig.loglog file.
• The Log Analytics agent for data is backed up.
• DSC recordsThe current configuration does not exist. Run the Start-DscConfiguration command with the -Path parameter to specify a configuration file and create a current configuration first.Insideomsconfig.loglog file, but there is no log message about itPerform required configuration checksThe operation.

resolution

1. Install all dependencies as an auditd package.

   (Video) Azure Monitor, Log Analytics, Diagnostic settings

2. Verify that the integration with Azure Monitor was successful by verifying that the following file is present:/etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsadmin.conf🇧🇷 If this is not the case, perform the reintegration using the omsadmin.sh command lineinstructions.

3. If you are using a proxy, review the previous proxy troubleshooting steps.

4. On some Azure distribution systems, the OMI server daemon omid does not start after restarting the virtual machine. If this is the case, you will not see any data related to the Audit, ChangeTracking, or UpdateManagement solution. The solution is to start the OMI server manually by running itRestart sudo /opt/omi/bin/service_control.

5. After manually updating the OMI package to a newer version, it needs to be restarted manually for the Log Analytics agent to continue working. This step is required for some distributions where the OMI server does not start automatically after the upgrade. To runRestart sudo /opt/omi/bin/service_controlto restart imo.

   In some situations, the OMI may be frozen. The OMS agent can go into a blocked state waiting for OMI, blocking all data collection. The OMS agent process is running, but there is no activity, as indicated by the lack of new log lines (e.g. heartbeats sent).omsagent.log🇧🇷 Restart the OMI withRestart sudo /opt/omi/bin/service_controlcall the agent.

6. If you see a DSC resourceClass not foundError in omsconfig.log, runRestart sudo /opt/omi/bin/service_control.

7. In some cases, when the Log Analytics agent for Linux cannot communicate with Azure Monitor, the data on the agent is backed up up to the total buffer size of 50MB. The agent needs to be restarted by running the following command:/opt/microsoft/omsagent/bin/service_control reiniciar.

   monitoring

   This issue has been fixed in agent version 1.1.0-28 or later.

   • Se oomsconfig.logThe log file does not show thisPerform required configuration checksIf there are regular operations running on the system, there may be a problem with the cron job/service. Check if the cron job exists at/etc/cron.d/OMSConsistencyInvoker🇧🇷 If necessary, run the following commands to create the cron job:

     mkdir -p /etc/cron.d/echo "*/15 * * * * aboutagent /opt/omi/bin/OMSConsistencyInvokes >/dev/null 2>&1" | sweat tea /etc/cron.d/OMSConsistencyInvoker

   • Also make sure that the cron service is running. You can useCron status of the servicewith Debian, Ubuntu and SUSE orCrond status of the servicewith RHEL, CentOS and Oracle Linux to check the status of this service. If the service does not exist, you can install the binaries and start the service using the following instructions:

     Ubuntu/Debian

     # To install service binaries sudo apt-get install -y cron# To start services sudo service cron start

     SUSE

     # To install binary service sudo zypper in cron -y# To start services sudo systemctl enable cronsudo systemctl start cron

     RHEL/CeonOS

     # To install the binary service sudo yum install -y crond# To start the service sudo service crond start

     OracleLinux

     # To install the binary service sudo yum install -y cronie# To start the service sudo service crond start

Issue: When you configure portal collection for syslog or linux performance counters, the settings are not applied

Probable Causes

• The Log Analytics agent for Linux did not select the latest configuration.
• Changed settings in the portal were not adopted.

resolution

Background: omsconfigis the Log Analytics configuration agent for Linux that checks for a new portal-side configuration every five minutes. This setting is then applied to the Log Analytics agent for Linux configuration files located in /etc/opt/microsoft/omsagent/conf/omsagent.conf.

In some cases, the Log Analytics configuration agent for Linux might not be able to communicate with the portal configuration service. This scenario causes the latest configuration not to be applied.

1. Check if theomsconfigAgent is installed by runningdpkg --list omsconfigorrpm -qi omsconfig🇧🇷 If it's not installed, reinstall the latest version of the Log Analytics agent for Linux.

2. Check if theomsconfigThe agent can communicate with Azure Monitor by running the following command:sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/GetDscConfiguration.py'🇧🇷 This command returns the configuration that the agent receives from the service, including syslog settings, Linux performance counters, and custom logs. If this command fails, run the following command:sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/PerformRequiredConfigurationChecks.py'🇧🇷 This command forces the omsconfig agent to talk to Azure Monitor and get the latest configuration.

Problem: You don't see any custom registration data

Probable Causes

• Integration with Azure Monitor failed.
• The configurationApply the following configuration to my Linux serverswas not selected.
• omsconfigdid not pick up the latest custom logging configuration from the service.
• The Log Analytics agent for Linux usersOmsagentUnable to access custom log due to permissions or not found. You may see the following errors:
  • [DATETIME] [Warning]: File not found. Proceed without following it.
  • [DATETIME] [Error]: Unable to get file from omsagent.
• Fixed known issue with race condition in Log Analytics agent for Linux version 1.1.0-217.

resolution

1. Verify that the integration with Azure Monitor was successful by verifying that the following file is present:/etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsadmin.conf🇧🇷 If not, not either:

   1. Perform the reintegration from the omsadmin.sh command lineinstructions.
   2. UnderAdvanced settingsIn the Azure portal, ensure that the configurationApply the following configuration to my Linux serversactivated.

2. Check if theomsconfigThe agent can communicate with Azure Monitor by running the following command:sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/GetDscConfiguration.py'🇧🇷 This command returns the configuration that the agent receives from the service, including syslog settings, Linux performance counters, and custom logs. If this command fails, run the following command:sudo su omsagent -c 'python /opt/microsoft/omsconfig/Scripts/PerformRequiredConfigurationChecks.py'🇧🇷 This command forces theomsconfigAgent to talk to Azure Monitor and get the latest configuration.

Background:Instead of running the Log Analytics agent for Linux as a privileged user –Those, the agent runs asOmsagentof the user. In most cases, explicit permission must be given to that user in order for certain files to be read. to grant permissionOmsagentUser, run the following commands:

1. AddOmsagentUsers for the specific group:sudo usermod -a -G <GROUPNAME> <USERNAME>.
2. Grant universal read access to the required file:sudo chmod -R ugo+rx <DATEIVERZEICHNIS>.

There is a known issue with a race condition with the Log Analytics agent for Linux versions earlier than 1.1.0-217. After updating to the latest agent, run the following command to get the latest version of the output plugin:sudo cp /etc/opt/microsoft/omsagent/sysconf/omsagent.conf /etc/opt/microsoft/omsagent/<Arbeitsbereich-ID>/conf/omsagent.conf.

Issue: You are trying to rejoin a new workspace

If you attempt to reintegrate an agent into a new workspace, the Log Analytics agent configuration must be cleared before reintegrating. To clean up the old agent configuration, run the shell bundle with--clean:

sudo sh ./omsagent-*.universal.x64.sh --purge

or

sudo sh ./onboard_agent.sh --purge

You can continue to reboard after using--cleanPossibility.

Issue: The Log Analytics agent extension in the Azure portal is flagged with an error status: Deployment failed

Probable Causes

• The Log Analytics agent has been removed from the operating system.
• The Log Analytics agent service is down, disabled, or not configured.

resolution

1. Remove the extension from the Azure portal.
2. Install the agent after theinstructions.
3. Restart the agent by running the following command:
   Restart sudo /opt/microsoft/omsagent/bin/service_control.
4. Wait a few minutes for the deployment status to changesuccessful deployment.

Issue: Updating the Log Analytics agent on demand

Probable Causes

The Log Analytics agent packages on the host are outdated.

resolution

1. Check the latest version belowthis GitHub page.

2. Download the installation script (1.4.2-124 is an example version):

   wget https://github.com/Microsoft/OMS-Agent-for-Linux/releases/download/OMSAgent_GA_v1.4.2-124/omsagent-1.4.2-124.universal.x64.sh

3. Update packages by running themsudo sh ./omsagent-*.universal.x64.sh --upgrade.

Problem: The installation fails and it says Python2 doesn't support ctypes even though Python3 is used

Probable Causes

In this known issue, if the language of the VM is not English, a check to determine which version of Python is being used will fail. This issue causes the agent to always assume Python2 is being used and fail if Python2 isn't present.

resolution

Change the language of the VM environment to English:

export LANG=en_US.UTF-8