VCAP5-DCA – Objective 6.1 – Configure, Manage, and Analyze vSphere Log Files

For this objective I used the following documents:

  • Documents listed in the Tools section

Objective 6.1 – Configure, Manage, and Analyze vSphere Log Files

Knowledge

**ITEMS IN BOLD ARE TOPICS PULLED FROM THE BLUEPRINT**

  • Identify vCenter Server log file names and locations
    • Below are a list of the log file names and their locations  (I used VMware KB1021804 to help me with this)
    • vCenter Server (Windows and Virtual Appliance)
      • vpxd-###– this is the main vCenter log and it will have a number at the end of it. The highest number is the most current log.
        • (Windows) c:\programdata\VMware\VMware VirtualCenter\Logs
        • (Virtual Appliance) /var/log/vmware/vpx
      • vpxd-profiler-### – this log stores some form of metrics. These metrics are viewable at the following URL: https://<ipor hostname of vcenter>/vod/index.html
        • You will be challenged for credentials. Here is a screenshot

image

        • (Windows) c:\programdata\VMware\VMware VirtualCenter\Logs
        • (Virtual Appliance) /var/log/vmware/vpx
      • cim-diag and vws– these logs hold CIM (Common Information Model) information, such as vCenter <—> Hose communications via CIM
        • (Windows) c:\programdata\VMware\VMware VirtualCenter\Logs
        • (Virtual Appliance) /var/log/vmware/vpx
      • There are multiple folders located in the drmdumpfolder at the location below. These folders contain DRS proposed actions and actions taken. These logs are compressed
        • (Windows)c:\programdata\VMware\VMware VirtualCenter\Logs\drmdump\cluster###
        • I couldn’t find these logs in the virtual appliance

 

  • Identify ESXi log file names and locations
    • Here is a list of ESXi log file names and their locations (I used VMware KB2004201 to help me with this)
      • Log and configuration files can be found by going to the following URL:
        • https:<ip or hostname of ESXi host>/host

image

      • You can access logs via the DCUI (Direct Console User Interface)
      • You can access logs by SSH’ing into the host at the following location(s)
        • auth.log– ESXi shell authentication success/failures
          • /var/log/auth.log
        • dhclient.log– DHCP client service log
          • /var/log/dhclient.log
        • esxupdate.log– Update installation log
          • /var/log/esxupdate.log
        • hostd.log– Host management log–also for VM tasks and events
          • /var/log/hostd.log
        • shell.log– logs things that happen in the ESXi shell
          • /var/log/shell.log
        • sysboot.log– early VMkernel startup and module loading
          • /var/log/sysboot.log
        • syslog.log– scheduled tasks, DCUI use and watchdogs
          • /var/log/syslog.log
        • usb.log– USB arbitration events
          • /var/log/usb.log
        • vob.log– VMkernel observation events
          • /var/log/vob.log
        • vmkernel.log– core VMkernel logs, storage and networking device driver events and virtual machine startup
          • /var/log/vmkernel.log
        • vmkwarning.log– summary of warning and alert log messages from the VMkernel logs
          • /var/log/vmkwarning.log
        • vmksummary.log– summary of ESXi startup and shutdown, hourly heartbeat w/uptime, number of VMs running and service resource consumption
          • /var/log/vmksummary.log
        • vpxa.log– vCenter server vpxa agent logs, communication between vCenter and hostd
          • /var/log/vpxa.log
        • fdm.log– vSphere HA (High Availability) events generated by the FDM service
          • /var/log/fdm.log

 

  • Identify tools used to view vSphere log files

Skills and Abilities

  • Generate vCenter Server and ESXi log bundles
    • You can generate log bundles from the GUI or via the command line. I’ll cover both
    • Generate log bundles from the GUI
      • Log into the vSphere client
      • Click File from the top menu > click Export > click Export System Logs…

image

      • Select at which level you want to export logs from. Select the topmost object (the vCenter object) to export ALL logs > leave the checkbox Include Information from vCenter and vSphere Clientchecked if you want to also export those logs
      • Click Next
      • Select which logs you want to export; by default they are all checked. As you can see there are a TON of logs

image

      • Click Next
      • Click the Browse button to select a location to save the log bundle to > click Next
      • Click Finish

image

    • Generate log bundles from the command line
      • SSH into an ESXi host
      • To export logs you’ll need to use the vm-support command. You can export information based on a list of groups or certain ‘manifests’ or you can export information on a particular virtual machine. You can also set a log level with the –logleveloption; values are 0-50 with 0 being the most verbose
      • Generate a generic bundle by executing vm-supportand it will generate a log bundle in a .gz format and by default place it in the /var/tmp directory
        • use the –w option to change the working directory of where the bundle will be saved
  • Use esxcli system syslog to configure centralized logging on ESXi hosts
    • Before you configure your hosts for syslog you can check the current configuration from the command line using the following command
[sourcecode language=”bash” padlinenumbers=”true”] esxcli system syslog config get
[/sourcecode]
    • Here is an example of what it will look like when nothing has been configured for syslog

image

    • You can also use the following command to list out the same details seen in the previous command output, but for each individual log. This also will tell you what all of the log files are
[sourcecode language=”bash”] esxcli system syslog config logger list
[/sourcecode]
    • Before you configure centralized logging you will need to know the location of the remote syslog server. I’m using the VMware syslog Collector in my lab, and these example. I’ll go over how to set this up in the last objective of this section
    • Here is a list of commands on how to change the different syslogging option on an ESXi host. Keep in mind that all of the commands in the following examples are being defined globally, so any sub-logs that have the same value as the default global will also be changed. . You don’t have to execute all commands individually, you can group some into the same command
[sourcecode language=”bash”] # change the default rotation size by executing the following command
# in this example we are changing it to 3MB
# you shouldn’t receive any output (no output is good output); this applies to all the commands listed here

esxcli system syslog config set –default-size=3072

# change the default rotations by executing the following command
# in this example we are changing them to 16

esxcli system syslog config set –default-rotate=16

# set the host to send logs to a remote syslog by executing the following command
# in this example we are sending them to a host with the IP of 10.90.90.10

esxcli system syslog config set –loghost 10.90.190.10

# to load the changes into runtime execute the following command

esxcli system syslog reload

[/sourcecode]
    • Once you’ve changed your configuration and reloaded the syslog daemon, run the following command to ensure the proper changes
[sourcecode language=”bash”] # view your configuration by running the following command

esxcli system syslog config get
[/sourcecode]

    • Now you should see the log rotations set to 16, the size set to 3072 and the loghost set to 10.90.190.10

image

    • If you still don’t see the logs showing up on your remote syslog server after configuring the remote host, ensure that the ESXi firewall has the syslog ports open
      • Log into the vSphere client
      • From the inventory tree, click the ESXi host that you configured the remote syslog server on > click the Configurationtab
      • Under the Software pane click the Security Profilehyperlink
      • In the right-pane click the Properties…hyperlink
      • Find the syslog service located under the Ungroupedservices
      • If the checkbox next to the syslog service isn’t checked, check it

image

      • Click OK

  • Test centralized logging configuration
    • Testing your logging configuration can be down pretty simply. There is a command in the esxcli system syslog namespace that allows you to send a message to all your logs at the same time. You can use this to send a message, and then check the log on your remote syslog system and see if it shows up. Here’s the command
[sourcecode language=”bash” padlinenumbers=”true”] # this command will send the message "vcap5-test-configuration" to all your logs

esxcli system syslog mark –message="vcap5-test-configuration"
[/sourcecode]

    • Here you can see the message was logged, which means your centralized logging is configured properly

image

  • Analyze log entries to obtain configuration information
  • Analyze log entries to identify and resolve issues
    • Unfortunately I’m not to sure how to document these two sections. The list of logs that I detailed above tells you the type of information that you’ll find in each one, which may help in deciphering the configuration or finding relevant entries within a log to help you narrow down a problem
  • Install and configure VMware syslog Collector and ESXi Dump Collector
    • Before we begin you need to have the vCenter bits downloaded
    • Install and Configure VMware syslog Collector
      • Log onto the server you plan on installing syslog Collector
      • From the location of the vCenter bits, double click the autorun.exefile
      • Under vCenter Support Tools click VMware Syslog Collector > click the Install button

image

      • Choose a language and click OK
      • When the install dialog comes up, click Next
      • Click Next > Click the I accept the terms… radial button > click Next
      • Here you can change the default installation directory and the default log repository location
      • You can also set the log rotation frequency and size (2MB and 8 by default, respectively)

image

      • Make changes if desired, click Next
      • Choose whether you want to perform a stand-alone installation or an installation that is integrated with vSphere. For these purposes I’m using the VMware vCenter Server installation
      • Choose an option and click Next
      • If you chose the VMware vCenter Server installation setup type, enter in the IP Address/Nameof the vCenter server along with the username and password
      • Click Next
      • Unless you want to change the ports, leave the defaults at the vSphere Syslog Collector Port Settings page > click Next
      • Click Nextat the identification screen (choose to use the IP or name)
      • Click Install
      • Once the installation is complete, click Finish
    • Install and Configure VMware syslog Collector
      • Log onto the server you plan on installing syslog Collector
      • From the location of the vCenter bits, double click the autorun.exefile
      • Under vCenter Support Tools click VMware ESXi Dump Collector > click the Install button

image

      • Choose a language and click OK
      • When the install dialog comes up, click Next
      • Click Next > Click the I accept the terms… radial button > click Next
      • Here you can change the default installation directory and the default log repository location
      • You can also set the maximum size for the collector repository (default is 2GB)

image

      • Click Next
      • Choose whether you want to perform a stand-alone installation or an installation that is integrated with vSphere. For these purposes I’m using the VMware vCenter Server installation
      • Choose an option and click Next
      • If you chose the VMware vCenter Server installation setup type, enter in the IP Address/Nameof the vCenter server along with the username and password
      • Click Next
      • Unless you want to change the collector server port, click Next
      • Click Nextat the identification screen (choose to use the IP or name)
      • Click Install
      • Once the installation is complete, click Finish

Tools

Comments 2

Leave a Reply

Your email address will not be published. Required fields are marked *

*