Tuesday, May 20, 2014

Vmware KB Testing VMkernel network connectivity with the vmkping command (1003728)

Testing VMkernel network connectivity with the vmkping command (1003728)

Purpose

For troubleshooting purposes, it may be necessary to test VMkernel network connectivity between ESX hosts in your environment.

This article provides you with the steps to perform a vmkping test between your ESX hosts.

Resolution

The vmkping command sources a ping from the local VMkernel port.
 
To initiate a vmkping test from the console of an ESX Server host:
  1. Connect to the ESX/ESXi host using an SSH session. For information see Tech Support Mode for Emergency Support (1003677) andUsing Tech Support Mode in ESXi 4.1 and 5.0 (1017910).
  2. In the command shell, run the command:

    # vmkping x.x.x.x
    where x.x.x.x is the hostname or IP address of the server that you want to ping.
  3. If you have Jumbo Frames configured in your environment, run the vmkping command with the -s and -d options.

    # vmkping -d -s 8972 x.x.x.x
    Note: If you have more then one vmkernel port on the same network (such as a heartbeat vmkernel port for iSCSI) then all vmkernel ports on the host on the network would need to be configured with Jumbo Frames (MTU: 9000) too. If there are other vmkernel ports on the same network with a lower MTU then the vmkping command will fail with the -s 8972 option. Here in the command -d option sets DF (Don't Fragment) bit on the IPv4 packet.

  4. In ESXi 5.1, you can specify which vmkernel port to use for outgoing ICMP traffic with the -I option:

    # vmkping -I vmkX x.x.x.x

    Notes: 
    • ICMP response behavior has changed in ESXi 5.1. For more information, see Change to ICMP ping response behavior in ESXi 5.1 (2042189).
    • In releases prior to ESXi 5.1, the host will automatically select the vmkernel port based on the host's vmkernel routing/forwarding table. To display the host's vmkernel routing table, use the esxcfg-route -l command.
    • Verification of your MTU size can be obtained from a SSH session by using
esxcfg-nics -l

Output should be similar to:

# esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description
vmnic0  0000:02:00.00 e1000       Up   1000Mbps  Full   00:50:56:17:0a:60 9000   Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
vmnic1  0000:02:01.00 e1000       Up   1000Mbps  Full   00:50:56:17:0a:65 9000   Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)

esxcfg-vmknic -l 

Output should be similar to:

# esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                    Netmask         Broadcast   MAC Address          MTU     TSO     MSS       Enabled Type

vmk1       iSCSI  IPv4      10.10.10.10                                255.255.255.0   10.10.10.255 00:50:56:XX:XX:64    9000    65535     true    STATIC

A successful ping response is similar to:
 
# vmkping 10.0.0.1
PING server(10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=10.245 ms
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.935 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.926 ms
--- server ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.926/4.035/10.245 ms
 
An unsuccessful ping response is similar to:
 
# vmkping 10.0.0.2
PING server (10.0.0.2) 56(84) bytes of data.
--- server ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 3017ms

 
Notes:
  • If you see intermittent ping success, this might indicate you have incompatible NICs teamed on the VMotion port. Either team compatible NICs or set one of the NICs to standby.
  • If you do not see a response when pinging by the hostname of the server, initiate a ping to the IP address. Initiating a ping to the IP address allows you to determine if the problem is a result of an issue with hostname resolution. If you are testing connectivity to another VMkernel port on another server remember to use the VMkernel port IP address because the server's hostname usually resolves to the service console address on the remote server.


Vmware Cluster datastore path dead

Symptoms:

  1. have 4 esxi5.1 hosts (esxi01-04) in vmware cluster, VM in two specific datastore cannot start up on esxi01 host
  2. checked the iSCSI Network adapter should be fine as only two datastore have issue
  3. checked the LUN mapping is fine
  4. Try rescan and refresh datatstore didn't solve the issue
  5. Try to disconnect and re-mount the datastore on esxi01 didn't solve the problem . 
  6. For a reason to resume service asap, migrate all VMs to TEMP datastore, unmount and detach the datastore for all esxi host
  7. unmaping LUN on storage side, rescan datastore on vmware cluster the problem LUN should be disappear.
  8. Remap LUN to esxi hosts, add to datasore cluster again, re-signature and format the LUN, put VM on it for testing 

Search through esxi logs and get result below:

standard input)-2014-05-xxT03:09:31.297Z cpu6:8198)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800037536d72502444xxxxxxxxb" state in doubt; requested fast path state update...
(standard input)-2014-05-xxT03:09:31.594Z cpu4:8401)ALERT: NMP: vmk_NmpVerifyPathUID:1167:The physical media represented by device naa.60a9800037536d72502444xxxxxx (path vmhba32:C0:T0:L13) has changed. If this is a data LUN, this is a critical error. Detect
(standard input)-2014-05-xxT03:09:31.594Z cpu4:8401)WARNING: ScsiDevice: 1422: Device :naa.60a9800037536d72502444xxxxxxx has been removed or is permanently inaccessible.
(standard input):2014-05-xxT03:09:32.410Z cpu4:8196)WARNING: HBX: 1548: HB failed due to no connectivity on [HB state abcdef02 offset 4059136 gen 17 stampUS 1847607014728 uuid 535950b3-29e2ce5d-488a-0025b501011f jrnl <FB 70200> drv 14.58] on vol 'XXXXXXX'
(standard input)-2014-05-xxT03:10:50.461Z cpu10:8202)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a9800037536d72502xxxxxxxx" state in doubt; requested fast path state update...
(standard input)-2014-05-xxT03:16:42.380Z cpu15:10515)WARNING: Vol3: 1717: Failed to refresh FS 535daf66-4b90cce6-fa7c-0025b50100ee descriptor: Device is permanently unavailable
(standard input)-2014-05-xxT03:16:42.661Z cpu15:10515)WARNING: Vol3: 1717: Failed to refresh FS 535daf66-4b90cce6-fa7c-0025b50100ee descriptor: Device is permanently unavailable

not sure whether this KB describe the exact issue but at least they got similar symptoms

VMFS Resignature causes thrashing between multiple VMware ESXi 4.x/5.x and ESX 4.x hosts (1026710

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026710

Unmounting a LUN or detaching a datastore/storage device from multiple VMware ESXi 5.x hosts (2004605)
This article provides steps to unmount a LUN from an ESXi 5.x host, which includes unmounting the file system and detaching the device. These steps must be performed for each ESXi host.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004605

Thursday, May 15, 2014

networking device SIM note

For building a Lab consist of number of networks and cloud related products, i a am looking for some Router softwares which can connect different networks as the real router doing, There are some reference link below:

Cisco CSR1000v For Home Labs
http://www.fryguy.net/2013/12/27/cisco-csr1000v-for-home-labs/

Cisco CSR1000V vs the Fabled IOU
http://lamejournal.com/2013/12/28/cisco-csr1000v-vs-fabled-iou/

How to upgrade Titanium VMware image to 6.1.1

http://51sec.blogspot.hk/2013/03/how-to-upgrade-titanium-vmware-image-to.html

Tuesday, April 22, 2014

UCS bug CSCun10760 Legacy CD/DVD” not “Local” or “Remote” in your boot order policy

CSCun10760.  The workaround for this issue is to set "Legacy CD/DVD” not “Local” or “Remote” in your boot order policy (see screenshot)
https://supportforums.cisco.com/discussion/12146621/service-profile-resource-issue

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/2-2/b_UCSM_GUI_Configuration_Guide_2_2/b_UCSM_GUI_Configuration_Guide_2_1_chapter_011111.html#concept_F6AF7F0342224EF28B65ABC2C382FFB1

From Cisco Forum:

Second-level devices are only available for Cisco UCS M3 blade and rack servers using enhanced boot order. ForCisco UCS M1 and M2 blade and rack servers using standard boot order, you can choose only the top-level Add Local Disk.

If you select 'Add Local CD/DVD' or 'Add Remote CD/DVD' the service profile will fail to apply to a non-M3 blade, those options are uefi 2nd level boot.  You must delete any CD/DVD from the service profile, then add the top level Add CD/DVD when Legacy Boot is checked.
CSCun10760 was opened so that second order boot options are greyed out when using legacy boot.

Wednesday, April 16, 2014

Vmware KB 2035510 "Cannot complete login due to an incorrect username or password"


Logging into vCenter Server 5.1 or 5.5 using the vSphere Client with an Active Directory domain account and selecting the Use Windows session credentials checkbox, fails with the error:

Cannot complete login due to an incorrect username or password

Cause:
Active Directory Identity Sources must be added to the Single Sign On (SSO) configuration with the domain NETBIOS (short name) as the Domain Alias. If a Domain Alias is not configured with the domain short name, authentication using session credentials may fail.

Resolution:

To resolve this issue, remove the existing Active Directory Identity Source, and recreate it with a Domain Alias.
To remove the existing Active Directory Identity Source, and recreate it with a Domain Alias:
  1. Log into the vSphere Web Client using the Admin@System-Domain (for 5.1) or administrator@vsphere.local credentials (for 5.5).
  2. Click Administration.
  3. Under Sign-On and Discovery, click Configuration.
  4. Click the Active Directory identity source.
  5. Under Actions, click Edit Identity Source.
  6. Make note of the information in the identity source.
  7. Click Cancel.
  8. Under Actions, click Delete Identity Source.
  9. Recreate the identity source using the short NETBIOS name in the Domain Alias field.
  10. Click Test Connection.
  11. Click OK.

Ref:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035510

Tuesday, April 1, 2014

BugId CSCul20788 UCS Firmware issue

The background:

When installing a new B200 M3 server, it will either fail discovery at 56% or service profile association at 39% with the below:
service profile association at 39%
 Slot Status: Equipped
 Equipped Conn Path: A,B
 Equipped Conn Status: A,B
 Equipped Managing Instance: B
 Availability: Unavailable
 Admin State: In Service
 Overall Status: Config Failure
 Oper Qualifier: N/A
 Discovery: Complete
 Current Task: Waiting for BIOS POST completion from CIMC on server 1/4(FSM-STAGE:sam:dme:ComputePhysicalAssociate:BiosPostCompletion)
 Check Point: Discovered


Conditions:
B200 M3 with E5-2600 V2 series CPU which begins support in UCS 2.1.3a.
UCS infrastructure running 2.1.3a


For service profile association failing at 39%
CIMC==2.1.3a
Bios==2.1.1a
For Discovery failing at 56%L
CIMC==2.1.1e
BIOS==2.1.1a
Workaround:
Correct firmware version for blade to supported version for all components.


Ref:
BugId CSCul20788

UCS Firmware Upgrade: 2.1

http://www.techshifter.com/2013/02/12/ucs-firmware-upgrade-2-1/

Tuesday, March 25, 2014

EMC Networker Note


In the  NetWorker-VMware-7.6-Service-Pack-2-Integration-Guide:

The software and hardware requirements for VADP include:
◆ One or more VADP proxy systems running any of the following operating
systems (English versions only):
• Windows Server 2003 SP1 (32-bit or 64-bit)
• Windows Server 2003 R2 (32-bit or 64-bit)
• Windows Server 2003 SP2 (32-bit or 64-bit)
• Windows 2008 (32-bit or 64-bit)
• Windows 2008 R2


The VADP FLR option has not been qualified with dynamic disks and is not supported by Networker. Here are some of the limitations that are not supported by the Networker FLR option:

  • Virtual Machine OS containing GPT or dynamic disks. 
  • Virtual Machine OS containing uninitialized disks. 
  • Virtual Machine OS containing unformatted partitions 
  • Virtual Machine OS containing partitions without drive letters. 
  • Virtual Machine configuration with Virtual IDE Disk Devices (only SCSI). 
  • Virtual Machine configuration with independant disk mode.



https://community.emc.com/thread/149443?tstart=0

Using vSphere Changed Block Tracking CBT with EMC Networker Backup

Tuesday, March 18, 2014

how to copy tech-support log with Putty


  1. In Putty Configurations > Session >Logging 
  2. Select "All Session output"
  3. Select "Flush log file frequently" will overwrite the log file every time when start the session  
  4. connect the device 
  5. in the EXEC mode of the command prompt, type terminal length 0 which will make the command output at once without interrupt
vsg1#terminal length 0 
vsg1#show tech-support all 

     6.then we will have the output in the Putty log file ^ ^
Ref:
http://ccienet.wordpress.com/2012/01/22/how-to-copy-large-show-tech-support-from-cisco-device-using-putty/

Monday, March 17, 2014

Create persistent scratch location on ESXi host using a VMware vSphere client for using USB boot Esxi

VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap. (This is not a requirement, however.) Persistent scratch space may be provisioned on a FAT 16, VMFS, or NFS partition accessible by the ESXi host.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033696

Using a VMware vSphere client
http://www-01.ibm.com/support/docview.wss?uid=isg3T1013015

Configuring syslog on ESXi 5.x (2003322)

VMware vSphere ESXi 5.x hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk. To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk, and to send the logs across the network to a syslog server.

Thursday, March 13, 2014

Using Power CLI/vim-cmd to upgrade VM version of virtual machine

vim-cmd vmsvc/getallvms
vim-cmd vmsvc/upgradM-ID] vmx-[e [VHW-VERSION]

PowerCLI:
$vm=Get-VM -Name VM-NAME
$vm.Version

Set-Vm -VM (Get-VM -Name [VM-NAME]) -Version v[HW-VERSION]

Ref:
http://www.virtuallyghetto.com/2013/10/quick-tip-using-cli-to-upgrade-to.html

Thursday, March 6, 2014

vCenter vpxd.cfg Advanced Configuration

The post about details in vpxd.cfg


The vpxd.cfg file is located on the VMware vCenter Server by default at %ALLUSERPROFILE%\Application Data\VMware\VMware VirtualCenter\vpxd.cfg

On Windows Server 2008, 
C:\ProgramData\VMware\VMware VirtualCenter\vpxd.cfg
On Windows Server 2003,
C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\vpxd.cfg


http://www.boche.net/blog/index.php/2010/03/13/vpxd-cfg-advanced-configuration/

Wednesday, March 5, 2014

Implementing VMware vSphere 5 Auto Deploy on the Cisco Unified Computing System


This document provides an overview and detailed procedure for installing and configuring the VMware vSphere 5 Auto Deploy feature on a Microsoft Windows server. VMware vSphere is an industry-leading virtualization platform, transforming data centers into dramatically simplified cloud infrastructures and enabling the next generation of flexible, reliable IT services. Cisco Unified Computing System ™ (Cisco UCS ™) and VMware vSphere together enable rapid application and infrastructure deployment with greater security, availability, and performance leading to increased return on investment (ROI) and reduced total cost of ownership (TCO) for computing environments.


http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/whitepaper_c11-701953.html

vCenter Server and ESXi/ESX host build numbers to update levels

get ESX build # command

  • uname -a 
  • esxcli system version get


http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014508

VMware vCenter Server performance and database troubleshooting

The following links provide detailed steps for vCenter server performance troubleshooting:

  • vCenter Server does not start.
  • vCenter Server is slow to respond.
  • vCenter Server fails after an indefinite amount of time.
  • When connected to a Microsoft SQL database, the vCenter Server vpxd.log contains an entry similar to:

    ODBC error: (42000) - [Microsoft][SQL Native Client][SQL Server]Could not allocate space for object <object> in database '<database>' because the '<filegroup>' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup." is returned when executing SQL statement "<sql statement>".
  • When connected to an Oracle database, the vCenter Server vpxd.log contains an entry similar to:
    ODBC error: (HY000) - [Oracle][ODBC][Ora]ORA-01653: unable to extend table <table> by <size> in tablespace <tablespace> is returned when executing SQL statement <SQL Statement>.
  • You see this error when attempting to connect to the SQL database server:
    ODBC error: (08S01) - [Microsoft][SQL Native Client]Named Pipes Provider: No process is on the other end of the pipe.
  • The VirtualCenter Server service cannot authenticate with the database.
  • You cannot connect to the database repository through SQL Server or Oracle.
  • The vCenter Server database is corrupt.
Note: The vpxd.log files are located in %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003979#growth
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003979#slowdb

  • High disk usage of the VMware vCenter Server database
  • High disk usage of the VMware vCenter Server database transactional logs
Note: If you are using Microsoft SQL, there is a way to validate where space is being consumed within a Microsoft SQL database.
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1028356

For troubleshooting or maintenance purposes it may be necessary to defragment the indexes on your Microsoft SQL database server. This article provides the steps perform a defragmentation of the indexes to increase performance of the database.

Due to the database schema enhancements made in vSphere 5.1.x and later versions, defragmentation of the vCenter Server database is no longer required.

SQL Server Integrity Check tools CHECKDB

Checks the logical and physical integrity of all the objects in the specified database by performing the following operations:

  • Runs DBCC CHECKALLOC on the database.
  • Runs DBCC CHECKTABLE on every table and view in the database.
  • Runs DBCC CHECKCATALOG on the database.
  • Validates the contents of every indexed view in the database.
  • Validates link-level consistency between table metadata and file system directories and files when storing varbinary(max) data in the file system using FILESTREAM.
  • Validates the Service Broker data in the database.
Syntax
DBCC CHECKDB 
[
    [ ( database_name | database_id | 0
        [ , NOINDEX 
        | , { REPAIR_ALLOW_DATA_LOSS | REPAIR_FAST | REPAIR_REBUILD } ]
    ) ]
    [ WITH 
        {
            [ ALL_ERRORMSGS ]
            [ , EXTENDED_LOGICAL_CHECKS ] 
            [ , NO_INFOMSGS ]
            [ , TABLOCK ]
            [ , ESTIMATEONLY ]
            [ , { PHYSICAL_ONLY | DATA_PURITY } ]
        }
    ]
]


Ref:
http://technet.microsoft.com/en-us/library/ms176064.aspx

SQL Server Integrity Check – A Necessary Routine

http://www.pythian.com/blog/sql-server-integrity-check-a-necessary-routine/

vCenter Performance Overview Tab fails to display


There are several possible causes for this issue.check the following step by step:
  1. Confirm that the vCenter Web Management Service is running.
  2. To resolve this issue, connect to vCenter locally with a vSphere Client located on the vCenter Server and follow Stopping, starting, or restarting vCenter services.
  3. Ensure that the correct DNS settings and IP address are being used
  4. Disable any third party web services that may be interfering with the vCenter Web Management Services.(port 8443)
  5. Check if vCenter Server is using custom SSL certificates as a result of a recent upgrade to vCenter Server 4.0 Update 1. 
  6. Disable the proxy settings from the browser. 



Ref: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014454

Tuesday, March 4, 2014

Error on SQL Server 2005: "Create maintenance plan failed"

The cause of this error is lack of  Integration Services (SSIS) service when installing SQL server 2005.

When you create the maintenance plan, the following code is run.
EXEC msdb.dbo.sp_update_jobstep @job_name=N'System DB Maintenance Plan', @step_id=1 , @subsystem=N'SSIS', @command=N'/SQL 
"\Maintenance Plans\System DB Maintenance Plan" /SERVER ASHBSQL701 /MAXCONCURRENT " -1 " /CHECKPOINTING OFF'
This code tries to access the Integration Services service.
The solution is to install back the Integration Services from the SQL server 2005 setup CD


Ref: http://support.microsoft.com/kb/909036