Solved: Netscaler VPX (on VMWare) Problem.

During my latest project I ran into a problem where Netscaler VPX stops responding to any sort of request.

  • Logon to Netscalers management interface: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway VPN server: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway Logon Interface: Netscaler immediately stops responding.

The state of the netscaler is as follows:

  • Netscaler is no longer reachable via management Interfaces (SSL/SSH, etc)
  • Netscaler is not replying to ping requests (to NSIP)
  • Netscaler IS reachable and response via VMWare console

Via WMWare console the netscaler is responsive and when issueing command “Show Interface” netscaler responds by listing al it’s network interfaces. I thing I noticed that the interface of the NSIP was shutdown because of administrative reasons (cannot recall the exact message). When enabling this Interface with command: enable interface 0/1 everything seemed to be working again until you try one of earlier mentioned actions.

If you experience this, there is a good change that the VMWare server on which the Netscaler VPX is running was upgraded with patches from VMware ESXi 5.5.0 U2 both VMWware and Citrix Have released KB documents about this issue:

from VMware document we learn more about the issue:

  • This issue occurs when the NetScaler virtual machine driver resets TDT to 0 after 511 while the TX ring size is shown as 1024.
  • This is not a VMware issue. To resolve this issue, upgrade the NetScaler appliance.

According to Citrix there are 3 workaround

  1. Revert to a non updated VMWware host (not recommended)
  2. Upgrade to NetScaler 10.5 build 55.8 or above (recommended if possible)
  3. change the TX ringsize (last option if no other option works out)

from the document we can extract the procedure:

  1. SSH and log on to Citrix NetScaler VPX appliance as nsroot.
  2. Type shell.
  3. Change directory (cd) to /flash/boot.
  4. Create file /flash/boot/loader.conf.local (if not present) with same permissions as /flash/boot/loader.conf. Add the following line and reboot:
    hw.em.txd=512
    Note: To create the file, use command touch loader.conf.local.

vi Commands

The following are the vi commands to edit the document:

  1. From NetScaler shell type:
    vi <filename>
  2. Move the cursor to the last character of text in the file, type “a” and click Enter.
  3. Type the line:
    hw.em.txd=512
  4. Press the ESC key and then “:” key. The cursor will move to the bottom of the page, then type wq!.

 

After this procedure reboot the netscaler and all should be working fine again.

Leave a Reply