Netscaler

Solution

Solved: Netscaler VPX (on VMWare) Problem.

During my latest project I ran into a problem where Netscaler VPX stops responding to any sort of request.

  • Logon to Netscalers management interface: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway VPN server: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway Logon Interface: Netscaler immediately stops responding.

The state of the netscaler is as follows:

  • Netscaler is no longer reachable via management Interfaces (SSL/SSH, etc)
  • Netscaler is not replying to ping requests (to NSIP)
  • Netscaler IS reachable and response via VMWare console

Via WMWare console the netscaler is responsive and when issueing command “Show Interface” netscaler responds by listing al it’s network interfaces. I thing I noticed that the interface of the NSIP was shutdown because of administrative reasons (cannot recall the exact message). When enabling this Interface with command: enable interface 0/1 everything seemed to be working again until you try one of earlier mentioned actions.

If you experience this, there is a good change that the VMWare server on which the Netscaler VPX is running was upgraded with patches from VMware ESXi 5.5.0 U2 both VMWware and Citrix Have released KB documents about this issue:

from VMware document we learn more about the issue:

  • This issue occurs when the NetScaler virtual machine driver resets TDT to 0 after 511 while the TX ring size is shown as 1024.
  • This is not a VMware issue. To resolve this issue, upgrade the NetScaler appliance.

According to Citrix there are 3 workaround

  1. Revert to a non updated VMWware host (not recommended)
  2. Upgrade to NetScaler 10.5 build 55.8 or above (recommended if possible)
  3. change the TX ringsize (last option if no other option works out)

from the document we can extract the procedure:

  1. SSH and log on to Citrix NetScaler VPX appliance as nsroot.
  2. Type shell.
  3. Change directory (cd) to /flash/boot.
  4. Create file /flash/boot/loader.conf.local (if not present) with same permissions as /flash/boot/loader.conf. Add the following line and reboot:
    hw.em.txd=512
    Note: To create the file, use command touch loader.conf.local.

vi Commands

The following are the vi commands to edit the document:

  1. From NetScaler shell type:
    vi <filename>
  2. Move the cursor to the last character of text in the file, type “a” and click Enter.
  3. Type the line:
    hw.em.txd=512
  4. Press the ESC key and then “:” key. The cursor will move to the bottom of the page, then type wq!.

 

After this procedure reboot the netscaler and all should be working fine again.

Solution

HOW TO: Netscaler 10.5 – Change Password / Secure LDAP configuration

During my current project I had to build a Netscaler cluster for Access Gateway functionality. After initially configuring the Access Gateway vServer I noticed that user account that are marked for “Change Password on next Logon” could not authenticate not via Access Gateway Logon Page nor Via Dell Wyse ThinClient that are configured for StoreFront access (via Netscaler Access Gateway). Password Change direct via StoreFront 2.6 was working flawlessly. After some googling I managed to get this working I followed these steps to succesfully configure “Allow Password Change” via Netscaler.

  1. Create Root CA on Netscaler
  2. Enable Secure LDAP on domain controllers (without Microsoft CA services)
  3. Configure Authentication Server object on Netscaler
  4. Configure Access Gateway vServer on the Netscaler

1. Create Root CA, RSA Key

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create RSA Key

Enter required information:

  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Size = 4096
  • Public Exponent Value = 3
  • Key Format = PEM
  • PEM Encoding Algorithm = DES3
  • PEM Passphrase = somepassword (2x)
create ssl rsa /nsconfig/ssl/RootCA.key 4096 -keyform PEM -des3 -password somepassword

2. Create Certificate Signing Request

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create Certificate Signing Request (CSR)

Enter required information:

  • Request File Name = /nsconfig/ssl/RootCA.csr
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Format = PEM
  • PEM Passphrase (For Encrypted Key) = Earlier created password
  • Country = Enter desired value
  • State or Province = Enter desired value
  • Organization Name = Enter desired value
  • Common Name = Enter desired value (This name is visible on the certificate)
create ssl certreq /nsconfig/ssl/RootCA.csr -keyFile /nsconfig/ssl/RootCA.key -keyform PEM -CountryName US -StateName NY -OrganizationName PrivateInc -commonName RootCAPrivateInc

3. Create RootCA Certificate

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create Certificate

Enter required information:

  • Enter Certificate File name  = /nsconfig/ssl/RootCA.cer
  • Certificate Format = PEM
  • Certificate Type = Root-CA
  • CSR File = /nsconfig/ssl/RootCA.csr
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Format = PEM
  • PEM Passphrase = Earlier created .key password
  • Validity Period = 365 (max 3650)

Now import this new certificate to the Netscaler Certificates store.

  • Browse to NetScaler > Traffic Management > SSL > Certificates and click Install
  • Certificate-Key Pair Name = “Name it you like
  • Certificate File Name = /nsconfig/ssl/RootCA.cer
  • Key File Name = /nsconfig/ssl/RootCA.key
  • Certificate Format = PEM
  • Password = Earlier created .key password
  • Certificate Bundle = Enabled

 

4. Enable Secure LDAP on domain controllers  

After we created the RootCA account we need to enable secure LDAP om the domain controllers. For this to work we need to create a CSR on the Domain Controllers. To do this you need to login on (all) your domain controller(s) and create a CSR. Copy the contents of this file to notepad and name the file request.inf. Save the file to c:\windows\temp

;----------------- request.inf ----------------- 

[Version] 

Signature="$Windows NT$" 

[NewRequest]

Subject = "CN=servername.domain.com"
KeySpec = 1 
KeyLength = 4096 
Exportable = TRUE 
MachineKeySet = TRUE 
SMIME = False 
PrivateKeyArchive = FALSE 
UserProtected = FALSE 
UseExistingKeySet = FALSE 
ProviderName = "Microsoft RSA SChannel Cryptographic Provider" 
ProviderType = 12
RequestType = PKCS10 
KeyUsage = 0xa0 

[EnhancedKeyUsageExtension] 

OID=1.3.6.1.5.5.7.3.1 

;-----------------------------------------------

 

now open a command-prompt (elevated rights) and issue the command:

cd \windows\temp
certreq -new c:\request.inf servername.csr

Upload  the newly created CSR file(s) to the Netscaler /nsconfig/ssl/ folder (via Manage Certificate > Upload)

5. Sign Domain Controller certificate.

Now Sign this CSR with the RootCA certificate created earlier via Create Certificate

  • Enter Certificate File name  = /nsconfig/ssl/servername.cer
  • Certificate Format = PEM
  • Certificate Type = SERVER
  • CSR File = /nsconfig/ssl/servername.csr
  • Key Format = PEM
  • Validity Period = 365 (max 3650)
  • Key Filename = /nsconfig/ssl/RootCA.key
  • CA Certificate File Name = /nsconfig/ssl/RootCA.cer
  • CA Certificate File format = PEM
  • CA Key File Name = /nsconfig/ssl/RootCA.key
  • CA Key File Format = PEM
  • PEM Passphrase = RootCA PEM Password
  • CA Serial File Number = /nsconfig/ssl/servername.serial

 6. Export RootCA as PFX

On Netscaler SSL Administration page click Export PKCS#12

  • PKCS12 File Name = /nsconfig/ssl/RootCA.pfx
  • Certificate File Name = /nsconfig/ssl/RootCA.cer
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Export Password =  thisisanewpassword
  • PEM Passphrase = RootCA PEM Password

7. Import RootCA Certificate on Domain Controllers

Download the PFX file to C:\windows\temp on your domain controller and import it to the Trusted Root part of the Machine Certificate Store. When importing select “mark this key as exportable” repeat this step on all your Domain Controllers.

8. Import Server Certificate

Download the servername.cer file from the Netscaler to the domaincontroller c:\windows\temp, open command-prompt with elevated rights and issue command(s)

cd \windows\temp
certreq -accept servername.cer

Test your secure ldap with ldp.exe, select connect and enter the servername on which you just imported the certificates. Use port 636, and check the SSL option.

 

9. Configure Authentication Server object on Netscaler

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > System > Authentication > LDAP and in the right pane click Servers and then Add.

Enter required information make sure to use these settings to enable secure LDAP connections:

  • Security Type: SSL
  • Port: 636
  • DO NOT enable “Validate LDAP Server Certificate”
  • Allow Password Change: Enabled

Bind this new server object to a Authentication Policy.

10. Configure Access Gateway vServer on the Netscaler

Open your Access Gateway vServer:

  • Bind Root Certificate as to CA Certificates
  • Bind the secure LDAP Autentication Policy to the vServer.
  • Enable “Client Authentication” in “SSL Parameters

 

All is done, now test it and hopefully this works for you too.

 

Sources

Microsoft KB321051

 

Netscaler vs CVE-2014-0160 (Heartbleed) OpenSSL vulnerability

Yesterday a lot of attention was created about the latest OpenSSL vulnerability (CVE-2014-0160). This vulnerability exposes a lot of SSL implementations to a great risk because OpenSSL is a very popular SSL implementation and used in a great range of Unix/Linux based application and appliances.

Being very busy with Citrix Netscaler lately I immediately recognized the great potential risk of this vulnerability because Netscaler Firmware also uses this OpenSSL implementation. So I investigated this risk based on my own up-to-date netscaler firmware (124.13) to find out if this firmware version and possible older versions are vulnerable to this  CVE-2014-0160 (Heartbleed) bug.

  • 1st test I did was browsing to a site that checks your site for this specific vulnerability  http://filippo.io/Heartbleed the result of the test was not very conclusive “write tcp xxx.xxx.xxx.xxx:443: broken pipe”

After this check I wondered which versions of OpenSSL are affected by this vulnerability according to OpenSSL.org own site the vulnerability exists in versions: 1.0.1f, 1.0.1e, 1.0.1d, 1.0.1c, 1.0.1b, 1.0.1a, 1.0.1

So my immediately I logged in to Netscaler’s SSH console and entered the following commands:

  •  > shell
  • # openssl version

This command resulted in the OpenSSL response: “OpenSSL 0.9.7e-p1 25 Oct 2004”

So i’m very glad to see that the latest version of Netscaler’s  firmware 124.13 does not contain this vulnerability. However I’m shocked by the ancient version of OpenSSL (release date 25 Oct 2004!!!!) that is used by this latest Netscaler firmware. There is a whole list of vulnerabilities that have been repaired since.

Update 1 09-042014 : Citrix’s security team seams to confirm that Netscaler is not at risk. A public statement has not yet been released.

Update 2 09-042014 : Citrix now officially announces Citrix Netscaler/Access Gateway/StoreFront products are NOT vulnerable to CVE-2014-0160 the Citrix support document can be found here