Solution

Solved: VDA client unattended install failure.

When you’re solving the same problem after a couple of months it’s probably a good idea to write a post about it so the solution is close at hand. When you perform an unattended install of VDA client from the 7.x XenApp/XenDesktop media the installation may fail, after this first failure all other attempts will fail instantly.

Like every other self respecting IT engineer I always deploy my servers unattended, for this I like to use a product like RES One Automation 2015. When installing the VDA Client for the first time the installation failed, when examining the log file I noticed following lines:

  • XenDesktopSetup:Starting synchronous process ‘C:\Windows\TEMP\wisE4BD.tmp\Support\DotNet451\NDP451-KB2858728-x86-x64-AllOS-ENU.exe’ with args ‘ /norestart /quiet /q:a’

  • XenDesktopSetup:Process completed with error code 3010

  • XenDesktopSetup:Reboot level increased on behalf of ‘Microsoft .NET Framework 4.5.1’ from NoReboot to ImmediateReboot

  • XenDesktopSetup:InstallComponent: Exit (return False)

  • XenDesktopSetup:Install tasks for this session have finished.

  • XenDesktopSetup:Installation partially complete, reboot is required

  • XenDesktopSetup:Installation Manager returned PartialSuccessAndRebootNeeded

So the installation failed because it installed some dependencies from the support folder, and needed a reboot.

After rebooting the machine, I launched the installation again. Surprisingly it fail instantly and each and other try I did the result was the same. when examining the log file again I noticed the following error.

  • XenDesktopSetup:VerifyCDRoot: Failed to find the MediaID file at ‘C:\Windows\TEMP\wisE4BD.tmp\MediaId_F8D0A1D9-128F-4a86-842D-C61294BA0A02_x64’

  • XenDesktopSetup:VerifyCDRoot: Failed to find the MediaID file at ‘C:\Windows\TEMP\wisE4BD.tmp\x64\x64\MediaId_F8D0A1D9-128F-4a86-842D-C61294BA0A02_x64’

  • XenDesktopSetup:OleSafeThread: Install media not found at ‘C:\Windows\TEMP\wisE4BD.tmp\x64’. Exiting.

  • XenDesktopSetup:Installation media cannot be found

I know the wisxxxx.tmp folder is being automatically generated by RES One Automation, and changes each time. So I copied the source files to a fixed location and launched the installation again but still the log file states the same error message, when look up in the logfile I noticed the line

  • CDRoot = C:\Windows\TEMP\wisE4BD.tmp\x64

So the old CDRoot information is saved somewhere and causes the problem, after some search I found out the after the first installation a Citrix folder was created in the %programdata% location. In this location are some files I did not need. So I deleted this folder and tried again. Now the installation succeeded.

 

 

 

 

Solution

Solved: Netscaler VPX (on VMWare) Problem.

During my latest project I ran into a problem where Netscaler VPX stops responding to any sort of request.

  • Logon to Netscalers management interface: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway VPN server: Netscaler immediately stops responding.
  • Connect to Netscalers Access Gateway Logon Interface: Netscaler immediately stops responding.

The state of the netscaler is as follows:

  • Netscaler is no longer reachable via management Interfaces (SSL/SSH, etc)
  • Netscaler is not replying to ping requests (to NSIP)
  • Netscaler IS reachable and response via VMWare console

Via WMWare console the netscaler is responsive and when issueing command “Show Interface” netscaler responds by listing al it’s network interfaces. I thing I noticed that the interface of the NSIP was shutdown because of administrative reasons (cannot recall the exact message). When enabling this Interface with command: enable interface 0/1 everything seemed to be working again until you try one of earlier mentioned actions.

If you experience this, there is a good change that the VMWare server on which the Netscaler VPX is running was upgraded with patches from VMware ESXi 5.5.0 U2 both VMWware and Citrix Have released KB documents about this issue:

from VMware document we learn more about the issue:

  • This issue occurs when the NetScaler virtual machine driver resets TDT to 0 after 511 while the TX ring size is shown as 1024.
  • This is not a VMware issue. To resolve this issue, upgrade the NetScaler appliance.

According to Citrix there are 3 workaround

  1. Revert to a non updated VMWware host (not recommended)
  2. Upgrade to NetScaler 10.5 build 55.8 or above (recommended if possible)
  3. change the TX ringsize (last option if no other option works out)

from the document we can extract the procedure:

  1. SSH and log on to Citrix NetScaler VPX appliance as nsroot.
  2. Type shell.
  3. Change directory (cd) to /flash/boot.
  4. Create file /flash/boot/loader.conf.local (if not present) with same permissions as /flash/boot/loader.conf. Add the following line and reboot:
    hw.em.txd=512
    Note: To create the file, use command touch loader.conf.local.

vi Commands

The following are the vi commands to edit the document:

  1. From NetScaler shell type:
    vi <filename>
  2. Move the cursor to the last character of text in the file, type “a” and click Enter.
  3. Type the line:
    hw.em.txd=512
  4. Press the ESC key and then “:” key. The cursor will move to the bottom of the page, then type wq!.

 

After this procedure reboot the netscaler and all should be working fine again.

Solution

HOW TO: Netscaler 10.5 – Change Password / Secure LDAP configuration

During my current project I had to build a Netscaler cluster for Access Gateway functionality. After initially configuring the Access Gateway vServer I noticed that user account that are marked for “Change Password on next Logon” could not authenticate not via Access Gateway Logon Page nor Via Dell Wyse ThinClient that are configured for StoreFront access (via Netscaler Access Gateway). Password Change direct via StoreFront 2.6 was working flawlessly. After some googling I managed to get this working I followed these steps to succesfully configure “Allow Password Change” via Netscaler.

  1. Create Root CA on Netscaler
  2. Enable Secure LDAP on domain controllers (without Microsoft CA services)
  3. Configure Authentication Server object on Netscaler
  4. Configure Access Gateway vServer on the Netscaler

1. Create Root CA, RSA Key

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create RSA Key

Enter required information:

  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Size = 4096
  • Public Exponent Value = 3
  • Key Format = PEM
  • PEM Encoding Algorithm = DES3
  • PEM Passphrase = somepassword (2x)
create ssl rsa /nsconfig/ssl/RootCA.key 4096 -keyform PEM -des3 -password somepassword

2. Create Certificate Signing Request

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create Certificate Signing Request (CSR)

Enter required information:

  • Request File Name = /nsconfig/ssl/RootCA.csr
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Format = PEM
  • PEM Passphrase (For Encrypted Key) = Earlier created password
  • Country = Enter desired value
  • State or Province = Enter desired value
  • Organization Name = Enter desired value
  • Common Name = Enter desired value (This name is visible on the certificate)
create ssl certreq /nsconfig/ssl/RootCA.csr -keyFile /nsconfig/ssl/RootCA.key -keyform PEM -CountryName US -StateName NY -OrganizationName PrivateInc -commonName RootCAPrivateInc

3. Create RootCA Certificate

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > Traffic Management > SSL and click Create Certificate

Enter required information:

  • Enter Certificate File name  = /nsconfig/ssl/RootCA.cer
  • Certificate Format = PEM
  • Certificate Type = Root-CA
  • CSR File = /nsconfig/ssl/RootCA.csr
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Key Format = PEM
  • PEM Passphrase = Earlier created .key password
  • Validity Period = 365 (max 3650)

Now import this new certificate to the Netscaler Certificates store.

  • Browse to NetScaler > Traffic Management > SSL > Certificates and click Install
  • Certificate-Key Pair Name = “Name it you like
  • Certificate File Name = /nsconfig/ssl/RootCA.cer
  • Key File Name = /nsconfig/ssl/RootCA.key
  • Certificate Format = PEM
  • Password = Earlier created .key password
  • Certificate Bundle = Enabled

 

4. Enable Secure LDAP on domain controllers  

After we created the RootCA account we need to enable secure LDAP om the domain controllers. For this to work we need to create a CSR on the Domain Controllers. To do this you need to login on (all) your domain controller(s) and create a CSR. Copy the contents of this file to notepad and name the file request.inf. Save the file to c:\windows\temp

;----------------- request.inf ----------------- 

[Version] 

Signature="$Windows NT$" 

[NewRequest]

Subject = "CN=servername.domain.com"
KeySpec = 1 
KeyLength = 4096 
Exportable = TRUE 
MachineKeySet = TRUE 
SMIME = False 
PrivateKeyArchive = FALSE 
UserProtected = FALSE 
UseExistingKeySet = FALSE 
ProviderName = "Microsoft RSA SChannel Cryptographic Provider" 
ProviderType = 12
RequestType = PKCS10 
KeyUsage = 0xa0 

[EnhancedKeyUsageExtension] 

OID=1.3.6.1.5.5.7.3.1 

;-----------------------------------------------

 

now open a command-prompt (elevated rights) and issue the command:

cd \windows\temp
certreq -new c:\request.inf servername.csr

Upload  the newly created CSR file(s) to the Netscaler /nsconfig/ssl/ folder (via Manage Certificate > Upload)

5. Sign Domain Controller certificate.

Now Sign this CSR with the RootCA certificate created earlier via Create Certificate

  • Enter Certificate File name  = /nsconfig/ssl/servername.cer
  • Certificate Format = PEM
  • Certificate Type = SERVER
  • CSR File = /nsconfig/ssl/servername.csr
  • Key Format = PEM
  • Validity Period = 365 (max 3650)
  • Key Filename = /nsconfig/ssl/RootCA.key
  • CA Certificate File Name = /nsconfig/ssl/RootCA.cer
  • CA Certificate File format = PEM
  • CA Key File Name = /nsconfig/ssl/RootCA.key
  • CA Key File Format = PEM
  • PEM Passphrase = RootCA PEM Password
  • CA Serial File Number = /nsconfig/ssl/servername.serial

 6. Export RootCA as PFX

On Netscaler SSL Administration page click Export PKCS#12

  • PKCS12 File Name = /nsconfig/ssl/RootCA.pfx
  • Certificate File Name = /nsconfig/ssl/RootCA.cer
  • Key Filename = /nsconfig/ssl/RootCA.key
  • Export Password =  thisisanewpassword
  • PEM Passphrase = RootCA PEM Password

7. Import RootCA Certificate on Domain Controllers

Download the PFX file to C:\windows\temp on your domain controller and import it to the Trusted Root part of the Machine Certificate Store. When importing select “mark this key as exportable” repeat this step on all your Domain Controllers.

8. Import Server Certificate

Download the servername.cer file from the Netscaler to the domaincontroller c:\windows\temp, open command-prompt with elevated rights and issue command(s)

cd \windows\temp
certreq -accept servername.cer

Test your secure ldap with ldp.exe, select connect and enter the servername on which you just imported the certificates. Use port 636, and check the SSL option.

 

9. Configure Authentication Server object on Netscaler

  • Login to Netscaler Administration Console.
  • Browse to NetScaler > System > Authentication > LDAP and in the right pane click Servers and then Add.

Enter required information make sure to use these settings to enable secure LDAP connections:

  • Security Type: SSL
  • Port: 636
  • DO NOT enable “Validate LDAP Server Certificate”
  • Allow Password Change: Enabled

Bind this new server object to a Authentication Policy.

10. Configure Access Gateway vServer on the Netscaler

Open your Access Gateway vServer:

  • Bind Root Certificate as to CA Certificates
  • Bind the secure LDAP Autentication Policy to the vServer.
  • Enable “Client Authentication” in “SSL Parameters

 

All is done, now test it and hopefully this works for you too.

 

Sources

Microsoft KB321051

 

Solution

Windows 2012 R2: Black logonscreen after App-V 5 (SP2) Client installation

While I was building a proof of concept for my customer I ran into some issues that where seemingly unrelated. I was building a brand new Windows 2012 R2 / XenApp 7.5 environment including RES Automation Manager, RES Workspace Manager and App-V 5 (SP2). After installing and testing the core XenApp infrastructure, I started to configure the XenApp (Worker/Application Servers) this is when I ran into problems. After installing the App-V (SP2) client and rebooting the server I was no longer able to logon to the Published Desktop I’ve created and successfully tested earlier.

The Citrix receiver keeps hanging on a black logon screen (did not launch pfwsmgr.exe) and after a while the session was ended. I knew there was an issue on Windows 2008 R2 that resulted in a black screen only this was merely a cosmetic issue, because in Windows 2008 logon succeeded and a desktop was presented. To solve this issue you had to create the registry key:

Registry Key: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Logon
Name: DisableStatus
Type: REG_DWORD
Value: 00000001

Source

As I was not sure if this still applies to Windows Server 2012 R2 I decided to add this registry key to the application server. After logging on Windows revealed its error message that was not show before (User profile service). The EventVwr error message corresponding to this visual error contains:

Windows cannot copy file \C:\Users\Default\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\Control Panel.lnk to location \?\C:\Users[userid]\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch\Control Panel.lnk. This error may be caused by network problems or insufficient security rights.

DETAIL – The process cannot access the file because it is being used by another process.

This error led me to following knowledge base document: 2985344

This document addresses problems that some may experience after installation of hotfix KB2919355. The document also links to  hotfix 5 for App-V 5 (SP2) Client after installing this hotfix all was working again.

Netscaler Patching and Heartbleed Information

After the publicity of the latest OpenSSL vulnerability CVE-2014-0160 (aka Heartbleed). I’ve investigated if Netscaler’s firmware was vulnerable to this vulnerability. Soon I found out that Netscaler’s software is NOT vulnerable. But during my investigation I noticed that the OpenSSL implementation is very old (2004). This raised some questions like:

  • Why this old version is still actively used by Netscaler firmware?
  • Is OpenSSL used for SSL Authentication?
  • Is this version subject to all vulnerabilities that where discovered since 2004?

After some discussions on Citrix Forums, finally today a conclusive statement was given by Citrix Netscaler product manager Steve Shah.

Starting with the most important detail: NetScaler is NOT vulnerable to CVE-2014-0160. We have an up to date set of information at http://support.citrix.com/article/CTX140605

There seems to be some confusion around our SSL implementation and our patch strategy as well.

The NetScaler has two halves: the Internet-facing front half which uses our own SSL stack and is not vulnerable to OpenSSL. Our SSL stack is not OpenSSL. The second half is the management half which does use OpenSSL.

We validate our internal SSL stack against all SSL vulnerabilities posted against all SSL stacks out there.

For our OpenSSL implementation, we keep it patched for all the latest security issues, but not for the latest features. This is done to mitigate risk and maintain a stable code base where possible. Because the features do not change, we leave the version number of OpenSSL alone (openssl version) so that components that look at the version number to determine which API to use behave themselves.

We use this strategy across all of the relevant opensource we use, including OpenSSH. This leads to false positives from some security scanners that only use the version number to determine if a stack is vulnerable.

Juniper SA VPN Appliance vs. Heartbleed OpenSSL vulnerability

According to some tests on SSL Labs Juniper SA appliances (and client software) ARE vulnerable to the OpenSSL Heartbleed vulnerability. This is confirmed by Juniper in this support document. According to Juniper’s support document:

PROBLEM:
The TLS and DTLS implementations in OpenSSL 1.0.1 before 1.0.1g do not properly handle Heartbeat Extension packets, which allows remote attackers to obtain sensitive information (such as private keys, username and passwords, or contents of encrypted traffic) from process memory via crafted packets that trigger a buffer over-read. This issue is also known as The Heartbleed Bug.

Status of different OpenSSL versions:
OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
OpenSSL 1.0.1g is NOT vulnerable
OpenSSL 1.0.0 branch is NOT vulnerable
OpenSSL 0.9.8 branch is NOT vulnerable

Vulnerable Products

  • Junos OS 13.3R1 (Fixed code is listed in the “Solution” section)
  • SSL VPN (IVEOS) 7.4r1 and later, and SSL VPN (IVEOS) 8.0r1 and later (Fixed code is listed in the “Solution” section)
  • UAC 4.4r1 and later, and UAC 5.0r1 and later (Fixed code is listed in the “Solution” section)
  • Junos Pulse (Desktop) 5.0r1 and later, and Junos Pulse (Desktop) 4.0r5 and later (Fixed code is listed in the “Solution” section)
  • Network Connect (windows only) version 7.4R5 to 7.4R9.1 & 8.0R1 to 8.0R3.1. (This client is only impacted when used in FIPS mode.) (Fixed code is listed in the “Solution” section)
  • Junos Pulse (Mobile) on Android version 4.2R1 and higher. (Fixed code is listed in the “Solution” section)
  • Junos Pulse (Mobile) on iOS version 4.2R1 and higher. (This client is only impacted when used in FIPS mode.) (Fixed code is listed in the “Solution” section)
  • WebApp Secure (Fixed code is listed in the “Solution” section)
  • Odyssey client 5.6r5 and later

Source

Netscaler vs CVE-2014-0160 (Heartbleed) OpenSSL vulnerability

Yesterday a lot of attention was created about the latest OpenSSL vulnerability (CVE-2014-0160). This vulnerability exposes a lot of SSL implementations to a great risk because OpenSSL is a very popular SSL implementation and used in a great range of Unix/Linux based application and appliances.

Being very busy with Citrix Netscaler lately I immediately recognized the great potential risk of this vulnerability because Netscaler Firmware also uses this OpenSSL implementation. So I investigated this risk based on my own up-to-date netscaler firmware (124.13) to find out if this firmware version and possible older versions are vulnerable to this  CVE-2014-0160 (Heartbleed) bug.

  • 1st test I did was browsing to a site that checks your site for this specific vulnerability  http://filippo.io/Heartbleed the result of the test was not very conclusive “write tcp xxx.xxx.xxx.xxx:443: broken pipe”

After this check I wondered which versions of OpenSSL are affected by this vulnerability according to OpenSSL.org own site the vulnerability exists in versions: 1.0.1f, 1.0.1e, 1.0.1d, 1.0.1c, 1.0.1b, 1.0.1a, 1.0.1

So my immediately I logged in to Netscaler’s SSH console and entered the following commands:

  •  > shell
  • # openssl version

This command resulted in the OpenSSL response: “OpenSSL 0.9.7e-p1 25 Oct 2004”

So i’m very glad to see that the latest version of Netscaler’s  firmware 124.13 does not contain this vulnerability. However I’m shocked by the ancient version of OpenSSL (release date 25 Oct 2004!!!!) that is used by this latest Netscaler firmware. There is a whole list of vulnerabilities that have been repaired since.

Update 1 09-042014 : Citrix’s security team seams to confirm that Netscaler is not at risk. A public statement has not yet been released.

Update 2 09-042014 : Citrix now officially announces Citrix Netscaler/Access Gateway/StoreFront products are NOT vulnerable to CVE-2014-0160 the Citrix support document can be found here

Shame on Me!

Shame on me, it has been more than a year between blogposts. It has been a busy year so that’s good. Hopefully I will find some time next weeks to start writing again.

Powershell

SCRIPT: Check ICA Listener response

Last week I was troubleshooting at a customer which was experiencing some “random” periods in which end users could not login at all. When looking at the Load Evaluator I noticed the least loaded server in those periods were the same. When trying to logon to the server through ICA, the server sits forever at connecting…

Instantly I opened a DOS prompt and did a telnet to the servers ICA port (1494).  The server responded and a connection was made, but when I was expecting to see the ICA heartbeat the telnet session stayed blank, this told me the ICA listener is corrupted. I rebooted the server and I came up perfectly. Problem solved you think, not quiet yet.

The infrastructure consisted of 100+ XenApp servers which are provisioned by Citrix Provisioning Services. So potentially more XenApp servers were having a corrupted ICA listener. But entering a telnet session for each server manually was not an option. and scripting a powershell script which launches an external (telnet) process did was not sufficient either. (because is does not report, the telnet output back to powershell). That’s why I searched the internet how to read a “telnet” sessions output. while searching I stumbled upon a blog that needed to do some telnet magic on Cisco devices. I decided to adapt the script for my own needs

After some alteration I ended up with a script that connects to the ICA port (1494) and reports it’s output. I’ve tested it and it work really fine, but you might have to change the TimeOut settings. When a server reports a potential issue you can test it against a manual telnet session.

  •  Test a Single Computer
  • Test-ICA -ComputerName <your computername here> -TimeOut <Time Out in ms>
  •  Test Multiple Computers
  •  101..190 |% {.\Test-ICA.ps1 -ComputerName CTX$_ -TimeOut <Time Out in ms>}

You can also filter output for your needs. Let me know if you appreciate the script.

Download Test-ICA (2284)

 

Solution

Send Message to User Limitations

Today I received a question from a customer who wanted to send “Large” message to all XenApp users in the farm. They asked me what the maximum number of characters was that you can send to a user. I honestly did not have a clue and answered to the client that I was not aware of any limitation but that I did not have any problems even with longer messages. The customer was happy with my answer and did send its “Large” message to their XenApp 6 sessions.

The question got stuck in my mind and I decided I had to find out if there were limitations. So I prepared a 1000 character sentence and send it with Citrix Delivery Console, result was the message popped up to the session as it supposed to. So I tried a 5000 character message and it did not pop-up.

  • 1000 characters,No problems both CDC as Send-XASessionMessage cmdlet
  • 2000 characters,No problems both CDC as Send-XASessionMessage cmdlet
  • 3000 characters,No problems both CDC as Send-XASessionMessage cmdlet
  • 4000 characters,No problems both CDC as Send-XASessionMessage cmdlet
  • 5000 characters, both CDC as Send-XASessionMessage cmdlet the message did not arrive.

After some narrowing down,  the maximum character in a message send with:

  • Citrix Delivery Console   = 4055
  • Send-XASessionMessage= 4094

I’m not sure why the limit is not the same, but that the conclusion I ended up with. So if you send a message to your user, Keep it short 😉

Cheerio!

 

1 2 3 4