A Presentation Server 4.5 server may stop accepting new connections, and existing connections may stop responding. Although slow, the console can be logged onto, but appears to be in a deadlocked condition.

A particular symptom of this issue is that after logging onto the console and starting Task Manager, it only appears in the System Tray, and not as a full window.

This is typically seen on high utilisation Citrix servers, such as virtual machines, although not exclusively.

This issue occurs because of two root causes.

1) Microsoft
This issue occurs because of incorrect synchronization within a very small timing window in the Terminal Services device driver (TermDD.sys). Resolved by MSKB956438.

2) Citrix
A Citrix thread runs out of terminal service kernel mode output buffers (OUTBUF), while holding the connection resource and indirectly causing the terminal service threads to hang (mod_cgp.so module of the Citrix XTE service). Resolved by CTX123083.

Update: The limited release CTX123083 has been superseded by a general release CTX123956 for R05 or Hotfix Rollup Pack 6 (CTX122756)

To determine if this is caused by the Citrix issue, restart the Citrix XTE service. If the XTE service is the issue, the server will become busy as it processes the queue, then users will be able to re-connect to their sessions that have become disconnected due to the XTE Services role in Session Reliability. This can be worked around by disabling the XTE service (and therefore Session Reliability).

Hotfix Rollup Pack 6 for Citrix XenApp 5.0 and Citrix Presentation Server 4.5 for Microsoft Windows Server 2003 32-bit Edition
http://support.citrix.com/article/CTX122756

Hotfix PSE450R05W2K3026 - For Citrix Presentation Server 4.5 and XenApp 5.0 for Windows Server 2003 x86
http://support.citrix.com/article/CTX123956

A Windows Server 2003-based or Windows Server 2008-based terminal server stops accepting new connections, and existing connections stop responding
http://support.microsoft.com/kb/956438

SUPERSEDED: LIMITED RELEASE - Hotfix PSE450R05W2K3003 - For Citrix Presentation Server 4.5 and XenApp 5.0 for Windows Server 2003 x86
http://support.citrix.com/article/CTX123083