Monday, March 26, 2012

Tcpsvcs.exe process memory usage continuously growing

In my case Wndows Server 2003 DC with dhcp server and DNS dynamic update enabled, tcpsvcs.exe process memory usage was continuously growing. Here are the counters for handle count and private bytes for tcpsvsc.exe, after reboot of the server:


After 30+ days, tcpsvcs.exe has occupied around 500MB:


After applying microsoft hotfix for this issue kb 939928 http://support.microsoft.com/kb/939928/en-us , tcpsvcs.exe memory leak was fixed. After 30+ days, tcpsvcs.exe process has occupied only ~ 13MB :

Tuesday, March 20, 2012

How to debug IIS 7.5 application pool (w3wp.exe) crashes

In my case IIS 7.5 was hosting a faulty application (framework 4) that was causing it's application pool to crash. Developers were unable to find out what was causing the application pool to crash. Two error event messages were logged, one in application and one in system log.

Application Log error event with event id 1000 from source Application Error :
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: oci.dll, version: 11.2.0.1, time stamp: 0x4bb1da76
Exception code: 0xc00000fd
Fault offset: 0x000000000006f837
Faulting process id: 0x11a4
Faulting application start time: 0x01cd02af9bce5a0e
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: c:\<oracle client installation path>\oci.dll
Report Id: d8925b5a-6ea9-11e1-906a-00215e63edd4

System Log warning event with event id 5011 from source WAS:
A process serving application pool 'name of the application pool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4084'. The data field contains the error number.

After the error event 1000 in application log there is information event containing minidump files for debugging from source Windows error reporting:

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: oci.dll
P5: 11.2.0.1
P6: 4bb1da76
P7: c00000fd
P8: 000000000006f837
P9:
P10:

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_....

To debug with wndbg x64 download and install windows sdk from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8442

After installing windbg, open the crash dump file. First, you can setup symbol path files (local and internet):

.sympath SRV*C:\localsymbols*http://msdl.microsoft.com/download/symbols

Load SOS debugging extension

.loadby sos clr
----------------------------------------------------------------------------
The user dump currently examined is a minidump. Consequently, only a subset
of sos.dll functionality will be available. If needed, attaching to the live
process or debugging a full dump will allow access to sos.dll's full feature
set.
To create a full user dump use the command: .dump /ma <filename>
----------------------------------------------------------------------------


Few commands are available when debugging minidump. CLRstack and PrintException are available for executing. After running !CLRstack in windbg there were couple of functions that were executing in loop. Two of them were :
System.Net.Mail.SmtpConnection.GetConnection(System.Net.ServicePoint)
System.Net.Mail.SmtpClient.Send(System.Net.Mail.MailMessage)

From the event log, faulting module is oci.dll, suggesting that application is trying to run something using installed oracle client. To get clearer picture what has happened I have run !PrintException -nested. There were multiple nested exceptions from function to send email when something is wrong with application :

Exception type:   System.Net.Mail.SmtpException
Message:          Failure sending mail.
InnerException:   System.Net.WebException

and, one exception that triggered send email function :
Exception object: 00000001c0925e70
Exception type:   <Unknown>
Message:          could not insert : [....][SQL: INSERT INTO ...]InnerException:  System.Data.OracleClient.OracleException, Use !PrintException 00000001c0923aa0 to see more

After running !PrintException 00000001c0923aa0 :

Exception object: 00000001c0923aa0
Exception type:   System.Data.OracleClient.OracleException
Message:          ORA-00001: unique constraint (....) violated

So, application pool crashing was caused by badly written recursion function to send notification email message to the developer when something was wrong with IIS application. SQL statement Insert Into using oracle client has failed, and called send email function to notify developer, but email server was unreachable. Email sending function has entered into endless loop causing IIS application to crash.

Thursday, March 15, 2012

Error Event ID 11852 SMSClient

In my case XP client with installed Forefront Endpoint Protection 2010 client and SCCM 2007 client was notified that is not reporting to the configured FEP 2010 server. In the client's application event log there were error events with event id 11852 from smsclient source:

Failed to evaluate baseline content
1. Error executing discovery transform. The SDM model may be bad. Exception :System.Xml.Xsl.XslTransformException: An error occurred during a call to extension function 'query'. See InnerException for a complete description of the error. ---> System.TypeInitializationException: The type initializer for 'System.Management.MTAHelper' threw an exception. ---> System.Runtime.InteropServices.COMException (0x80131701): Retrieving the COM class factory for component with CLSID {A8F03BE3-EDB7-4972-821F-AF6F8EA34884} failed due to the following error: 80131701.
Also, in the client's system event log there were error events with event id 59,58 from SideBySide source:
Generate Activation Context failed for C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\WMINet_Utils.dll. Reference error message: The operation completed successfully.
and

Syntax error in manifest or policy file "C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\WMINet_Utils.dll" on line 0.

XP client had already installed framework 3.5 sp1. After repairing framework installation, fep 2010 client (SCCM) started reporting to fep2010 (SCCM) server and error events in event logs were gone.

Friday, March 9, 2012

Installing Windows Server 2008 R2 on HP DL 360 G4

Installing Windows Server 2008 R2 on HP DL 360 G4p is not supported, but Windows 2008 Server x64 is on the list of supported operating systems, so using the information in this post is at your own risk. If you try to install the OS using SmartStart, Windows Server 2008 R2 will not be on the list of operating systems that can be installed, but this does not prevent you to install Windows Server 2008 R2 from installation CD of this operating system and after successful OS installation to install latest PSP (Proliant Support Pack). But, if you try to install PSP there will be critical message: iLO Management Controller Driver is missing.

Installation for “HP Insight Management Agents for Windows Server 2003/2008 x64 Editions” requires one or more of the following that is not currently installed or in the install set:
- HP ProLiant Advanced System Management Controller Driver for Windows
- HP ProLiant iLO Advanced and Enhanced System Management Controller Driver for Windows
- HP ProLiant iLO 2 Management Controller Driver for Windows
- HP ProLiant iLO 3 Management Controller Driver for Windows
- HP ProLiant 100-Series Management Controller Driver for Windows

You can download HP ProLiant ILO Advanced controller driver for Windows Server 2008 x64 editions and run the setup using compatibility mode (Windows Server 2008 Service Pack 1). After installing iLO Management Controller Driver successfully and SNMP feature, you will be able to install latest PSP without errors.

Tuesday, March 6, 2012

Userenv Event ID 1053 "Windows cannot determine the user or computer name"

In my case system was running on Windows Server 2003 R2 with latest patches installed. Network interfaces ware teamed with default settings using hp network utility. The machine was member of domain and following event was logged in application event log :

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1053
Date:  3/5/2012
Time:  5:20:09 PM
User:  NT AUTHORITY\SYSTEM
Description:
Windows cannot determine the user or computer name. (The specified domain either does not exist or could not be contacted. ). Group Policy processing aborted.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Network interface settings for DNS were correct and servers reachable. Server was connected on two manageable cisco switches. After enabling portfast on both interfaces, problem was resolved and error event disappeared.

Friday, March 2, 2012

The task sequence execution engine failed execution of a task sequence (Message ID 11141, 11170,11141)

In my case I was unable to deploy operating system (Windows 7) using SCCM 2007 OSD on machine (A) and following error events were logged :

The task sequence execution engine failed executing the action (Apply Operating System) in the group (Install Operating System) with the error code 2148077575
The task sequence execution engine failed execution of a task sequence. The operating system reported error 2148077575: The hash value is not correct.
The task sequence manager could not successfully complete execution of the task sequence. A failure exit code of 16389 was returned. The operating system reported error 2148077575: The hash value is not correct.
After refreshing package's distribution point, operating system was still failing to install with same errors logged. Since there were no changes on SCCM, since last successful deployment of the same Windows 7 image, I tried to install the same image on other box. Windows 7 was successfully deployed using same task sequence on machine (B). So, there was some problem with hardware on that machine (A). I've checked BIOS time settings and it was current. I've upgraded BIOS to latest version, and still no luck with OSD. Machine (A) had 2 memory modules (1GB each). After removing one memory module from slot, and leaving the machine with 1GB, task sequence completed successfully and Windows 7 was deployed. So, the problem with SCMM 2007 task sequenced OS deployment, was with faulty memory module on client machine (A).