NetInverse Developers Blog

April 11, 2009
Category: Debugging — Tags: , , , , — admin @ 4:13 pm

SOS Commmand: !DumpMT [-MD] <MethodTable address>

Examine a MethodTable. Each managed object has a MethodTable pointer at the start. If you pass the “-MD” flag, you’ll also see a list of all the methods defined on the object.

.load sos
...
!clrstack
...
OS Thread Id: 0x6f8 (1784)
ESP       EIP
0012f3d4 00ff01b4 FindProduct.Program.AppendProduct(System.Collections.Generic.List`1<FindProduct.Product>)
...
!ip2md 00ff01b4
MethodDesc: 00933008
Method Name: FindProduct.Program.AppendProduct(System.Collections.Generic.List`1<FindProduct.Product>)
Class: 00931300
MethodTable: 00933024
...
!dumpmt 00933024
EEClass: 00931300
Module: 00932c5c
Name: FindProduct.Program
mdToken: 02000002  (C:svnNetInverseFindProductbinDebugFindProduct.exe)
BaseSize: 0xc
ComponentSize: 0x0
Number of IFaces in IFaceMap: 0
Slots in VTable: 7
Category: Debugging — Tags: , , , , — admin @ 4:00 pm

SOS Command: !SyncBlk [-all | <syncblk number>]

A SyncBlock is a holder for extra information that doesn’t need to be created for every object. It can hold COM Interop data, HashCodes, and locking information for thread-safe operations.

When called without arguments, !SyncBlk will print the list of SyncBlocks corresponding to objects that are owned by a thread. For example, a

    lock(MyObject)
    {
        ....
    }

statement will set MyObject to be owned by the current thread. A SyncBlock will be created for MyObject, and the thread ownership information stored there (this is an oversimplification, see NOTE below). If another thread tries to execute the same code, they won’t be able to enter the block until the first thread exits.

This makes !SyncBlk useful for detecting managed deadlocks. Consider that the following code is executed by Threads A & B:

    Resource r1 = new Resource();
    Resource r2 = new Resource();
    ...
    lock(r1)                             lock(r2)
    {                                    {
        lock(r2)                             lock(r1)
        {                                    {
            ...                                  ...
        }                                    }
    }                                    }

This is a deadlock situation, as Thread A could take r1, and Thread B r2, leaving both threads with no option but to wait forever in the second lock statement. !SyncBlk will detect this with the following output:

0:003> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info   SyncBlock Owner
  238 001e40ec            3         1 001e4e60   e04   3   00a7a194 Resource
  239 001e4124            3         1 001e5980   ab8   4   00a7a1a4 Resource

It means that Thread e04 owns object 00a7a194, and Thread ab8 owns object 00a7a1a4. Combine that information with the call stacks of the deadlock: (threads 3 and 4 have similar output)

0:003> k

ChildEBP RetAddr
0404ea04 77f5c524 SharedUserData!SystemCallStub+0x4
0404ea08 77e75ee0 ntdll!NtWaitForMultipleObjects+0xc
0404eaa4 5d9de9d6 KERNEL32!WaitForMultipleObjectsEx+0x12c
0404eb38 5d9def80 mscorwks!Thread::DoAppropriateAptStateWait+0x156
0404ecc4 5d9dd8bb mscorwks!Thread::DoAppropriateWaitWorker+0x360
0404ed20 5da628dd mscorwks!Thread::DoAppropriateWait+0xbb
0404ede4 5da4e2e2 mscorwks!CLREvent::Wait+0x29d
0404ee70 5da4dd41 mscorwks!AwareLock::EnterEpilog+0x132
0404ef34 5da4efa3 mscorwks!AwareLock::Enter+0x2c1
0404f09c 5d767880 mscorwks!AwareLock::Contention+0x483
0404f1c4 03f00229 mscorwks!JITutil_MonContention+0x2c0
0404f1f4 5b6ef077 image00400000!Worker.Work()+0x79
...

By looking at the code corresponding to Worker.Work()+0×79 (run “!u 03f00229″), you can see that thread 3 is attempting to acquire the Resource 00a7a1a4, which is owned by thread 4.

NOTE:

It is not always the case that a SyncBlock will be created for every object that is locked by a thread. In version 2.0 of the CLR and above, a mechanism called a ThinLock will be used if there is not already a SyncBlock for the object in question. ThinLocks will not be reported by the !SyncBlk command. You can use “!DumpHeap -thinlock” to list objects locked in this way.

Category: Debugging — Tags: , , , , — admin @ 3:48 pm

SOS Command: !Name2EE <module name> <type or method name>

This function allows you to turn a class name into a MethodTable and EEClass. It turns a method name into a MethodDesc. Here is an example for a method:

0:000> !name2ee unittest.exe MainClass.Main
Module: 001caa38
Token: 0x0600000d
MethodDesc: 00902f40
Name: MainClass.Main()
JITTED Code Address: 03ef00b8

and for a class:

0:000> !name2ee unittest!MainClass
Module: 001caa38
Token: 0x02000005
MethodTable: 009032d8
EEClass: 03ee1424
Name: MainClass

The module you are “browsing” with Name2EE needs to be loaded in the process. To get a type name exactly right, first browse the module with ILDASM. You can also pass * as the <module name> to search all loaded managed modules. <module name> can also be the debugger’s name for a module, such as mscorlib or image00400000.

The Windows Debugger syntax of <module>!<type> is also supported. You can use an asterisk on the left of the !, but the type on the right side needs to be fully qualified.

If you are looking for a way to display a static field of a class (and you don’t have an instance of the class, so !dumpobj won’t help you), note that once you have the EEClass, you can run !DumpClass, which will display the value of all static fields.

There is yet one more way to specify a module name. In the case of modules loaded from an assembly store (such as a SQL db) rather than disk, the module name will look like this: price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null For this kind of module, simply use price as the module name:

0:044> !name2ee price Price
Module: 10f028b0 (price, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null)
Token: 0x02000002
MethodTable: 11a47ae0
EEClass: 11a538c8
Name: Price

Where are we getting these module names from? Run !DumpDomain to see a list of all loaded modules in all domains. And remember that you can browse all the types in a module with !DumpModule -mt <module pointer>. Note: Above information is compiled based on the SOS help.

Category: Debugging — Tags: , , , , , — admin @ 3:35 pm

SOS Command: !EEHeap [-gc] [-loader]

!EEHeap enumerates process memory consumed by internal CLR data structures. You can limit the output by passing “-gc” or “-loader”. All information will be displayed otherwise.

The information for the Garbage Collector lists the ranges of each Segment in the managed heap. This can be useful if you believe you have an object pointer. If the pointer falls within a segment range given by “!EEHeap -gc”, then you do have an object pointer, and can attempt to run “!DumpObj” on it.

Here is output for a simple program:

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00a71018
generation 1 starts at 0x00a7100c
generation 2 starts at 0x00a71000

 segment    begin allocated     size
00a70000 00a71000  00a7e01c 0000d01c(53276)
Large object heap starts at 0x01a71000

 segment    begin allocated     size
01a70000 01a71000  01a76000 0x00005000(20480)

Total Size   0x1201c(73756)
------------------------------
GC Heap Size   0x1201c(73756)

So the total size of the GC Heap is only 72K. On a large web server, with multiple processors, you can expect to see a GC Heap of 400MB or more. The Garbage Collector attempts to collect and reclaim memory only when required to by memory pressure for better performance. You can also see the notion of “generations,” wherein the youngest objects live in generation 0, and long-lived objects eventually get “promoted” to generation 2. The loader output lists various private heaps associated with AppDomains. It also lists heaps associated with the JIT compiler, and heaps associated with Modules. For example:

0:000> !EEHeap -loader

Loader Heap:
--------------------------------------
System Domain: 5e0662a0
LowFrequencyHeap:008f0000(00002000:00001000) Size: 0x00001000 bytes.
HighFrequencyHeap:008f2000(00008000:00001000) Size: 0x00001000 bytes.
StubHeap:008fa000(00002000:00001000) Size: 0x00001000 bytes.

Total size: 0x3000(12288)bytes
--------------------------------------
Shared Domain: 5e066970
LowFrequencyHeap:00920000(00002000:00001000) 03e30000(00010000:00003000) Size: 0x00004000 bytes.
Wasted: 0x00001000 bytes.
HighFrequencyHeap:00922000(00008000:00001000) Size: 0x00001000 bytes.
StubHeap:0092a000(00002000:00001000) Size: 0x00001000 bytes.

Total size: 0x6000(24576)bytes
--------------------------------------
Domain 1: 14f000
LowFrequencyHeap:00900000(00002000:00001000) 03ee0000(00010000:00003000) Size: 0x00004000 bytes.
Wasted: 0x00001000 bytes.
HighFrequencyHeap:00902000(00008000:00003000) Size: 0x00003000 bytes.
StubHeap:0090a000(00002000:00001000) Size: 0x00001000 bytes.

Total size: 0x8000(32768)bytes
--------------------------------------
Jit code heap:
Normal JIT:03ef0000(00010000:00002000) Size: 0x00002000 bytes.

Total size: 0x2000(8192)bytes
--------------------------------------
Module Thunk heaps:
Module 5ba22410: Size: 0x00000000 bytes.
Module 001c1320: Size: 0x00000000 bytes.
Module 001c03f0: Size: 0x00000000 bytes.
Module 001caa38: Size: 0x00000000 bytes.

Total size: 0x0(0)bytes
--------------------------------------
Module Lookup Table heaps:
Module 5ba22410:Size: 0x00000000 bytes.
Module 001c1320:Size: 0x00000000 bytes.
Module 001c03f0:Size: 0x00000000 bytes.
Module 001caa38:03ec0000(00010000:00002000) Size: 0x00002000 bytes.

Total size: 0x2000(8192)bytes
--------------------------------------
Total LoaderHeap size: 0x15000(86016)bytes
=======================================

By using !EEHeap to keep track of the growth of these private heaps, we are able to rule out or include them as a source of a memory leak. Memory leak will cause OOM(Out of Memory) exception. You can inspect the modules from the above dump by using command !dumpmodule -mt 001c1320.

Category: Debugging — Tags: , , , , — admin @ 3:20 pm

SOS Command: !DumpDomain [<Domain address>]

When called with no parameters, !DumpDomain will list all the AppDomains in the process. It enumerates each Assembly loaded into those AppDomains as well.

In addition to your application domain, and any domains it might create, there are two special domains: the Shared Domain and the System Domain.

Any Assembly pointer in the output can be passed to !DumpAssembly. Any Module pointer in the output can be passed to !DumpModule. Any AppDomain pointer can be passed to !DumpDomain to limit output only to that AppDomain. Other functions provide an AppDomain pointer as well, such as !Threads where it lists the current AppDomain for each thread.

!dumpdomain
--------------------------------------
System Domain: 7a3bd058
LowFrequencyHeap: 7a3bd07c
HighFrequencyHeap: 7a3bd0c8
StubHeap: 7a3bd114
Stage: OPEN
Name: None
--------------------------------------
Shared Domain: 7a3bc9a8
LowFrequencyHeap: 7a3bc9cc
HighFrequencyHeap: 7a3bca18
StubHeap: 7a3bca64
Stage: OPEN
Name: None
Assembly: 001a58e8
--------------------------------------
Domain 1: 0015d2e0
LowFrequencyHeap: 0015d304
HighFrequencyHeap: 0015d350
StubHeap: 0015d39c
Stage: OPEN
SecurityDescriptor: 0015e608
Name: FindProduct.exe
Assembly: 001a58e8 [C:WINDOWSassemblyGAC_32mscorlib2.0.0.0__b77a5c561934e089mscorlib.dll]
ClassLoader: 001a5968
SecurityDescriptor: 001a4720
  Module Name
790c1000 C:WINDOWSassemblyGAC_32mscorlib2.0.0.0__b77a5c561934e089mscorlib.dll

Assembly: 001b0788 [C:svnNetInverseFindProductbinDebugFindProduct.exe]
ClassLoader: 001b0808
SecurityDescriptor: 001b0688
  Module Name
00932c5c C:svnNetInverseFindProductbinDebugFindProduct.exe
System Domain

The SystemDomain is responsible for creating and initializing the SharedDomain and the default AppDomain. It loads the system library mscorlib.dll into SharedDomain. It also keeps process-wide string literals interned implicitly or explicitly. String interning is an optimization feature that saves memory by having only a single instance of the string for a given literal across all the application domains. SystemDomain is also responsible for generating process-wide interface IDs, which are used in creating InterfaceVtableMaps in each AppDomain. SystemDomain keeps track of all the domains in the process and implements functionality for loading and unloading the AppDomains.

SharedDomain

All of the domain-neutral code is loaded into SharedDomain. Mscorlib, the system library, is needed by the user code in all the AppDomains. It is automatically loaded into SharedDomain. Fundamental types from the System namespace like Object, ValueType, Array, Enum, String, and Delegate get preloaded into this domain during the CLR bootstrapping process. User code can also be loaded into this domain, using LoaderOptimization attributes specified by the CLR hosting app while calling CorBindToRuntimeEx. Console programs can load code into SharedDomain by annotating the app’s Main method with a System.LoaderOptimizationAttribute. SharedDomain also manages an assembly map indexed by the base address, which acts as a lookup table for managing shared dependencies of assemblies being loaded into DefaultDomain and of other AppDomains created in managed code. DefaultDomain is where non-shared user code is loaded.

DefaultDomain

DefaultDomain is an instance of AppDomain within which application code is typically executed. While some applications require additional AppDomains to be created at runtime (such as apps that have plug-in architectures or apps doing a significant amount of run-time code generation), most applications create one domain during their lifetime. All code that executes in this domain is context-bound at the domain level. If an application has multiple AppDomains, any cross-domain access will occur through .NET Remoting proxies. Additional intra-domain context boundaries can be created using types inherited from System.ContextBoundObject. Each AppDomain has its own SecurityDescriptor, SecurityContext, and DefaultContext, as well as its own loader heaps (High-Frequency Heap, Low-Frequency Heap, and Stub Heap), Handle Tables (Handle Table, Large Object Heap Handle Table), Interface Vtable Map Manager, and Assembly Cache.

LoadHeaps

Frequently accessed artifacts like MethodTables, MethodDescs, FieldDescs, and Interface Maps get allocated on a HighFrequencyHeap, while less frequently accessed data structures, such as EEClass and ClassLoader and its lookup tables, get allocated on a LowFrequencyHeap. The StubHeap hosts stubs that facilitate code access security (CAS), COM wrapper calls, and P/I

Category: Debugging — Tags: , , , , — admin @ 3:14 pm

SOS Comand: !TraverseHeap

!TraverseHeap [-xml] <filename>

!TraverseHeap writes out a file in a format understood by the CLR Profiler.
You can download the CLR Profiler from this link:

http://www.microsoft.com/downloads/details.aspx?FamilyId=86CE6052-D7F4-4AEB-9B7A-94635BEEBDDA&displaylang=en

It creates a graphical display of the GC heap to help you analyze the state of your application. If you pass the “-xml” flag, the file is instead written out in an easy-to-understand xml format:

    <gcheap>
        <types>
            <type id="1" name="System.String">
            ...
        </types>
        <roots>
            <root kind="handle" address="0x00a73ff0"/>
            <root kind="stack" address="0x0069f0e0"/>
            ...
        </roots>
        <objects>
            <object address="0x00b73030" typeid="1" size="300"/>
            <object address="0x00b75054" typeid="5" size="20">
                <member address="0x00b75088" />
                ...
            </object>
            ...
        </objects>
    </gcheap>

You can break into your process, load SOS, take a snapshot of your heap with this function, then continue.

Category: Debugging — Tags: , , , , — admin @ 3:10 pm
!PrintException [-nested] [<Exception object address>]

This will format fields of any object derived from System.Exception. One of the more useful aspects is that it will format the _stackTrace field, which is a binary array. If _stackTraceString field is not filled in, that can be helpful for debugging. You can of course use !DumpObj on the same exception object to explore more fields.

If called with no parameters, PrintException will look for the last outstanding exception on the current thread and print it. This will be the same exception that shows up in a run of !Threads.

!PrintException will notify you if there are any nested exceptions on the current managed thread. (A nested exception occurs when you throw another exception within a catch handler already being called for another exception).

If there are nested exceptions, you can re-run !PrintException with the “-nested” option to get full details on the nested exception objects. The !Threads command will also tell you which threads have nested exceptions. The abbreviation !pe can be used for brevity.

Category: Debugging — Tags: , , , , — admin @ 3:04 pm

SOS Command: !FinalizeQueue

!FinalizeQueue [-detail]

This command lists the objects registered for finalization. Here is output from a simple program:

0:000> !finalizequeue

SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 1

generation 0 has 4 finalizable objects (0015bc90->0015bca0)
generation 1 has 0 finalizable objects (0015bc90->0015bc90)
generation 2 has 0 finalizable objects (0015bc90->0015bc90)

Ready for finalization 0 objects (0015bca0->0015bca0)

Statistics:

      MT    Count TotalSize Class Name
5ba6cf78        1        24 Microsoft.Win32.SafeHandles.SafeFileHandle
5ba5db04        1        68 System.Threading.Thread
5ba73e28        2       112 System.IO.StreamWriter
Total 4 objects

The GC heap is divided into generations, and objects are listed accordingly. We see that only generation 0 (the youngest generation) has any objects registered for finalization. The notation “(0015bc90->0015bca0)” means that if you look at memory in that range, you’ll see the object pointers that are registered:

0:000> dd 15bc90 15bca0-4

0015bc90  00a743f4 00a79f00 00a7b3d8 00a7b47c

You could run !DumpObj on any of those pointers to learn more. In this example, there are no objects ready for finalization, presumably because they still have roots (You can use !GCRoot to find out). The statistics section provides a higher-level summary of the objects registered for finalization. Note that objects ready for finalization are also included in the statistics (if any).

If you pass -detail then you get extra information on any SyncBlocks that need to be cleaned up, and on any RuntimeCallableWrappers (RCWs) that await cleanup. Both of these data structures are cached and cleaned up by the finalizer thread when it gets a chance to run.

Category: Debugging — Tags: , , , , — admin @ 9:59 am

SOS Command: !ObjSize [<Object address>]

With no parameters, !ObjSize lists the size of all objects found on managed threads. It also enumerates all GCHandles in the process, and totals the size of any objects pointed to by those handles. In calculating object size, !ObjSize includes the size of all child objects in addition to the parent.

For example, !DumpObj lists a size of 20 bytes for this Customer object:

0:000> !do a79d40

Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
 (C:pubunittest.exe)

Fields:
      MT    Field   Offset                 Type       Attr    Value Name
009038ec  4000008        4                CLASS   instance 00a79ce4 name
009038ec  4000009        8                CLASS   instance 00a79d2c bank
009038ec  400000a        c       System.Boolean   instance        1 valid

but !ObjSize lists 152 bytes:

0:000> !ObjSize a79d40
sizeof(00a79d40) =      152 (    0x98) bytes (Customer)

This is because a Customer points to a Bank, has a name, and the Bank points to an Address string. You can use !ObjSize to identify any particularly large objects, such as a managed cache in a web server. Note: above information was from SOS help.

April 10, 2009
Category: Debugging — Tags: , , , , — admin @ 5:45 pm

SOS Command: !DumpHeap

!DumpHeap [-stat]
          [-strings]
          [-short]
          [-min <size>]
          [-max <size>]
          [-thinlock]
          [-startAtLowerBound]
          [-mt <MethodTable address>]
          [-type <partial type name>]
          [start [end]]

!DumpHeap is a powerful command that traverses the garbage collected heap, collection statistics about objects. With it’s various options, it can look for particular types, restrict to a range, or look for ThinLocks (see !SyncBlk documentation). Finally, it will provide a warning if it detects excessive fragmentation in the GC heap.

When called without options, the output is first a list of objects in the heap, followed by a report listing all the types found, their size and number:

0:000> !dumpheap

 Address       MT     Size

00a71000 0015cde8       12 Free
00a71024 5ba58328       68
...

total 619 objects

Statistics:
      MT    Count TotalSize Class Name

5ba7607c        1        12 System.Security.Permissions.HostProtectionResource
5ba75d54        1        12 System.Security.Permissions.SecurityPermissionFlag
5ba61f18        1        12 System.Collections.CaseInsensitiveComparer
...

0015cde8        6     10260      Free
5ba57bf8      318     18136 System.String
...

“Free” objects are simply regions of space the garbage collector can use later. If 30% or more of the heap contains “Free” objects, the process may suffer fromheap fragmentation. This is usually caused by pinning objects for a long time combined with a high rate of allocation. Here is example output where !DumpHeap provides a warning about fragmentation:

<After the Statistics section> Fragmented blocks larger than 1MB:

    Addr     Size Followed by
00a780c0    1.5MB    00bec800 System.Byte[]
00da4e38    1.2MB    00ed2c00 System.Byte[]
00f16df0    1.2MB    01044338 System.Byte[]

The arguments in detail:

-stat     Restrict the output to the statistical type summary
-strings  Restrict the output to a statistical string value summary
-short    Limits output to just the address of each object. This allows you
          to easily pipe output from the command to another debugger
          command for automation.
-min      Ignore objects less than the size given in bytes
-max      Ignore objects larger than the size given in bytes
-thinlock Report on any ThinLocks (an efficient locking scheme, see !SyncBlk
          documentation for more info)
-startAtLowerBound
          Force heap walk to begin at lower bound of a supplied address range.
          (During plan phase, the heap is often not walkable because objects
          are being moved. In this case, DumpHeap may report spurious errors,
          in particular bad objects. It may be possible to traverse more of
          the heap after the reported bad object. Even if you specify an
          address range, !DumpHeap will start its walk from the beginning of
          the heap by default. If it finds a bad object before the specified
          range, it will stop before displaying the part of the heap in which
          you are interested. This switch will force !DumpHeap to begin its
          walk at the specified lower bound. You must supply the address of a
          good object as the lower bound for this to work. Display memory at
          the address of the bad object to manually find the next method
          table (use !dumpmt to verify). If the GC is currently in a call to
          memcopy, You may also be able to find the next object's address by
          adding the size to the start address given as parameters.)
-mt       List only those objects with the MethodTable given
-type     List only those objects whose type name is a substring match of the
          string provided.
start     Begin listing from this address
end       Stop listing at this address

A special note about -type: Often, you’d like to find not only Strings, but System.Object arrays that are constrained to contain Strings. (”new String[100]” actually creates a System.Object array, but it can only hold System.String object pointers). You can use -type in a special way to find these arrays. Just pass “-type System.String[]” and those Object arrays will be returned. More generally, “-type <Substring of interesting type>[]“. The start/end parameters can be obtained from the output of !EEHeap -gc. For example, if you only want to list objects in the large heap segment:

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00c32754
generation 1 starts at 0x00c32748
generation 2 starts at 0x00a71000

 segment    begin allocated     size
00a70000 00a71000  010443a8 005d33a8(6108072)
Large object heap starts at 0x01a71000
 segment    begin allocated     size
01a70000 01a71000  01a75000 0x00004000(16384)

Total Size  0x5d73a8(6124456)
------------------------------
GC Heap Size  0x5d73a8(6124456)

0:000> !dumpheap 1a71000 1a75000

 Address       MT     Size
01a71000 5ba88bd8     2064
01a71810 0019fe48     2032 Free
01a72000 5ba88bd8     4096
01a73000 0019fe48     4096 Free
01a74000 5ba88bd8     4096
total 5 objects

Statistics:
      MT    Count TotalSize Class Name
0019fe48        2      6128      Free
5ba88bd8        3     10256 System.Object[]

Total 5 objects

Finally, if gc heap corruption is present, you may see an error like this:

0:000> !dumpheap -stat

object 00a73d24: does not have valid MT
curr_object : 00a73d24
Last good object: 00a73d14
----------------

That indicates a serious problem. See the help for !VerifyHeap for more information on diagnosing the cause.

Note: above information is compiled based on SOS online help.

« Newer PostsOlder Posts »

©2009 NetInverse. All rights reserved. Powered by WordPress