大家好,我是本期的微软 MVP 实验室研究员——冯辉。本篇文章次要介绍如何利用Windbg剖析利用过程中的内存问题,从托管堆到非托管堆的摸索以及到内存的调配,接下来咱们一起来摸索吧。
近期有几位敌人应用咱们的Magicodes.IE反馈在导出过程中内存暴涨,接下来咱们通过windbg来看一下什么起因导致的。
咱们先通过address -summary
来看一下以后利用内存占用量。
0:000> !address -summary--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotalFree 581 7df8`ef0c9000 ( 125.972 TB) 98.42%<unknown> 1678 206`ffb9e000 ( 2.027 TB) 99.99% 1.58%Image 950 0`064fd000 ( 100.988 MB) 0.00% 0.00%Heap 58 0`050f6000 ( 80.961 MB) 0.00% 0.00%Stack 156 0`04380000 ( 67.500 MB) 0.00% 0.00%Other 11 0`019ad000 ( 25.676 MB) 0.00% 0.00%TEB 52 0`00068000 ( 416.000 kB) 0.00% 0.00%PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotalMEM_MAPPED 282 200`038a6000 ( 2.000 TB) 98.64% 1.56%MEM_PRIVATE 1674 7`07184000 ( 28.111 GB) 1.35% 0.02%MEM_IMAGE 950 0`064fd000 ( 100.988 MB) 0.00% 0.00%--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotalMEM_FREE 581 7df8`ef0c9000 ( 125.972 TB) 98.42%MEM_RESERVE 295 205`f8659000 ( 2.023 TB) 99.79% 1.58%MEM_COMMIT 2611 1`188ce000 ( 4.384 GB) 0.21% 0.00%--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotalPAGE_READWRITE 1595 1`0dc6c000 ( 4.215 GB) 0.20% 0.00%PAGE_EXECUTE_READ 156 0`04d66000 ( 77.398 MB) 0.00% 0.00%PAGE_READONLY 600 0`03851000 ( 56.316 MB) 0.00% 0.00%PAGE_NOACCESS 99 0`021f2000 ( 33.945 MB) 0.00% 0.00%PAGE_EXECUTE_READWRITE 19 0`0027b000 ( 2.480 MB) 0.00% 0.00%PAGE_WRITECOPY 90 0`001a0000 ( 1.625 MB) 0.00% 0.00%PAGE_READWRITE | PAGE_GUARD 52 0`0009e000 ( 632.000 kB) 0.00% 0.00%--- Largest Region by Usage ----------- Base Address -------- Region Size ----------Free 189`0413c000 7c6b`01ed4000 ( 124.418 TB)<unknown> 7dfb`2a153000 1f9`bd2ef000 ( 1.976 TB)Image 7ffc`883c1000 0`009ba000 ( 9.727 MB)Heap 183`0e9a1000 0`00f01000 ( 15.004 MB)Stack 37`62980000 0`0017b000 ( 1.480 MB)Other 183`77707000 0`01775000 ( 23.457 MB)TEB 37`62600000 0`00002000 ( 8.000 kB)PEB 37`627dd000 0`00001000 ( 4.000 kB)
MEM_COMMIT占用了4.384G,接下来咱们利用eeheap -gc
来查看托管堆。
0:000> !eeheap -gcGC Allocated Heap Size: Size: 0x11ac2568 (296494440) bytes.GC Committed Heap Size: Size: 0x120e7000 (302936064) bytes.
依据这些内存来看,仿佛问题不是这里,大量的内存还是呈现在非托管。咱们利用Windows NT堆来看一下,其实在Windows中大多数的用户堆分配器都在ntdll.dll中的NT堆管理器API(RtlAllocateHeap/RtlFreeHeap)上建设,比如说C中的malloc/free和new/delete,另外还有COM框架中的SysAllocString以及在Win32中的LocalAlloc、GlobalAlloc和HeapAlloc,尽管说这些分配器都会创立不同的堆来存储它们的内存,然而他们最终都要调用ntdll.dll中的NT堆来实现。
0:000> !heap -s************************************************************************************************************************ NT HEAP STATS BELOW************************************************************************************************************************NtGlobalFlag enables following debugging aids for new heaps: stack back tracesLFH Key : 0x7cfd4cc2db4ddb4dTermination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap -------------------------------------------------------------------------------------0000018378fd0000 08000002 65128 15296 64928 1720 177 17 2 c LFH External fragmentation 11 % (177 free blocks)00000183775c0000 08008000 64 4 64 2 1 1 0 0 000001837aa90000 08001002 1280 108 1080 26 3 2 0 0 LFH000001837ad20000 08001002 60 8 60 2 1 1 0 0 000001837aca0000 08041002 60 8 60 5 1 1 0 0 000001887bfd0000 08001002 60 20 60 1 2 1 0 0 000001830cf30000 08001002 3324 1364 3124 19 10 3 0 0 LFH000001830ce30000 08001002 60 8 60 5 1 1 0 0 -------------------------------------------------------------------------------------
输入后果如上所示,NT堆内容好少....什么起因....好吧依据maoni所说,仿佛是验证出了问题。
在Windows下面所有的user mode allocations最终都是通过VirtualAlloc来取得内存,bitmaps也好,GC heap也好。
不同的是你去间接调用VirtualAlloc还是应用其余形式去调用它。如果是不托管的内存,GC并不管辖它,当然也不晓得它们的存在。
GC没有管辖这些内存,所以说还是咱们编写的代码有问题,咱们返过去再思考一个事件,“导出进行时,内存会大量减少,导出实现后内存会升高上来”。咱们来看一下代码,如下所示,其实咱们当初明确的是,在咱们执行期间必定是这些内存始终“持有”,并没有被开释掉。
app.MapGet("/excel", async content =>{ string path = Path.Combine(Directory.GetCurrentDirectory(), "test.xlsx"); List<TestDto> list = new(); for (int i = 0; i < 400; i++) { list.Add(new TestDto { ImageUrl = "https://gimg2.baidu.com/image_search/src=http%3A%2F%2Fup.enterdesk.com%2Fedpic_source%2F53%2F0a%2Fda%2F530adad966630fce548cd408237ff200.jpg&refer=http%3A%2F%2Fup.enterdesk.com&app=2002&size=f9999,10000&q=a80&n=0&g=0n&fmt=jpeg?sec=1641193100&t=417a589da8c9ba3103ed74c33fbd6c70" }); } Stopwatch stopwatch = Stopwatch.StartNew(); ExcelExporter exporter = new ExcelExporter(); await exporter.Export(path, list); stopwatch.Stop(); await content.Response.WriteAsync(stopwatch.Elapsed.TotalSeconds.ToString());});
依据内存的体现和咱们的实践,咱们持续利用Windbg来排查一下,当初其实咱们能够发现,这些对象最终还是被GC发出了,带着实践咱们持续构思,GC是晓得哪些对象能够终结的对吧?并且它们在变成不可达到时调用它们的终结器,在GC中会利用finalization queue来记录这些终结对象。所以说咱们是不是能够查一下?如下所示,咱们来看一下。
0:000> !finalizequeue----------------------------------Statistics for all finalizable objects (including all objects ready for finalization): MT Count TotalSize Class Name00007ffc2dc23818 1 24 System.Net.Security.SafeCredentialReference00007ffc2dac4238 1 24 System.WeakReference00007ffc2d6eb908 1 24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]]00007ffc2d6e4120 1 24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]]00007ffc2d572b68 1 24 System.WeakReference`1[[Microsoft.Extensions.DependencyInjection.ServiceProvider, Microsoft.Extensions.DependencyInjection]]00007ffc2d429258 1 24 System.WeakReference`1[[System.IO.FileSystemWatcher, System.IO.FileSystem.Watcher]]00007ffc2dd15c20 1 32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle00007ffc2d6de4d8 1 32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle00007ffc2d68fa00 1 32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle00007ffc2d3a5cc0 1 32 System.Net.Quic.Implementations.MsQuic.Internal.SafeMsQuicRegistrationHandle00007ffc2db390c8 1 40 Interop+WinHttp+SafeWinHttpHandle00007ffc2d69a420 1 40 Internal.Cryptography.Pal.Native.SafeCertContextHandle00007ffc2d5bea18 1 40 System.Diagnostics.EventLog00007ffc2dc29a38 1 48 System.Net.Security.SafeFreeCredential_SECURITY00007ffc2d963f80 2 48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]]00007ffc2d7a3750 2 48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]]00007ffc2d685e10 1 56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Char, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]00007ffc2d44c4d0 1 56 System.Runtime.CompilerServices.ConditionalWeakTable`2+Container[[System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1+ThreadLocalArray[[System.Byte, System.Private.CoreLib]][], System.Private.CoreLib],[System.Object, System.Private.CoreLib]]00007ffc2d96be68 1 64 CellStore`1[[System.Uri, System.Private.Uri]]00007ffc2d96b780 1 64 FlagCellStore00007ffc2d96af48 1 64 CellStore`1[[System.Object, System.Private.CoreLib]]00007ffc2d96a5b8 1 64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]]00007ffc2d6ddab8 2 64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle00007ffc2d69d528 2 64 Internal.Win32.SafeHandles.SafeRegistryHandle00007ffc2d685bc8 2 64 Microsoft.Win32.SafeHandles.SafeWaitHandle00007ffc2d685280 3 72 System.Threading.ThreadInt64PersistentCounter+ThreadLocalNodeFinalizationHelper00007ffc2d5f5f50 3 72 System.Runtime.InteropServices.PosixSignalRegistration00007ffc2d4299d0 1 72 Microsoft.Win32.SafeHandles.SafeFileHandle00007ffc2d6e40b8 1 80 System.Runtime.Loader.DefaultAssemblyLoadContext00007ffc2dac9ed0 2 96 PageIndex00007ffc2d96d0c8 2 96 ColumnIndex00007ffc2d464470 3 120 System.Gen2GcCallback00007ffc2d40a620 1 120 System.IO.FileSystemWatcher00007ffc2d96bc18 2 128 CellStore`1[[System.Int32, System.Private.CoreLib]]00007ffc2dac20c8 2 144 System.Reflection.Emit.DynamicResolver00007ffc2d680f10 3 144 System.Threading.LowLevelLock00007ffc2d683c48 3 168 System.Threading.ThreadPoolWorkQueueThreadLocals00007ffc2d681e80 1 176 System.Threading.LowLevelLifoSemaphore00007ffc2dc25ef0 1 184 System.Collections.Concurrent.CDSCollectionETWBCLProvider00007ffc2db8e658 1 184 System.Net.NetEventSource00007ffc2db8c378 1 184 System.Net.NetEventSource00007ffc2db38f90 1 184 System.Net.NetEventSource00007ffc2d90c658 1 184 Microsoft.IO.RecyclableMemoryStreamManager+Events00007ffc2d689b48 1 184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource00007ffc2d66f9f8 1 184 System.Diagnostics.Tracing.FrameworkEventSource00007ffc2d66b720 1 184 System.Net.NetEventSource00007ffc2d44d128 1 184 System.Buffers.ArrayPoolEventSource00007ffc2d2e2ec8 1 184 System.Diagnostics.Tracing.NativeRuntimeEventSource00007ffc2d694e10 1 192 System.Threading.Tasks.TplEventSource00007ffc2d572ab0 1 192 Microsoft.Extensions.DependencyInjection.DependencyInjectionEventSource00007ffc2d505f00 1 200 Microsoft.Extensions.Logging.EventSource.LoggingEventSource00007ffc2db8ade8 1 224 System.Net.NameResolutionTelemetry00007ffc2d428b08 7 224 System.Threading.PreAllocatedOverlapped00007ffc2d563c78 1 232 System.Diagnostics.DiagnosticSourceEventSource00007ffc2d61fe88 1 240 Microsoft.AspNetCore.Hosting.HostingEventSource00007ffc2db6b788 8 256 System.Threading.TimerQueue+AppDomainTimerSafeHandle00007ffc2d690270 1 280 System.Net.Sockets.SocketsTelemetry00007ffc2db6bc80 1 296 System.Net.Http.HttpTelemetry00007ffc2d68b998 1 336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource00007ffc2dc21998 1 360 System.Net.Security.NetSecurityTelemetry00007ffc2d2dae28 1 384 System.Diagnostics.Tracing.RuntimeEventSource00007ffc2d66ad60 10 480 System.Net.Sockets.SafeSocketHandle00007ffc2d2e0240 21 504 System.WeakReference`1[[System.Diagnostics.Tracing.EventSource, System.Private.CoreLib]]00007ffc2d2b0538 9 648 System.Threading.Thread00007ffc2d77a188 2 704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver00007ffc2d90cec0 6 960 Microsoft.IO.RecyclableMemoryStream00007ffc2d5fc658 10 1280 System.Net.Sockets.Socket00007ffc2d68d898 4 1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs00007ffc2d2dc778 42 4704 System.Diagnostics.Tracing.EventSource+OverrideEventProvider00007ffc2daec058 356 14240 System.Drawing.BitmapTotal 553 objects
WOW!!!,看下面356个System.Drawing.Bitmap
在期待回收,看起来这是咱们的影响因素,咱们来查一下代码。
try{ cell.Value = string.Empty; Bitmap bitmap; if (url.IsBase64StringValid()) { bitmap = url.Base64StringToBitmap(); } else { bitmap = Extension.GetBitmapByUrl(url); } if (bitmap == null) { cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt; } else { ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap); AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)), colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset); CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height; pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height); }}catch (Exception){ cell.Value = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Alt;}
在ExcelPicture对象中去应用Bitmap对象,对于在线图片源来说,咱们会读取并存储到Bitmap中,然而咱们发现并没有对该对象进行开释操作,所以导致大量的Bitmap始终没有开释,咱们通过using来解决一下。
using (ExcelPicture pic = CurrentExcelWorksheet.Drawings.AddPicture(Guid.NewGuid().ToString(), bitmap)){ AddImage((rowIndex + (ExcelExporterSettings.HeaderRowIndex > 1 ? ExcelExporterSettings.HeaderRowIndex : 0)), colIndex - ignoreCount, pic, ExporterHeaderList[colIndex].ExportImageFieldAttribute.YOffset, ExporterHeaderList[colIndex].ExportImageFieldAttribute.XOffset); CurrentExcelWorksheet.Row(rowIndex + 1).Height = ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height; pic.SetSize(ExporterHeaderList[colIndex].ExportImageFieldAttribute.Width * 7, ExporterHeaderList[colIndex].ExportImageFieldAttribute.Height);}
一个带有终结器的新对象是必须要被增加进finalization queue中的,这个行为也被称为“终结注册(registering for finalization)”。当然我也倡议你抉择应用SOSEX扩大插件,它提供了finalization相似的内容,仿佛看起来更直观一些,如下所示。
下载地址:http://www.stevestechspot.com...
:000> .load D:\sosex_64\sosex.dllThis dump has no SOSEX heap index.The heap index makes searching for references and roots much faster.To create a heap index, run !bhi0:000> !finq -statGeneration 0: Count Total Size Type--------------------------------------------------------- 54 2160 System.Drawing.Bitmap54 objects, 2,160 bytesGeneration 1: Count Total Size Type--------------------------------------------------------- 1 184 Microsoft.AspNetCore.Certificates.Generation.CertificateManager+CertificateManagerEventSource 1 336 Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelEventSource 4 1536 System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs 1 32 Internal.Cryptography.Pal.Native.SafeCertStoreHandle 1 280 System.Net.Sockets.SocketsTelemetry 1 192 System.Threading.Tasks.TplEventSource 1 40 Internal.Cryptography.Pal.Native.SafeCertContextHandle 2 64 Internal.Win32.SafeHandles.SafeRegistryHandle 2 64 Internal.Cryptography.Pal.Native.SafeChainEngineHandle 1 32 Internal.Cryptography.Pal.Native.SafeLocalAllocHandle 1 80 System.Runtime.Loader.DefaultAssemblyLoadContext 1 24 System.WeakReference`1[[System.Runtime.Loader.AssemblyLoadContext, System.Private.CoreLib]] 1 24 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerOptions, Microsoft.AspNetCore.Server.Kestrel.Core]] 2 704 Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketReceiver 2 48 System.WeakReference`1[[Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection, Microsoft.AspNetCore.Server.Kestrel.Core]] 1 184 Microsoft.IO.RecyclableMemoryStreamManager+Events 6 960 Microsoft.IO.RecyclableMemoryStream 2 48 System.WeakReference`1[[System.Text.RegularExpressions.RegexReplacement, System.Text.RegularExpressions]] 1 64 CellStore`1[[OfficeOpenXml.ExcelCoreValue, Magicodes.IE.EPPlus]] 1 64 CellStore`1[[System.Object, System.Private.CoreLib]] 1 64 FlagCellStore 2 128 CellStore`1[[System.Int32, System.Private.CoreLib]] 1 64 CellStore`1[[System.Uri, System.Private.Uri]] 2 96 ColumnIndex 2 144 System.Reflection.Emit.DynamicResolver 1 24 System.WeakReference 2 96 PageIndex 302 12080 System.Drawing.Bitmap 1 184 System.Net.NetEventSource 1 40 Interop+WinHttp+SafeWinHttpHandle 8 256 System.Threading.TimerQueue+AppDomainTimerSafeHandle 1 296 System.Net.Http.HttpTelemetry 1 224 System.Net.NameResolutionTelemetry 1 184 System.Net.NetEventSource 1 184 System.Net.NetEventSource 1 360 System.Net.Security.NetSecurityTelemetry 1 24 System.Net.Security.SafeCredentialReference 1 184 System.Collections.Concurrent.CDSCollectionETWBCLProvider 1 48 System.Net.Security.SafeFreeCredential_SECURITY 1 32 Microsoft.Win32.SafeHandles.SafeBCryptAlgorithmHandle499 objects, 30,736 bytesGeneration 2:0 objects, 0 bytesTOTAL: 553 objects, 32,896 bytes
可能大家都会像我一开始有个疑难,你这个图片我看了...没有那么大,并且在Windbg中也没有体现大小呀。首先咱们先来看一下这个图片的品质。图片的像素为2560x1440,位深为24目前已知这些信息,咱们计算一下未压缩的图片大小。
2560x1440x24/8
10M左右一张图,已知图片数x10M=3G,其实对于这个问题来说,这并不属于内存透露。
总结
这篇文章次要介绍如何利用Windbg剖析利用过程中的内存问题,从托管堆到非托管堆的摸索以及到内存的调配,最终依据内存的体现和实践确认内存的问题,当然对于内存剖析倡议大家不肯定非要钟情一个工具,当然能够联合着PerfView一起做兴许成果更佳。
微软最有价值专家(MVP)
微软最有价值专家是微软公司授予第三方技术专业人士的一个寰球奖项。28年来,世界各地的技术社区领导者,因其在线上和线下的技术社区中分享专业知识和教训而取得此奖项。
MVP是通过严格筛选的专家团队,他们代表着技术最精湛且最具智慧的人,是对社区投入极大的激情并乐于助人的专家。MVP致力于通过演讲、论坛问答、创立网站、撰写博客、分享视频、开源我的项目、组织会议等形式来帮忙别人,并最大水平地帮忙微软技术社区用户应用Microsoft技术。
更多详情请登录官方网站:
https://mvp.microsoft.com/zh-cn
欢送关注微软中国MSDN订阅号,获取更多最新公布!