ASP.NET Hang: Authenticode signed assemblies

When you use authenticode signed assemblies in an application, the application needs to go out and check the certificate revocation lists (CRLs) to verify that the signature is still valid the first time it loads up the authenticode signed assembly.

If the server, serving your asp.net application, doesn’t have internet access or if the internet connection is slow this can lead to issues where you stall the process during startup or when the assembly in question first loads.

What you will typically see is a thread loading up an assembly with a call stack like this:

0:026> kL 200
ChildEBP RetAddr
0e82c1b4 7c822124 ntdll!KiFastSystemCallRet
0e82c1b8 77e6bad8 ntdll!NtWaitForSingleObject+0xc
0e82c228 73ca64ec kernel32!WaitForSingleObjectEx+0xac
0e82c254 73ca6742 cryptnet!CryptRetrieveObjectByUrlWithTimeout+0x12f
0e82c280 73ca3253 cryptnet!CryptRetrieveObjectByUrlW+0x9b
0e82c2f8 73ca6b26 cryptnet!RetrieveObjectByUrlValidForSubject+0x5b
0e82c348 73ca3568 cryptnet!RetrieveTimeValidObjectByUrl+0xbc
0e82c3b0 73ca37d6 cryptnet!CTVOAgent::GetTimeValidObjectByUrl+0xc2
0e82c460 73ca3673 cryptnet!CTVOAgent::GetTimeValidObject+0x2f1
0e82c490 73ca3136 cryptnet!CrlFromCertGetTimeValidObject+0x2d
0e82c4d4 73ca4ab1 cryptnet!CryptGetTimeValidObject+0x58
0e82c530 73ca284b cryptnet!GetTimeValidCrl+0x1e0
0e82c574 73ca27a9 cryptnet!GetBaseCrl+0x34
0e82c614 761d2a7a cryptnet!MicrosoftCertDllVerifyRevocation+0x128
0e82c6a4 761d25e4 crypt32!VerifyDefaultRevocation+0x1d0
0e82c714 761d2ec0 crypt32!CertVerifyRevocation+0xb7
0e82c794 761d2769 crypt32!CChainPathObject::CalculateRevocationStatus+0x1f2
0e82c7dc 761cbac3 crypt32!CChainPathObject::CalculateAdditionalStatus+0x147
0e82c898 761ccfdd crypt32!CCertChainEngine::CreateChainContextFromPathGraph+0x227
0e82c8c8 761c235a crypt32!CCertChainEngine::GetChainContext+0x44
0e82c8f0 76bb80ff crypt32!CertGetCertificateChain+0x60
0e82c954 76bb66f3 wintrust!_WalkChain+0x1a8
0e82c990 76bb4de1 wintrust!WintrustCertificateTrust+0xb7
0e82ca84 76bb2f5e wintrust!_VerifyTrust+0x144
0e82caa8 64025c5c wintrust!WinVerifyTrust+0x4e
0e82cb4c 7a02a405 mscorsec!GetPublisher+0xe4
0e82cba4 79e9650a mscorwks!PEFile::CheckSecurity+0xaa
0e82cbd0 79e96453 mscorwks!PEAssembly::CheckSecurity+0x3a
0e82cbf8 79ea20f1 mscorwks!PEAssembly::PEAssembly+0x106
0e82ce94 79ea1fd7 mscorwks!PEAssembly::DoOpen+0x103
0e82cf28 79e9ff07 mscorwks!PEAssembly::Open+0x79
0e82d08c 79e93f1f mscorwks!AppDomain::BindAssemblySpec+0x247
0e82d124 79e93df8 mscorwks!PEFile::LoadAssembly+0x95
0e82d1c4 79f20d93 mscorwks!Module::LoadAssembly+0xee
0e82d200 79f1cdda mscorwks!Assembly::FindModuleByTypeRef+0x113
0e82d250 79e7b956 mscorwks!ClassLoader::LoadTypeDefOrRefThrowing+0xfc
0e82d354 79f13114 mscorwks!SigPointer::GetTypeHandleThrowing+0x297
0e82d3fc 79084899 mscorwks!CEEInfo::getFieldType+0x14a
0e82d9c8 790635c3 mscorjit!Compiler::impImportBlockCode+0x3303
0e82da4c 7906355b mscorjit!Compiler::impImportBlock+0x20c
0e82da64 79063494 mscorjit!Compiler::impImport+0xe5
0e82da70 79064e1c mscorjit!Compiler::fgImport+0x20
0e82da7c 790614e6 mscorjit!Compiler::compCompile+0xb
0e82dad4 79061236 mscorjit!Compiler::compCompile+0x2df
0e82db68 7906118c mscorjit!jitNativeCode+0xb8
0e82dba0 79f0f9cf mscorjit!CILJit::compileMethod+0x3d
0e82dc0c 79f0f945 mscorwks!invokeCompileMethodHelper+0x72
0e82dc50 79f0f8da mscorwks!invokeCompileMethod+0x31
0e82dca8 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0x84
0e82e060 79ecbc28 mscorwks!UnsafeJitFunction+0x230
0e82e0a0 79ecb252 mscorwks!MethodDesc::IsVerifiable+0x9f
0e82e0d4 79ecb216 mscorwks!MethodDesc::IsVerifiable+0x32
0e82e140 79061344 mscorwks!CEEInfo::isInstantiationOfVerifiedGeneric+0xf9
0e82e188 79061236 mscorjit!Compiler::compCompile+0xbd
0e82e21c 7906118c mscorjit!jitNativeCode+0xb8
0e82e254 79f0f9cf mscorjit!CILJit::compileMethod+0x3d
0e82e2c0 79f0f945 mscorwks!invokeCompileMethodHelper+0x72
0e82e304 79f0f8da mscorwks!invokeCompileMethod+0x31
0e82e35c 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0x84
0e82e714 79f0e795 mscorwks!UnsafeJitFunction+0x230
0e82e7b8 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0x1c1
0e82e810 79e8809e mscorwks!MethodDesc::DoPrestub+0x486
0e82e860 01921f0e mscorwks!PreStubWorker+0xeb
...

And if you look at the .net stack with !clrstack you will see the authenticode signed assembly in question loading up, so you can identify which dll it was that required the sign verification.

This thread will call off to another thread that goes out to get the CRL:

0:026> kb 2000
ChildEBP RetAddr  Args to Child
03c0e084 7c822124 71b23a09 000007c0 00000001 ntdll!KiFastSystemCallRet
03c0e088 71b23a09 000007c0 00000001 03c0e0b0 ntdll!NtWaitForSingleObject+0xc
03c0e0c4 71b23a52 000007c0 000007c8 00000000 mswsock!SockWaitForSingleObject+0x19d
03c0e1b4 71c0470c 00000001 00000000 03c0e22c mswsock!WSPSelect+0x380
03c0e204 4e7de746 00000001 00000000 03c0e22c ws2_32!select+0xb9
03c0ea44 4e7de8a5 000007c8 03c0ea68 4e7d976b winhttp!ICSocket::Connect_Start+0x3a8
03c0ea50 4e7d976b 03e42000 4e7c4b5c 00000000 winhttp!CFsm_SocketConnect::RunSM+0x42
03c0ea68 4e7d9805 03d61000 00000000 00000000 winhttp!CFsm::Run+0x20
03c0ea8c 4e7de99b 03e42000 03d80000 03c0eab0 winhttp!DoFsm+0x2a
03c0ea9c 4e7de9b8 0000ea60 00000005 00000020 winhttp!ICSocket::Connect+0x32
03c0eab0 4e7ebc6a 0000ea60 00000005 0000ea60 winhttp!ICSocket::Connect+0x13
03c0eaf8 4e7ebde7 03d67280 03c0eb1c 4e7d976b winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection_Fsm+0x44a
03c0eb04 4e7d976b 03d67280 00000000 00000000 winhttp!CFsm_OpenConnection::RunSM+0x37
03c0eb1c 4e7d9805 03d61000 00000000 00000000 winhttp!CFsm::Run+0x20
03c0eb40 4e7ebe65 03d67280 03d80000 03c0eb78 winhttp!DoFsm+0x2a
03c0eb50 4e7edb71 00000000 00000000 03d61000 winhttp!HTTP_REQUEST_HANDLE_OBJECT::OpenConnection+0x2f
03c0eb78 4e7ede6d 03d671e0 03c0eb9c 4e7d976b winhttp!HTTP_REQUEST_HANDLE_OBJECT::MakeConnection_Fsm+0x9b
03c0eb84 4e7d976b 03d671e0 03d80000 00000000 winhttp!CFsm_MakeConnection::RunSM+0x37
03c0eb9c 4e7d9805 03d61000 00000000 00000000 winhttp!CFsm::Run+0x20
03c0ebc0 4e7ec68f 03d671e0 03d61000 03d67140 winhttp!DoFsm+0x2a
03c0ebf8 4e7ec8b9 03d67140 03c0ec1c 4e7d976b winhttp!HTTP_REQUEST_HANDLE_OBJECT::SendRequest_Fsm+0x8e
03c0ec04 4e7d976b 03d67140 03d80000 00000000 winhttp!CFsm_SendRequest::RunSM+0x37
03c0ec1c 4e7d9805 03d61000 00000000 00000000 winhttp!CFsm::Run+0x20
03c0ec40 4e7e779c 03d67140 03d61000 03e40000 winhttp!DoFsm+0x2a
03c0ec60 4e7e7d11 00000000 03c0ec84 4e7d976b winhttp!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x2a4
03c0ec6c 4e7d976b 03e40000 00000001 00000000 winhttp!CFsm_HttpSendRequest::RunSM+0x4c
03c0ec84 4e7d9902 03d61000 00000000 00000000 winhttp!CFsm::Run+0x20
03c0eccc 4e7d0528 03e40000 03d80000 00000000 winhttp!StartFsmChain+0xd9
03c0ed10 4e7d087e 03d80000 00000000 00000000 winhttp!HttpWrapSendRequest+0x199
03c0ed94 73cac488 03d80000 00000000 00000000 winhttp!WinHttpSendRequest+0x1ee
03c0fe10 73cac916 03d64000 03d80000 0324bdc4 cryptnet!InetSendAuthenticatedRequestAndReceiveResponse+0x108
03c0feb4 73cacbb2 03d64000 0324bdc4 00002005 cryptnet!InetSendReceiveUrlRequest+0x1b0
03c0fee4 73ca3026 0324bdc4 00000002 00002005 cryptnet!CInetSynchronousRetriever::RetrieveObjectByUrl+0x59
03c0ff20 73ca2e08 0324bdc4 00000002 00002005 cryptnet!InetRetrieveEncodedObject+0x66
03c0ff74 73ca62a0 0324bdc4 00000002 00002005 cryptnet!CObjectRetrievalManager::RetrieveObjectByUrl+0xb1
03c0ffb8 77e66063 0324bd80 00000000 00000000 cryptnet!CryptRetrieveObjectByUrlWithTimeoutThreadProc+0x56
03c0ffec 00000000 73ca6207 0324bd80 00000000 kernel32!BaseThreadStart+0x34
The 2nd parameter to cryptnet!InetSendReceiveUrlRequest (0324bdc4) is the URL we are getting the CRL from in this case http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl

0:026> du 0324bdc4
0324bdc4  "http://crl.microsoft.com/pki/crl"
0324be04  "/products/CodeSignPCA.crl"

It will attempt to make the web request for 60 seconds before it times out, so if there is something blocking this request the validation can be stalled for some time.

Another variation

Recently we had a case with this same issue, only it happened at shutdown so it caused the process not to shut down in a timely manner

Event Type:         Warning
Event Source:       W3SVC
Event Category:     None
Event ID:           1013
Date:               13/05/2008
Time:               10:02:49
User:               N/A
Computer:           MYMACHINE
Description:
A process serving application pool 'DefaultAppPool' exceeded time limits during shut down. The process id was '3368'.

For more information, see Help and Support Center at .

What can you do about it?

• Check if the IIS server has internet access and can browse to http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl, verify that the download is not blocked by a proxy or firewall.
• If the IIS server doesn’t have internet access, check if Authenticode signing is neccessary, or if strong naming would suffice for your security needs.
• If authenticode signing is required you may consider catalog signing, where you sign a deployment package/cab file but the individual assemblies will be strong named.
• .Net 2.0 Service Pack 1 contains a fix (KB 936707) that allows you disable the signature verification through the generatePublisherEvidence configuration element.

Additional information

Similar issues can also occur in Exchange and SQL Server, have a look at these kbs for more details

• KB915850 You may experience a 45-second delay when you run a full-text query in an instance of SQL Server 2005 that is running on a server without Internet access
• KB944752 Exchange 2007 managed code services do not start after you install an update rollup for Exchange 2007

Laters, Tess