The case encountered
When adding FIPS 140-2 compliance to our Windows software, we encountered an intermittent error:
This error happened randomly on various different machines. Once it happened on a particular machine, it would keep happening on that machine until rebooting. It then went back to being random.
We eventually found that this is because of a DLL that couldn't be relocated.
Keep reading to find out more!
The background to the case
FIPS (Federal Information Processing Standard) 140-2 is a U.S. government security computer standard used to accredit cryptographic modules. This standard is titled "Security Requirements for Cryptographic Modules".
OpenSSL is a general purpose cryptography library that provides an open source implementation of the Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protocols.
So what happens when you want to use OpenSSL to deliver FIPS validated cryptography on Windows?
You need to use OpenSSL FIPS Object Module validation! This requires specified FIPS build process to build the OpenSSL DLLs, including a fixed base address.
Why the case happened
At startup, using FIPS mode, the OpenSSL DLLs will perform an integrity check.
If the DLLs have been relocated, which will happen if the fixed base address is already being used by something else, the integrity check fails. This creates the error:
A popup window - 0xc0000018 "The application was unable to start correctly".
Windows tries to keep DLLs at the same address in future processes. So once this error happened on a particular machine, it was quite likely to happen the next time the process was run. Restarting Windows was needed to make the OpenSSL DLLs load at a different address.
Trying to make the DLL relocatable to avoid the case
You can change the OpenSSL FIPS 64-bit build to use /dynamicbase for libeay32.dll to make it relocatable, which avoids the problem.
However, for 32-bit builds you still need libeay32.dll to be loaded at a fixed address to pass validations.
Minimizing occurrences of the case
You can't reliably fix this problem. But you can minimize it.
Loading the OpenSSL DLLs as early as possible in the process reduces the chance of another DLL taking up the required addresses. Making your EXE directly depend on the OpenSSL DLLs before any other of your DLLs will help.
You might think that you can pick a base address not used by anything else to avoid the problem. But you can't control the Windows DLLs, such as ntdll.dll. Also, recent Windows versions also use ASLR (Address Space Layout Randomization) so you can't predict consistent load addresses.
Next steps
That's where we got up to - minimizing the problem. Does anyone have other solutions, or know any other DLLs that can't be relocated?
