Getting SIGSEGV in EVP_MD_CTX_free

[shubhamsingh397]

Hi Everyone, Recently I have moved to using OpenSSL 3.0.8 from OpenSSL 1.0.2 in SoftHSMv2 library.

I've been getting SIGSEGV 11 while doing EVP_MD_CTX_free(ctx) with the following error trace

`***Before first iteration
****** In HashInit ****:
***** In HashInit before EVPDigestInitex ****:
***** In getEVPHash() ******
***End of HashInit ****:
***** In HashUpdate ****:
***** In Hashupdate before EVPDigestUpdate ****:
***** End of HashUpdate ****: ***** In HashUpdate ****:
***** In Hashupdate before EVPDigestUpdate ****:
***** End of HashUpdate ****:
***** In HashFinal ****: 0x7fa730151470
***** In HashFinal before resize ****:
***** In getEVPHash() ****
***** In HashFinal EVP_MD_SIZE returns 32 *****:
***** In HashFinal after resize ****:
***** In HashFinal before EVPDigestFinal ****:
***** In HashFinal After EVPDigestFinal :
***** In HashFinal before resize - 2
********* In HashFinal after resize - 20x7fa730151470
****OPENSSL -> ENtered EVP_MD_CTX_free *** 806687856
****OPENSSL -> EVP_MD_CTX_free after null check *** 806687856
*OPENSSL -> Entered evp_md_ctx_reset_ex 806687856
*OPENSSL -> evp_md_ctx_reset_ex after null check806687856
****OPENSSL -> evp_md_ctx_reset_ex after evp_md_ctx_clear_digest **806687856
***OPENSSL -> evp_md_ctx_reset_ex after OPENSSL_cleanse *** 806687856
****OPENSSL -> EVP_MD_CTX_free after EVP_MD_CTX_reset *** 806687856
****OPENSSL -> EVP_MD_CTX_free after OPENSSL_free *** 806687856
***** After EVP_MD_CTX_FREE: 0x7fa730151470
***** End of HashFinal ****: 0
**After first iteration

*******iteration count : 1709

***** In HashInit ****:
***** In HashInit before EVPDigestInitex ****:
***** In getEVPHash() ****
*****End of HashInit ****:
***** In HashUpdate ****:
***** In Hashupdate before EVPDigestUpdate ****:
***** End of HashUpdate ****:
***** In HashFinal ****: 0x7fa730151470
***** In HashFinal before resize ****:
***** In getEVPHash() *****
**** In HashFinal EVP_MD_SIZE returns 32 *****:
***** In HashFinal after resize ****:
***** In HashFinal before EVPDigestFinal ****:
***** In HashFinal After EVPDigestFinal :
***** In HashFinal before resize - 2
********* In HashFinal after resize - 20x7fa730151470
****OPENSSL -> ENtered EVP_MD_CTX_free *** 806687856
****OPENSSL -> EVP_MD_CTX_free after null check *** 806687856
*OPENSSL -> Entered evp_md_ctx_reset_ex 806687856
*OPENSSL -> evp_md_ctx_reset_ex after null check806687856
*OPENSSL -> evp_md_ctx_reset_ex pctx value-720710000
*EVP_PKEY_CTX_free - value of ctx here -720710000
******** EVP_PKEY_CTX_free - if not returned from here

ATAL ERROR : Caught a fatal signal or exception.\n***********\n\n\n***********\nFATAL ERROR : Aborting the DTM process due to fatal signal or exception.\n***********\n\n\n***********\nFATAL ERROR : Signal Received: SIGSEGV (11)\n***********\n\n\nStack trace produced by process [pmdtm(:wf_mtt_01000017000000000002.s_mtt_01000017000000000002_e44ba218a6a545f18bdc664fea40cbf2)] with pid [47635].\n.
/libpmasrt.so(PmDumpStackTrace+0x79)[0x7fa7d4830d69]\n
/cldagnt/apps/Data_Integration_Server/67.0.1.1/ICS/main/bin/rdtm/pmdtm[0x5e89fa]\n/cldagnt/jdk/jre/lib/amd64/server/libjvm.so(+0x9771fb)[0x7fa7ae0951fb]\n
/cldagnt/jdk/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x325)[0x7fa7ae09a095]\n
/cldagnt/jdk/jre/lib/amd64/server/libjvm.so(+0x96e478)[0x7fa7ae08c478]\n
/lib64/libpthread.so.0(+0xf630)[0x7fa7d6cdc630]\n.
/libcrypto.so.3(EVP_PKEY_CTX_free+0x5b)[0x7fa7b53b3033]\n.
/libcrypto.so.3(+0x1f7e0d)[0x7fa7b5379e0d]\n./
libcrypto.so.3(EVP_MD_CTX_reset+0x1d)[0x7fa7b5379eb9]\n.
/libcrypto.so.3(EVP_MD_CTX_free+0x55)[0x7fa7b537a017]\n.
/libsofthsm2.so(+0xbb844)[0x7fa7b5b1a844]\n.
/libsofthsm2.so(+0xcbcbc)[0x7fa7b5b2acbc]\n.
/libsofthsm2.so(+0xcc4e9)[0x7fa7b5b2b4e9]\n.
/libsofthsm2.so(+0xccb2d)[0x7fa7b5b2bb2d]\n.
/libsofthsm2.so(+0xf003d)[0x7fa7b5b4f03d]\n.
/libsofthsm2.so(+0xee352)[0x7fa7b5b4d352]\n.
/libsofthsm2.so(+0x73283)[0x7fa7b5ad2283]\n.
/libsofthsm2.so(C_InitToken+0x3e)[0x7fa7b5ab1d19]\n.
/libiCPSDK.so.1(_ZN16CryptoOpsManager9initTokenERK17IHSMConfiguration+0x376)[0x7fa7b66ee36a]\n.
/libiCPSDK.so.1`

Below is the code for reference.

EVPHashAlgorithm.cpp link to file

`// Destructor
OSSLEVPHashAlgorithm::~OSSLEVPHashAlgorithm()
{
std::cout << "*****EVP Destructor called\n";
if(curCTX != NULL)
EVP_MD_CTX_free(curCTX);
}

// Hashing functions
bool OSSLEVPHashAlgorithm::hashInit()
{
std::cout << "***** In HashInit ****: ";
if (!HashAlgorithm::hashInit())
{
return false;
}

// Initialize the context
curCTX = EVP_MD_CTX_new();
if (curCTX == NULL)
{
ERROR_MSG("Failed to allocate space for EVP_MD_CTX");

return false;

}
std::cout << "***** In HashInit before EVPDigestInitex ****: ";
// Initialize EVP digesting
if (!EVP_DigestInit_ex(curCTX, getEVPHash(), NULL))
{
ERROR_MSG("EVP_DigestInit failed");
std::cout << "*****DigestInit failed\n";
EVP_MD_CTX_free(curCTX);
curCTX = NULL;

ByteString dummy;
HashAlgorithm::hashFinal(dummy);

return false;

}
std::cout << "*****End of HashInit ****: ";
return true;
}

bool OSSLEVPHashAlgorithm::hashUpdate(const ByteString& data)
{
std::cout << "***** In HashUpdate ****: ";
if (!HashAlgorithm::hashUpdate(data))
{
return false;
}

// Continue digesting
if (data.size() == 0)
{
return true;
}
std::cout << "***** In Hashupdate before EVPDigestUpdate ***: ";
if (!EVP_DigestUpdate(curCTX, (unsigned char) data.const_byte_str(), data.size()))
{
ERROR_MSG("EVP_DigestUpdate failed");
std::cout << "*****DigestUpdate failed\n";
EVP_MD_CTX_free(curCTX);
curCTX = NULL;

ByteString dummy;
HashAlgorithm::hashFinal(dummy);

return false;

}
std::cout << "***** End of HashUpdate ****: ";
return true;
}

bool OSSLEVPHashAlgorithm::hashFinal(ByteString& hashedData)
{
std::cout << "***** In HashFinal : " << curCTX;
if (!HashAlgorithm::hashFinal(hashedData))
{
return false;
}
std::cout << "* In HashFinal before resize : ";
size_t size = EVP_MD_size(getEVPHash());
std::cout << "* In HashFinal EVP_MD_SIZE returns " << size <<" : ";
hashedData.resize(size);
std::cout << " In HashFinal after resize : ";
unsigned int outLen = hashedData.size();
std::cout << "* In HashFinal before EVPDigestFinal ****: ";
if (!EVP_DigestFinal_ex(curCTX, &hashedData[0], &outLen))
{
ERROR_MSG("EVP_DigestFinal failed");
std::cout << "****EVP_DigestFinal failed\n";
EVP_MD_CTX_free(curCTX);
curCTX = NULL;

return false;

}
std::cout << "***** In HashFinal After EVPDigestFinal : ";
std::cout << "* In HashFinal before resize - 2****";
hashedData.resize(outLen);
std::cout << "***** In HashFinal after resize - 2****" << curCTX;
if (curCTX != NULL)
{ EVP_MD_CTX_free(curCTX);
curCTX = NULL;
}
std::cout << "***** After EVP_MD_CTX_FREE: " << curCTX;

std::cout << "***** End of HashFinal ****: " << curCTX;
return true;
}
`

These are getting called from RFC4880.cpp link to file

`// This function derives a 256-bit AES key from the supplied password data
bool RFC4880::PBEDeriveKey(const ByteString& password, ByteString& salt, AESKey** ppKey)
{
// Check that a proper salt value was supplied; it should be at least 8 bytes long
if (salt.size() < 8)
{
ERROR_MSG("Insufficient salt data supplied for password-based encryption");

return false;

}

// Check other parameters
if ((password.size() == 0) || (ppKey == NULL))
{
return false;
}

// Determine the iteration count based on the last byte of the salt
unsigned int iter = PBE_ITERATION_BASE_COUNT + salt[salt.size() - 1];

// Get a hash instance
HashAlgorithm* hash = CryptoFactory::i()->getHashAlgorithm(HashAlgo::SHA256);

if (hash == NULL)
{
ERROR_MSG("Could not get a SHA-256 instance");

return false;

}

// Perform the first iteration which takes as input the salt value and
// the password
ByteString intermediate;
std::cout << "**Before first iteration";
if (!hash->hashInit() ||
!hash->hashUpdate(salt) ||
!hash->hashUpdate(password) ||
!hash->hashFinal(intermediate))
{
ERROR_MSG("Hashing failed");

CryptoFactory::i()->recycleHashAlgorithm(hash);

return false;

}
std::cout << "**After first iteration";
// Perform the remaining iteration
std:: cout << "*******iteration count : " << iter << "********";
while (--iter > 0)
{
if (!hash->hashInit() ||
!hash->hashUpdate(intermediate) ||
!hash->hashFinal(intermediate))
{
ERROR_MSG("Hashing failed");

	CryptoFactory::i()->recycleHashAlgorithm(hash);

	return false;
}

}

// Create the AES key instance
*ppKey = new AESKey(256);
(*ppKey)->setKeyBits(intermediate);

// Release the hash instance
CryptoFactory::i()->recycleHashAlgorithm(hash);

return true;
}`

[shubhamsingh397]

It only happens on Linux machine while being called from another application, while using Windows build it works fine.

getting exception at line 416