Bad Certificate in WAS 5.1.0 (IBM JDK 1.4.1)

Once upon a time, there was a legacy web application system handling loans processing. It ran on Windows 2000 but used IBM WebSphere Application Server (WAS) 5.1.0 installation (yes, no fixpacks!). It ran well until recently when the system tried to access a remote URL using HTTPS/SSL, an error occurred. The log files were investigated and to the horrified faces of the support personnel...

javax.net.ssl.SSLHandshakeException: bad certificate

A few recommendations here and there led to them summoning me to have a look at this issue. At first I thought it was just a simple case of not importing the SSL certificates into the keystore. But after using ikeyman (WAS tool) to do the necessary, it still doesn't work. Ok, time to Google...

Version 5.1.0 of WAS runs on IBM JDK 1.4.1 (not even 1.4.2), which made matters worse. A quick search using Google yielded 2 most relevant results:

• http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14158375

• http://www.ibm.com/developerworks/forums/thread.jspa?messageID=5132527

Problem is, both links didn't actually provide any solution, but they did nudge me a little to a workaround which I am documenting here right now.

This problem is not common if the WAS installations are patched up to at least v5.1.1, but due to the system's legacy status and most likely be replaced by another system early next year, there was no real incentive to patch it and 'hope for the best'. The personnel of the vendor supporting the system was also a blur case contract worker who spew common buzzwords yet lack any substance and logic.

So, what to do, what to do??? There were no source codes available (the vendor decided not to give an earlier contract worker his time off, so he retaliated by deleting all the source codes...a case of coder gone cuckoo/crazy). So, I decided to decompile the source codes using JD-GUI (http://java.decompiler.free.fr/?q=jdgui) after learning from the IT support girl on the URL that triggers the remote host handshaking. A quick look at the web.xml file and the servlet responsible for the remote request was identified.

After looking through the servlet code, I was able to identify the root cause and reconfirmed it by writing some JUnit tests:

• Test #1 - Using Sun JDK1.4.2, no imported certs: Fail, "unknown certificate" (expected)

• Test #2 - Using Sun JDK1.4.2, with imported certs: Pass (expected)

• Test #3 - Using IBM JDK1.4.1 (similar to WAS 5.1.0), no imported certs: Fail (expected)

• Test #4 - Using IBM JDK1.4.1 (similar to WAS 5.1.0), with imported certs: Fail, "bad certificate" (expected)

So the problem is the JDK, not the application code (sort of). IBM JDK1.4.1 is using IBM JSSE v1 (not even v2). I wanted to try using IBM JSSE v2 API files, but after looking through several Google results, no one was triumphant doing this.

To call the remote URL using HTTPS/SSL, Apache HttpClient API was used. A quick look at the online guide (http://hc.apache.org/httpclient-3.x/sslguide.html) gave me an idea to replace the default class to handle HTTPS connections with a custom class which can be coded to use Sun JSSE implementation. To do this, some class-overriding was necessary. The final code for the socket factory (I had to dig through the Sun JSSE classes to find the implementation class which actually is 'internal' and not recommended to be referenced externally, but desperate times require desperate measures):

import java.io.IOException;
import java.net.InetAddress;
import java.net.Socket;
import java.net.UnknownHostException;

import org.apache.commons.httpclient.protocol.SecureProtocolSocketFactory;

import com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl;

public class SunJsseSslSocketFactory implements SecureProtocolSocketFactory {

 public Socket createSocket(String host, int port) 
  throws IOException, UnknownHostException
 {
  SSLSocketFactoryImpl sfi = new SSLSocketFactoryImpl();
  return sfi.createSocket(host, port);
 }

 public Socket createSocket(String host, int port, 
  InetAddress clientHost, int clientPort) 
   throws IOException, UnknownHostException
 {
  SSLSocketFactoryImpl sfi = new SSLSocketFactoryImpl();
  return sfi.createSocket(host, port, clientHost, clientPort);
 }

 public Socket createSocket(Socket socket, String host, 
  int port, boolean autoClose) 
   throws IOException, UnknownHostException
 {
  SSLSocketFactoryImpl sfi = new SSLSocketFactoryImpl();
  return sfi.createSocket(socket, host, port, autoClose);
 }

}

Then, to modify the HttpClient call to use this new socket factory class:

Protocol myHTTPS = new Protocol( "https", new SunJsseSslSocketFactory(), url.getPort() );

This line is placed before any calls to the remote URL, and only needed to be executed once.

2 other steps to do to complete the fix:

• Place the Sun JSSE file (jsse.jar) into [WAS_HOME]/java/jre/lib/ext

• Add the Sun JSSE provider into file [WAS_HOME]/java/jre/lib/security/java.security:

security.provider.6=com.sun.net.ssl.internal.ssl.Provider

In the end, the fix was applied to the production boxes and the remote call finally worked. All credit goes to the HttpClient API makers who at least coded some hooks for this customisation.