Sunday, March 29, 2015

JCodec - Grabbing and saving picture frames from video

JCodec is an open source pure java implementation of video and audio codecs and formats (this statement was ripped from the website (http://jcodec.org/index.html). It's a nifty API, with a nice license (BSD), but sadly, as with open source software, the documentation can get a little bit stale.

The code listed on the website (as of Mar 29, 2015) to get a single frame works on v0.1.5, but as usual, being a person who wants to work with the latest and greatest piece of API, I got the v0.1.9 version from Maven (http://search.maven.org/#search%7Cga%7C1%7Cjcodec) and tried to run the code sample.

Alas, it didn't compile. After searching high and low for a remedy with the javadocs, and sidetracked to trying Apache Imaging (still in incubation), and took a look on OpenIMAJ (www.openimaj.org/), I still can't find a way to get a single frame with v0.1.9.

I finally got frustrated and just downloaded the v0.1.5 sources and see how it was implemented then. I realised that v0.1.9 removed the dependency on javax.imageio. That's why the example code didn't work. So I took some code out in v0.1.5 and reapplied to v0.1.9 just to test it out:


import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.File;

import javax.imageio.ImageIO;

import org.jcodec.api.FrameGrab;
import org.jcodec.common.FileChannelWrapper;
import org.jcodec.common.NIOUtils;
import org.jcodec.common.model.ColorSpace;
import org.jcodec.common.model.Picture;
import org.jcodec.scale.ColorUtil;
import org.jcodec.scale.Transform;
import org.junit.Test;

public class JCodecTests {

 @Test
 public void testSingleFrame_v019() throws Exception {
  int frameNumber = 150;
  Picture frame = FrameGrab.getNativeFrame(new File("D:/yk/mymovie.mp4"), frameNumber);

        Transform transform = ColorUtil.getTransform(frame.getColor(), ColorSpace.RGB);
        Picture rgb = Picture.create(frame.getWidth(), frame.getHeight(), ColorSpace.RGB);
        transform.transform(frame, rgb);
        BufferedImage bi = toBufferedImage(rgb);
        ImageIO.write(bi, "png", new File("frame_150.png"));
 }
}

It's a little bit 'exposed', but this would work, for now.

Lesson learnt: Go directly to the source! Long live open source!

Friday, March 28, 2014

Base64 in Java 8 - It's Not Too Late To Join In The Fun

Finally, Java 8 is out. Finally, there's a standard way to do Base64 encoding. For too long we have been relying on Apache Commons Codec (which is great anyway). Memory-conscious coders will desperately use sun.misc.BASE64Encoder and sun.misc.BASE64Decoder just to avoid adding extra JAR files in their programs, provided they are super sure of using only Sun/Oracle JDK. These classes are still lurking around in Java 8.

To try things out, I've furnished a JUnit test to show how to use the following APIs to encode:

  • Commons Codec: org.apache.commons.codec.binary.Base64
  • Java 8's new java.util.Base64
  • The sort-of evergreen internal code of Sun/Oracle's JDK: sun.misc.BASE64Encoder
package org.gizmo.util;

import java.util.Random;

import org.apache.commons.codec.binary.Base64;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.assertArrayEquals;

import sun.misc.BASE64Encoder;

public class Base64Tests {

 private static byte[] randomBinaryData = new byte[5000000];
 private static long durationCommons = 0;
 private static long durationJava8 = 0;
 private static long durationSun = 0;
 
 private static byte[] encodedCommons;
 private static byte[] encodedJava8;
 private static String encodedSun;
 
 @BeforeClass
 public static void setUp() throws Exception {
  
  //We want to test the APIs against the same data
  new Random().nextBytes(randomBinaryData);  
 }

 @Test
 public void testSunBase64Encode() throws Exception {
  
  BASE64Encoder encoder = new BASE64Encoder();

  long before = System.currentTimeMillis();

  encodedSun = encoder.encode(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationSun = after-before;
  System.out.println("Sun: " + durationSun);
 } 
 
 @Test
 public void testJava8Base64Encode() throws Exception {
  
  long before = System.currentTimeMillis();

  java.util.Base64.Encoder encoder = java.util.Base64.getEncoder();
  encodedJava8 = encoder.encode(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationJava8 = after-before;
  System.out.println("Java8: " + durationJava8);
 }
 
 @Test
 public void testCommonsBase64Encode() throws Exception {
  
  long before = System.currentTimeMillis();
  
  encodedCommons = Base64.encodeBase64(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationCommons = after-before;
  System.out.println("Commons: " + durationCommons);
 }

 @AfterClass
 public static void report() throws Exception {

  //Sanity check
  assertArrayEquals(encodedCommons, encodedJava8);
  System.out.println(durationCommons*1.0/durationJava8);
 }
}


What about the performance of these 3 ways? Base64 seems to be a small enough method so there are less ways to screw it up, but you'll never know what lies beneath the surface. From general timing (in the JUnit tests), it seems that the 3 methods can be arranged like this, from the fastest to the slowest: Java 8, Commons, Sun. A sample of the timing (encoding a byte array of size 5,000,000):

Sun: 521
Commons: 160
Java8: 37

Java 8's method ran 4x faster than Commons, and 14x faster than Sun. But this sample is just simplistic. Do try to benchmark for yourselves to come to your own conclusions.

So, which APIs to use? As any expert will tell you...it depends. If you have enough power to dictate that your code should only run on Java 8 and above, then by all means use the new java.util.Base64. If you just need to support multiple JDK versions and vendors, you can stick with Commons Codec or some other 3rd party API. Or wait until the older Javas to be out of circulation or usage, and rewrite your precious codebase. Or move on to another programming language.

Note: I did not even mention about using sun.misc.BASE64Encoder. Avoid it when possible. Perhaps one day this class will be removed in another (alos) version of JDK...it isn't present in other (heteros) JDKs by other vendors.

References:

Thursday, December 05, 2013

Insidi-DoS: Internally negotiated, self-image-damaging DoS

What's more damning and ego-bruising than an external DoS (denial of service) attack? Why, an internally inspired one of course. Which self-respecting company would do such a thing to itself? What's the rationale to this irrational behaviour? Recently, the largest bank in Malaysia (let's call it Big Bank) chose to rollout a Metro-style UI 'improvement' on its well-known, critically-acclaimed Internet banking system. Mass hysteria ensued as public users were unable to perform even simple fund transfers as usual. As usual, the Head of IT's head was on the line. Big Bank promptly reverted to the previous UI and initiated damage-control measures, such as issuing an apology via Facebook.

There are numerous articles addressing the lessons to learn, root cause and best practices to avoid such incidents from happening again. This article instead will give tips to REPEAT such incidents in your company (if you are in a position of power), because people in general don't like to read boring stuff to avoid disasters in future.

As mentioned, some netizens blamed IT for causing this unfortunate incident. In a way, IT is part of the problem, but usually, they are just the receiving end of orders from above. The corporate attitude of IT in Malaysia is that they are just cooks; they just cook when given the rice. No rice, no cooking. Maybe no cook as well. So who's the real culprit? I would like to conject/suggest that the true culprit is actually a man (not being sexist, may be a woman, but this is my conjecture), who belongs to the business division, and an exuberant fan of new technology. So how does he concoct his strategy to rollout a new change and cause havoc to the Internet banking users of Big Bank?

This man with a mission may think this way:

1. Launch it quick
Launch it before the year ends. Why? Maybe people remember stuff better at year end. Great starts at the beginning of the year may be forgotten when the year is ending. Why not end the year with a bang? It doesn't matter if IT insisted that freeze period is around the corner. Launch it just before freeze period and the IT folks can also be given rest, right? Oh, but what about serious issues like...unable to login? Unable to transfer funds? Don't worry, users can still use phone banking, ATMs, even pay a visit to the branches. Alternative channels, dude! Not a serious concern, just launch it, NOW!

2. Launch it BIG
What good is a change if no one notices? The media may not cover it, the Facebook 'likes' may not even hit 4 digits, and no hoo-haa like 'Jom Heboh'. I know...just launch it during the MONTH-END! Payroll time is the best time. Even users that were forced to use Internet Banking will most likely need to login just to check if their salary is in. That way, you get maximum publicity and Big Bank can again bag yet another award for 'Most Innovative Design' for an Internet banking system.

3. No options aka I know what is good for you
Oh, why does Gmail and even Yahoo have to inform users that there's a new UI for their systems and pain-stakingly provide online cues and dialog boxes to guide the users? Come on, this UI change is awesome! Facebook did rollout a new UI and initially users were mad, but they then normalized and happily got back into broadcasting their personal data for free. Wait, Gmail, Yahoo and Facebook are free sites, but Big Bank is not. But Big Bank does not charge its users to use Internet Banking, right? We attempted to charge last decade but it backfired, so why should the users complain about the UI? It's GOOD for them! Good things must be shoved into their faces. Makes socialism such a fuzzy and 'noble' ideal... 

4. No need for stress testing, but when I do, I do it in PRODUCTION
Oh, stress testing stresses me up. The incessant whining of IT that our test environment is not up to par due to cost-saving measures to reduce IT expenditure really vexes me. What's wrong with saving cost? If I need to wait for stress testing to complete, I am not able to launch it quick, and big. Anyway, what's so performance-impacting than a UI change? It's like wearing a mask, right? You can still see, breathe, eat and hear when donning the mask. Oh, I love myself because I am so good at analogy.

Of course, in actual fact, it rarely boils down to a single person being the root cause to such incidents. Or maybe it really boils down to that. No one knows. I am not an insider in Big Bank. All I can say is, Bang2Bank.

Tuesday, November 26, 2013

Accessing Git Repositories using SSH behind HTTP proxy (TortoiseGit, Putty)

There's an easier way to do this since my previous post on the subject above. SSH key generation steps are not discussed here, but the only thing worth mentioning here is that the private key be in PPK format (use PuTTYgen to convert them)

Using TortoiseGit, use Putty to save a 'Session'. Name the session short and simple e.g. hornetsnest. This is significant and to make things simpler, no spaces for the session names.

Under the saved Putty session, the place to add proxy information is at Connection > Proxy. For HTTP proxy:
  • Proxy hostname - IP or host name of proxy server
  • Port - Proxy server port e.g. 8080
  • Username - User login Id if proxy server requires authentication
  • Password - Login Id password
To associate a SSH key, go to SSH > Auth and specify the PPK key file path under "Private key for authentication".

Remember to save the Putty session.

Now, shift over to TortoiseGit. In the window to specify the Git repository URL, instead of the usual host name value, replace it with the Putty session name. In this example, hornetsnest. So

ssh://workerbee@imperialpalace.ip/~/git/super.git

becomes

ssh://workerbee@hornetsnest/~/git/super.git

So the connection to the Git repository via SSH will use the settings in the Putty's session. Of course, when you are not behind a proxy, you can just revert to the original URL when cloning/pulling/pushing.

Minor update: You'll need to use TortoisePlink as SSH client:


Monday, September 02, 2013

Running Apache/IHS on privileged ports (80, 443) as a non-root user: The 'sudo' way

Overview

For some out-of-this-world reason, your Unix administrator fears giving you the root access. But your web server (Apache or IBM HTTP Server) needs to bind to a privileged port e.g. 80, 443, and root privilege is needed. Using 'sudo', you can at least not bother your administrator to start/stop/restart anymore. He would either be happy that no one's calling to bug him anymore, or fear that someone like you has found a way to do something without his 'power'.

Steps below are for IBM HTTP Server (IHS) 8.5 running on AIX. Applicable for other versions & OSes with slight variations.

Assumptions


  • IHS location: /opt/IBM/HTTPServer
  • Using default httpd.conf @ /opt/IBM/HTTPServer/conf/httpd.conf
  • HTTP port to listen/bind: 80
  • User & group to delegate after startup: wasadmin:wasadmin
  • Logs folder & files: /opt/IBM/HTTPServer/logs - access_log and error_log
  • You have root access (for the time being)

Steps


1. 'Touch' the log files:

touch /opt/IBM/HTTPServer/logs/access_log
touch /opt/IBM/HTTPServer/logs/error_log

2. Make the following changes in httpd.conf (ensure user 'wasadmin' and group 'wasadmin' has been created already)

User wasadmin
Group wasadmin

3. Change ownership of IHS files:

chown -R wasadmin:wasadmin /opt/IBM/HTTPServer

4. Edit sudoers file (visudo), add the following line:

%wasadmin ALL = (root) NOPASSWD: /opt/IBM/HTTPServer/bin/apachectl *

This means any user in the 'wasadmin' group (prefixed with %) can control (start/stop) the IHS instance as root without needing to enter password. If you only want to allow the user 'wasadmin' to perform this, then remove the '%' to denote a user.

5. Ok, let's start IHS as user 'wasadmin'.

From root prompt:
sudo su - wasadmin

To check current user: id

Sample output:
uid=12(wasadmin) gid=203(wasadmin) groups=0(system)

Run as root without password:
sudo -u root /opt/IBM/HTTPServer/bin/apachectl start

To verify, fire up web browser and access the default webpage e.g. http://192.168.1.128 (port 80 is implied if not specified for HTTP).

Note: If you moved/deleted the log files, you need to touch them again using the non-root user as in Step #1 (e.g. wasadmin). Else the log files will be created with root ownership.

Tuesday, May 14, 2013

Invoking EJB3 in WebSphere using Spring


 Here are the steps to call a method of an EJB (v3.0) deployed in WebSphere 7 from a plain old Java client, either using Spring to shield your code from unnecessary implementation details, or bare-bones straightforward coding style.

Note: Tested against WAS 7 only.

Common Steps

1. Firstly, the Remote interface class is needed and will be referenced in your client code to call the EJB method. Copy the class file or Java source to your Java client workspace. In my example, the remote interface class is org.gizmo.minyakangin.CapKapakRemote
2. If you have a local development WebSphere 7, run the following command to create the stub class

[Class_Folder_Path]:>[WAS_HOME]\AppServer\bin\createEJBStubs.bat org.gizmo.minyakangin.CapKapakRemote -cp .

You should see an output similar to the one below:

Processing the org.gizmo.minyakangin.CapKapakRemote input file.
Command Successful

A file with the name _CapKapakRemote_Stub.class will be created under folder [Class_Folder_Path]\org\gizmo\minyakangin.

3. A missing stub file may produce the following errors when getting a reference to the EJB:

java.lang.ClassCastException: Unable to load class: org.gizmo.minyakangin._CapKapakRemote_Stub

java.lang.ClassCastException: org.omg.stub.java.rmi._Remote_Stub incompatible with org.gizmo.minyakangin.CapKapakRemote

4. At the WebSphere Admin Console, locate the EJB application and ensure a proper JNDI reference is assigned e.g. ejb/CapKapak. It is best not to rely on the default JNDI reference. But if you insist, the default is the fully qualified remote class name i.e. org.gizmo.minyakangin.CapKapakRemote.



5. Also, check the bootstrap port of the application server instance hosting the EJB (my environment is 9812):



6. At your Java client workspace, ensure that com.ibm.ws.webservices.thinclient_7.0.0.jar (under [WAS_HOME]\AppServer\runtimes) and com.ibm.ws.runtime.jar (under [WAS_HOME]\AppServer\plugins) are included in the classpath. Not sure why including only com.ibm.ws.ejb.thinclient_7.0.0.jar doesn't work in my case.

Caveat: The steps only work using IBM JDK (v7). You may need to do some 'hacking' to get it working under a different JVM.


Using Spring 3.x

You can interface with a plain Java interface (no EJB classes or annotations needed) using Spring. Here's the interface:

package org.gizmo.minyakangin;

public interface CapKapak {
	public String getBrand();
}


An EJB3 'bean' can be easily constructed using a JNDI object factory bean, and it's even easier to just use the 'jee' namespace, as below:




	
		
			java.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
			java.naming.provider.url=corbaloc:iiop:localhost:9812
		
	





Note: The 'java.naming.provider.url' value should refer to the WebSphere server's bootstrap address as shown above.

Then in the code (using Spring's JUnit class):

package org.gizmo.minyakangin;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.AbstractJUnit4SpringContextTests;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {
		"classpath:org/minyakangin/spring-context.xml"})
public class EjbTests extends AbstractJUnit4SpringContextTests {

	CapKapakRemote capKapak;
	
	@Before
	public void before() {
		capKapak = (CapKapakRemote)applicationContext.getBean("capKapak");
	}
	
	@Test
	public void testLoginManager() throws Exception {
		String brand = capKapak.getBrand();
		System.out.println(brand);
	}
}


Using plain Java


package org.gizmo.minyakangin;

import java.util.Properties;

import javax.naming.InitialContext;
import javax.rmi.PortableRemoteObject;

import com.cv.ibs.user.session.LoginManagerRemote;
import com.cv.ibs.utils.SessionUser;

public class StandaloneEjbClient {

	public static void main (String args[]) throws Exception {
		Properties props = new Properties();
		props.put("java.naming.factory.initial", "com.ibm.websphere.naming.WsnInitialContextFactory");
		props.put("java.naming.provider.url", "corbaloc:iiop:localhost:9812");

		InitialContext ctx = new InitialContext(props);
		
		Object ejbBusIntf = ctx.lookup("ejb/CapKapak");
		CapKapakRemote loginManager = (CapKapakRemote)PortableRemoteObject.narrow(ejbBusIntf, CapKapakRemote.class);
		String brand = capKapak.getBrand();
		System.out.println(brand);
	}
}





Tuesday, April 16, 2013

Strategy Pattern Applied - KeyStoreTemplate

Update: I wrongly attributed the design pattern implementation below to the Template pattern, it's actually the Strategy pattern as pointed out by a user in Reddit. However, the article below is kept in its original form, with just the title changed. Sorry for the confusion :)

The Template design pattern is useful if you have methods that follow a similar pattern but differ only in a few places. Usually, the 'generic' method will call one or several abstract methods, which are then implemented in a concrete class.

The JdbcTemplate of the Spring Framework was (and still is) a life saver. No more unclosed Connection objects that may leak to JDBC connection leaks. No more try/catch blocks that obscure the code logic. Converting the JDBC DAOs of legacy systems to use JdbcTemplate objects provided immediate benefits in terms of simplicity (eliminating copy-and-paste codes) and robust resource management (e.g. closing Connections, ResultSets, Statements).

There are a lot of useful articles detailing what, why and how of this design pattern, so I will not rehash them here. I will just document down how I applied this pattern to solve some age-old problem.

I decided to follow the JdbcTemplate's design style to come up with a KeyStoreTemplate, a class that handles importing or exporting of keys to/from a keystore. To make things simple, this class will only handle 2 things: Importing a key, and exporting a key.

Firstly, the KeyStoreTemplate code (abridged for simplicity):

import org.apache.commons.io.IOUtils;

public class KeyStoreTemplate {

 /**
 * Just enough parameters to open a keystore.
 */
 public KeyStoreTemplate(File keyStoreFile, Provider provider, String keyStoreType, String keyStorePassword) {
  this.keyStoreFile = keyStoreFile;
  this.provider = provider;
  this.keyStoreType = keyStoreType;
  this.keyStorePassword = keyStorePassword;
 }

 /**
 * This is to get/export a key from the keystore.
 */
 public Key getFromKeyStore(KeyStoreExporter exporter) {
  FileInputStream fis = null;
  try {
   KeyStore ks = KeyStore.getInstance(keyStoreType, provider);
         fis = new FileInputStream(keyStoreFile);

         //Keystore file must exists!
         if(keyStoreFile.exists()) {
          ks.load(fis, keyStorePassword.toCharArray());
         } else {
          throw new RuntimeException("Keystore file cannot be located!");
         }
         
         String keyPass = keyStorePassword;

         return exporter.exportKey(ks, keyPass);
         
  } catch (Exception e) {
   throw new RuntimeException("Something went wrong!", e);
  } finally {
   IOUtils.closeQuietly(fis);
  }  
 }

 /**
 * This is to save/import a key to the keystore.
 */
 public void saveToKeyStore(KeyStoreImporter importer) {
  
  FileInputStream fis = null;
  FileOutputStream fos = null;
  try {
   KeyStore ks = KeyStore.getInstance(keyStoreType, provider);
         fis = new FileInputStream(keyStoreFile);

         //Keystore file must exists!
         if(keyStoreFile.exists()) {
          ks.load(fis, keyStorePassword.toCharArray());
         } else {
          throw new BusinessException("Keystore file cannot be located!");
         }
         
         String keyPass = keyStorePassword;

         importer.importPrivateKey(ks, keyPass);
         
         fos = new FileOutputStream(keyStoreFile);
         ks.store(fos, keyStorePassword.toCharArray());
         
  } catch (Exception e) {
   throw new RuntimeException("Something went wrong!", e);
  } finally {
   IOUtils.closeQuietly(fis);
   IOUtils.closeQuietly(fos);
  }
 }
 
}

Notice that there are 2 interfaces: KeyStoreExporter and KeyStoreImporter. These interfaces are implemented by the caller class of KeyStoreTemplate to do the 'effective' stuff, without worrying about the KeyStore details. The KeyStore object is passed in, much like the ResultSet object is passed in when using JdbcTemplate.

import java.security.GeneralSecurityException;
import java.security.KeyStore;

public interface KeyStoreImporter {

 public void importPrivateKey(KeyStore keyStore, String keyPassword) throws GeneralSecurityException;
}
import java.security.GeneralSecurityException;
import java.security.Key;
import java.security.KeyStore;

public interface KeyStoreExporter {

 public Key exportKey(KeyStore keyStore, String keyPassword) throws GeneralSecurityException;
}

So, how to use them? Example as below:

//Key generation and key import
KeyPair keyPair = generateRSAKeyPair();
final Certificate[] certs = CryptoUtils.generateCertificate(keyPair);

RSAPublicKey publicKey = (RSAPublicKey)keyPair.getPublic();
final RSAPrivateKey privateKey = (RSAPrivateKey)keyPair.getPrivate();

logger.debug("[generateAndStoreRSAKeyPair] pubKey={},privKey={}", publicKey.getClass(), privateKey.getClass());

keyStoreTemplate.saveToKeyStore(new KeyStoreImporter() {
 @Override
 public void importPrivateKey(KeyStore ks, String keyPassword) throws KeyStoreException {
  ks.setKeyEntry(alias, privateKey, keyPassword.toCharArray(), certs);
 }
});
//Method to get an RSA public key
public RSAPublicKey getRSAPublicKey(final String alias) throws BusinessException {

 return (RSAPublicKey)keyStoreTemplate.getFromKeyStore(new KeyStoreExporter() {
  @Override
  public Key exportKey(KeyStore keyStore, @SuppressWarnings("unused") String keyPassword) throws GeneralSecurityException {
   Certificate cert = keyStore.getCertificate(alias);
   RSAPublicKey publicKey = (RSAPublicKey)cert.getPublicKey();
   return publicKey;
  }
 });
}
//Method to get a secret key
public SecretKey getSecretKey(final String alias) throws BusinessException {

 return (SecretKey)keyStoreTemplate.getFromKeyStore(new KeyStoreExporter() {
  @Override
  public Key exportKey(KeyStore keyStore, String keyPassword) throws GeneralSecurityException {
   SecretKey secretKey = (SecretKey)keyStore.getKey(alias, keyPassword.toCharArray());
   return secretKey;
  }
 });
}