Sunday, June 09, 2019

wget on Windows


This is to document my steps to download all image (JPG) files along with PDF and regular HTML files instead of using the web browser, using only 1 command (wget).


Use Choco ( Follow installation instructions @

Then open a command prompt with administrative rights to install wget:
choco install wget


My target website (say is protected by BASIC authentication. I am only interested in downloading files with extensions *.jpg, *.pdf & *.html. So I will create a directory to have the files placed i.e. c:\abc. Then, just run the commands below:
cd c:\abc 
wget --user-agent="Googlebot/2.1 (+" --http-user=user123 --http-password=coder4life -A "*.jpg,*.html,*.pdf" -r -l=0

--user-agent = User agent string to let the web server of target website to know about the kind of client/browser that is connecting. If not specified the value is "wget" which some web servers may block access

--http-user = BASIC username

--http-password = BASIC password (plain text)

-A = Inclusion list to download

-r = Tells wget to recursively get files (search the website for all possible paths/files)

-l = How "deep" should wget go. Default is 5, meaning from the URL, wget can go until /folder123/1/2/3/4/5 and stop looking. The command above has value 0, which means "infinite" (until all possible paths are traversed)

Monday, April 24, 2017

North Korea: China or The USA's Opportunity?


North Korea's sabre-rattling tactics by launching missiles and conducting nuclear tests may be seen as an internal affair to boost confidence in Kim Jong-Un's regime, but with its neighbours being alarmed, and attracting the attention of the USA, the hermit kingdom is no longer insulated from world affairs. China, the emerging superpower, will be hard-pressed to act, while the USA, the current superpower, is also pondering on military action should things continue to escalate. Many see this geopolitical event as a problem. But this represent an opportunity for both sides.

Historical Background

The Korean peninsula has always been a buffer zone between its bigger neighbours; China to the west, Japan to the east, and Russia to the north. Chinese, Korean and Japanese cultures all share similarities, yet all have distinctives that separate them. In the early 20th century, China was so weakened by foreign intervention that Japan saw an opportunity to expand its territories westward. First, by annexing the Korean peninsula in 1910. Then, in 1931, Japan invaded and occupied Manchuria. China proper was then invaded in 1937.

The Chinese mainland territories captured by Japan were still controlled by them until 1945. After Europe was liberated by the Allies in May 1945, focus was shifted to Japan. USSR then agreed to enter the war against Japan. The Soviet Union invaded Japanese-occupied Manchuria in August 1945. At about the same time, the Americans dropped the atomic bombs on Hiroshima and Nagasaki. Japan then surrendered unconditionally.

The United Nations then developed plans for a trusteeship administration. The Soviet Union administered the peninsula north of the 38th parallel, while the United States administered the south. This resulted in the 1948 establishment of two separate governments, North Korea and South Korea. Both governments were molded in the image of their administrators.

In 1949, the Communist Party took control of China. The Soviet Union then welcomed China into its sphere of influence. Along with North Korea, the Soviets then have a buffer zone against the USA.

From this, after the Second World War, China needed to re-establish the Korean peninsula as a buffer zone against future invasions from the east. Even as the Japanese threat was eliminated by their defeat and surrender at the hands of the Allies, a new adversary suddenly emerged: The United States of America. Remember that the whole Korean peninsula was controlled by China. A southern Western-like state so close to the mainland gave the Chinese leaders restless nights.

When the Korean War broke out, China took a back seat at first, thinking that the North Korean Army, armed with Soviet weapons and trained by the Soviet army could do the job in unifying Korea. When the United Nations army was very close to capturing the entire peninsula, the Chinese Army suddenly mobilised and in a twist of events too long to discuss here, managed to get away with a ceasefire and retained the division of North & South Korea along the 38th parallel.

At that time, China's intervention was justified in the Communist world, having saved their smaller comrade from the jaws of defeat. A Korean peninsula half-controlled by another Communist regime is surely better than the entire peninsula controlled by the opposite camp. Chinese casualties were high but the fallen were celebrated as heroes for the great communist cause.

The Present Situation

North Korea's unbridled nuclear tests, along with missile launches from land and sea, is unnerving even for China, its greatest supporter. It is one thing to have this 'puppet' state occupy Western minds and shake the Japanese, but another if the provocations cause a military reaction to 'bring peace' to the region. It will be another Korean War, this time not of the communists' choosing. China now is not as dogmatic as China in the 1950s. This however does not eliminate the fact that she still needs a buffer zone.

Economics is a strong factor when going to war. The Japanese went all-out in the Second World War to capture resource-rich territories that they see as essential for their nation's benefit. Then, it was painful to have a lack of resources. Now, it is painful to expend resources in war. Both China and the US are seen as capable to intervene militarily, since both have similar interest to control the East Pacific regions.

Scenario A: The US Attacks, China Idles

As the incumbent superpower in the region, the US is still the protector for South Korea and Japan. A strike from North Korea to the 2 countries will surely pressure the US to respond. As the North Koreans have boasted on national TV that they are capable of challenging the US, the Trump administration can use this opportunity to eliminate this threat by military intervention.

A defeated North Korean regime can be a positive outcome, with which South Korea have the responsibility to rebuild its northern sibling. The US, through South Korea, can also offer assistance and this will spark demand for infrastructure and construction. The US can send a signal to the rest of the world of its status as a superpower & not to be messed with. It may also result in China being viewed as to have forsaken its neighbour for economic expediency, since ideology no longer is a strong factor in national decisions.

It will be a sign of weakness to the rest of the East Pacific nations, as China did not act to the threat so near its mainland, the likelihood of them really challenging Vietnam, the Philippines, Taiwan etc. over control of the nine-dash line (South China Sea region). This will boost the smaller nations' will to continue challenging China's claims of sovereignty.

Scenario B: The US Attacks, China Defends

This will surely spark internal outrage in China. Why send troops to defend North Korea in this time of economic uncertainty and at the cost of Chinese lifes just to maintain the Kim regime? It may boost the confidence of regional allies and 'neutral' nations, but internal backlash is something China could not afford.

If the outcome of the war is another stalemate, it will provoke further outrage as it will be compared to the first Korean War. This time, there's no strong ideological reason to justify shedding Chinese blood on foreign soil. Mao Zedong was the king in the party then, but Xi Jinping is no supreme commander now, though he is the leader.

Scenario C: The US Idles, China Attacks

This sounds preposterous, but in actual fact, is a possibility as history has shown. A region of instability will invite trouble from other nations that sees this as a 'low-hanging fruit'. That was how Japan annexed Korea in 1910. A weak China precipitated this. As the Kim regime is universally despised by the world, and internally-pressured to remain adored by its own people, a sudden invasion will give Kim Jong Un a heart attack and time to respond will be longer.

Militarily, this gives China the opportunity to gain experience by participating in combat within the same region. It can be a rehearsal for the invasion of Taiwan, though this requires dominance at sea. The Chinese army seen as liberating North Korea from the hegemony of the Kim regime will be a better morale-booster than defending the Kim regime.

China, emerging as the victor, will be able to strike a deal with South Korea in rebuilding the North. This will increase demand in the construction and infrastructure sector. This is surely a better way to use resources and manpower rather than building ghost cities in 'no man's land'. As happened in Germany, where the Western side had to pay for the Eastern side's debts and rebuilding costs, so will South Korea do the same for their Northern sibling.

This will be an eye-opener for South Korea as the US is no longer integral in its security and may look to China taking over the role. Perhaps there isn't a need for any nation to offer 'protection' if the Koreans view China as benevolent. But this may shake up Japan as China still has not forgotten World War II.

Scenario D: The US & China Attacks

This requires military coordination between both nations, which is a weak area since both nations regard each other with great suspicion, each trying to subvert rather than cooperate. A land invasion from China and an aerial assault by the US can be possible, as along as both militaries do not interfere with each other. Limited cooperation and coordination is still possible, as in the 2-side front in Europe against Nazi Germany by the Allies.


No doubt the US and China are challenging each other for influence in East Asia. In the Cold War era, choosing sides was easier, what more choosing a victor in World War II. China's emergence as a regional superpower is changing the status quo. Then, nations going to war with each other are not tightly coupled together economically/financially. When this coupling is severed, as happened between Japan and the US in the 1940s, war followed. Both the US and China are too interdependent with each other, but both also do not want to be seen as weak and losing ground. This complicates matters and given the circumstances, both nations will want to avoid going to war with each other.

If war is unavoidable, then Scenario C will favour China in the aftermath. China is one of the "Kings of the East". No power can take it lightly, even North Korea. I quipped to my South Korean friends of my next visit to "Korea" will be in Pyongyang. Perhaps this year will be the right year.

Tuesday, April 18, 2017

GoAccess: Visualize Your Web Traffic

If you want a visual representation of your web traffic, try GoAccess ( You will never let it go.

I am just generating a report using static log files. GoAccess has many other features (real-time updates, console access, JSON/CSV-exporting etc.) that I haven't got the need to use (for now).

Tools Used:

- Mac OS 10.12
- GoAccess v1.2


brew install goaccess


Retrieving logs from the web servers (Apache 2.4-based), sample output: - - [26/Mar/2017:02:41:09 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 200 196 314 - - [26/Mar/2017:02:36:26 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 304 - 314 - - [25/Mar/2017:22:47:19 +0800] "GET /retail/mce/css/ HTTP/1.1" 304 - 314 - - [25/Mar/2017:17:46:35 +0800] "GET /retail/mce/css/icons.fallback.css HTTP/1.1" 304 - 314

Tip #1: To remove 'noise' if your web server is shared by multiple apps, use 'grep' to filter patterns that you are interested:
grep -i 'mypattern' access_log-20170328 > filtered_access_log-20170328

Tip #2: Although GoAccess has the ability to get input from multiple files, you can also combine all your log files via 'cat':
cat a.log b.log c.log > d.log

Log format settings from httpd.conf:
LogFormat "%h %l %u %t \"%r\" %>s %b %D" common

Note: The assumption was to take the Apache log format verbatim, since GoAccess supports the Apache log file format. After much trial and error, the log format for GoAccess is as below. The most frustrating issue was the response times getting rendered as 0.00us due to the inclusion of ">" in %>s. GoAccess only recognises %s.

Open file: /usr/local/Cellar/goaccess/1.2/etc/goaccess.conf, and set the following (adjust based on your own log patterns):
time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %l %u [%d:%t %^] "%m %U %H" %s %b %D

Save the file after editing, and run the following (assuming log filename is access_log):
goaccess access_log -o report.html -q

-o: Report file. For HTML report, specify a filename with *.html extension
-q: Ignore query string i.e., =>

My output/report:

The man page ( has the full options available.

Side note: Why not AWStats ( It was my 1st choice, but I was working on a Macbook, and the easiest route was GoAccess (just use brew to install). AWStats needed Perl and only had ready RPMs at its website's download page. Plus, GoAccess has a more impressive UI dashboard, and I needed to show some fancy stats to biz...

Thursday, June 02, 2016

Update Microsoft Windows if Windows Update seems to be 'stuck'

Sometimes old laptops/PCs have trouble updating via Windows Update after a new installation. Even after leaving the machine running, you will wake up in dismay knowing that you left the machine running without achieving anything.

Steps to do so:
  1. Stop Windows Update service (Run: services.msc)
  2. Go to:
  3. Grab the latest download.
  4. Unzip, then there are 2 parts:
    1. Run UpdateGenerator.exe at folder root. Select appropriate Windows version (64-bit is x64). Then click Start button. Wait till it finishes downloading.
    2. Navigate to 'client' folder (e.g. wsusoffline\client). Run UpdateInstaller.exe. Then click Start button.
  5. Once done, there will be multiple restarts needed. After the dust is settled, can run Windows Update like normal.
Helpful link:

Sunday, March 29, 2015

JCodec - Grabbing and saving picture frames from video

JCodec is an open source pure java implementation of video and audio codecs and formats (this statement was ripped from the website ( It's a nifty API, with a nice license (BSD), but sadly, as with open source software, the documentation can get a little bit stale.

The code listed on the website (as of Mar 29, 2015) to get a single frame works on v0.1.5, but as usual, being a person who wants to work with the latest and greatest piece of API, I got the v0.1.9 version from Maven ( and tried to run the code sample.

Alas, it didn't compile. After searching high and low for a remedy with the javadocs, and sidetracked to trying Apache Imaging (still in incubation), and took a look on OpenIMAJ (, I still can't find a way to get a single frame with v0.1.9.

I finally got frustrated and just downloaded the v0.1.5 sources and see how it was implemented then. I realised that v0.1.9 removed the dependency on javax.imageio. That's why the example code didn't work. So I took some code out in v0.1.5 and reapplied to v0.1.9 just to test it out:

import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;

import javax.imageio.ImageIO;

import org.jcodec.api.FrameGrab;
import org.jcodec.common.FileChannelWrapper;
import org.jcodec.common.NIOUtils;
import org.jcodec.common.model.ColorSpace;
import org.jcodec.common.model.Picture;
import org.jcodec.scale.ColorUtil;
import org.jcodec.scale.Transform;
import org.junit.Test;

public class JCodecTests {

 public void testSingleFrame_v019() throws Exception {
  int frameNumber = 150;
  Picture frame = FrameGrab.getNativeFrame(new File("D:/yk/mymovie.mp4"), frameNumber);

        Transform transform = ColorUtil.getTransform(frame.getColor(), ColorSpace.RGB);
        Picture rgb = Picture.create(frame.getWidth(), frame.getHeight(), ColorSpace.RGB);
        transform.transform(frame, rgb);
        BufferedImage bi = toBufferedImage(rgb);
        ImageIO.write(bi, "png", new File("frame_150.png"));

It's a little bit 'exposed', but this would work, for now.

Lesson learnt: Go directly to the source! Long live open source!

Friday, March 28, 2014

Base64 in Java 8 - It's Not Too Late To Join In The Fun

Finally, Java 8 is out. Finally, there's a standard way to do Base64 encoding. For too long we have been relying on Apache Commons Codec (which is great anyway). Memory-conscious coders will desperately use sun.misc.BASE64Encoder and sun.misc.BASE64Decoder just to avoid adding extra JAR files in their programs, provided they are super sure of using only Sun/Oracle JDK. These classes are still lurking around in Java 8.

To try things out, I've furnished a JUnit test to show how to use the following APIs to encode:

  • Commons Codec: org.apache.commons.codec.binary.Base64
  • Java 8's new java.util.Base64
  • The sort-of evergreen internal code of Sun/Oracle's JDK: sun.misc.BASE64Encoder
package org.gizmo.util;

import java.util.Random;

import org.apache.commons.codec.binary.Base64;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.assertArrayEquals;

import sun.misc.BASE64Encoder;

public class Base64Tests {

 private static byte[] randomBinaryData = new byte[5000000];
 private static long durationCommons = 0;
 private static long durationJava8 = 0;
 private static long durationSun = 0;
 private static byte[] encodedCommons;
 private static byte[] encodedJava8;
 private static String encodedSun;
 public static void setUp() throws Exception {
  //We want to test the APIs against the same data
  new Random().nextBytes(randomBinaryData);  

 public void testSunBase64Encode() throws Exception {
  BASE64Encoder encoder = new BASE64Encoder();

  long before = System.currentTimeMillis();

  encodedSun = encoder.encode(randomBinaryData);
  long after = System.currentTimeMillis();
  durationSun = after-before;
  System.out.println("Sun: " + durationSun);
 public void testJava8Base64Encode() throws Exception {
  long before = System.currentTimeMillis();

  java.util.Base64.Encoder encoder = java.util.Base64.getEncoder();
  encodedJava8 = encoder.encode(randomBinaryData);
  long after = System.currentTimeMillis();
  durationJava8 = after-before;
  System.out.println("Java8: " + durationJava8);
 public void testCommonsBase64Encode() throws Exception {
  long before = System.currentTimeMillis();
  encodedCommons = Base64.encodeBase64(randomBinaryData);
  long after = System.currentTimeMillis();
  durationCommons = after-before;
  System.out.println("Commons: " + durationCommons);

 public static void report() throws Exception {

  //Sanity check
  assertArrayEquals(encodedCommons, encodedJava8);

What about the performance of these 3 ways? Base64 seems to be a small enough method so there are less ways to screw it up, but you'll never know what lies beneath the surface. From general timing (in the JUnit tests), it seems that the 3 methods can be arranged like this, from the fastest to the slowest: Java 8, Commons, Sun. A sample of the timing (encoding a byte array of size 5,000,000):

Sun: 521
Commons: 160
Java8: 37

Java 8's method ran 4x faster than Commons, and 14x faster than Sun. But this sample is just simplistic. Do try to benchmark for yourselves to come to your own conclusions.

So, which APIs to use? As any expert will tell depends. If you have enough power to dictate that your code should only run on Java 8 and above, then by all means use the new java.util.Base64. If you just need to support multiple JDK versions and vendors, you can stick with Commons Codec or some other 3rd party API. Or wait until the older Javas to be out of circulation or usage, and rewrite your precious codebase. Or move on to another programming language.

Note: I did not even mention about using sun.misc.BASE64Encoder. Avoid it when possible. Perhaps one day this class will be removed in another (alos) version of isn't present in other (heteros) JDKs by other vendors.


Thursday, December 05, 2013

Insidi-DoS: Internally negotiated, self-image-damaging DoS

What's more damning and ego-bruising than an external DoS (denial of service) attack? Why, an internally inspired one of course. Which self-respecting company would do such a thing to itself? What's the rationale to this irrational behaviour? Recently, the largest bank in Malaysia (let's call it Big Bank) chose to rollout a Metro-style UI 'improvement' on its well-known, critically-acclaimed Internet banking system. Mass hysteria ensued as public users were unable to perform even simple fund transfers as usual. As usual, the Head of IT's head was on the line. Big Bank promptly reverted to the previous UI and initiated damage-control measures, such as issuing an apology via Facebook.

There are numerous articles addressing the lessons to learn, root cause and best practices to avoid such incidents from happening again. This article instead will give tips to REPEAT such incidents in your company (if you are in a position of power), because people in general don't like to read boring stuff to avoid disasters in future.

As mentioned, some netizens blamed IT for causing this unfortunate incident. In a way, IT is part of the problem, but usually, they are just the receiving end of orders from above. The corporate attitude of IT in Malaysia is that they are just cooks; they just cook when given the rice. No rice, no cooking. Maybe no cook as well. So who's the real culprit? I would like to conject/suggest that the true culprit is actually a man (not being sexist, may be a woman, but this is my conjecture), who belongs to the business division, and an exuberant fan of new technology. So how does he concoct his strategy to rollout a new change and cause havoc to the Internet banking users of Big Bank?

This man with a mission may think this way:

1. Launch it quick
Launch it before the year ends. Why? Maybe people remember stuff better at year end. Great starts at the beginning of the year may be forgotten when the year is ending. Why not end the year with a bang? It doesn't matter if IT insisted that freeze period is around the corner. Launch it just before freeze period and the IT folks can also be given rest, right? Oh, but what about serious issues like...unable to login? Unable to transfer funds? Don't worry, users can still use phone banking, ATMs, even pay a visit to the branches. Alternative channels, dude! Not a serious concern, just launch it, NOW!

2. Launch it BIG
What good is a change if no one notices? The media may not cover it, the Facebook 'likes' may not even hit 4 digits, and no hoo-haa like 'Jom Heboh'. I know...just launch it during the MONTH-END! Payroll time is the best time. Even users that were forced to use Internet Banking will most likely need to login just to check if their salary is in. That way, you get maximum publicity and Big Bank can again bag yet another award for 'Most Innovative Design' for an Internet banking system.

3. No options aka I know what is good for you
Oh, why does Gmail and even Yahoo have to inform users that there's a new UI for their systems and pain-stakingly provide online cues and dialog boxes to guide the users? Come on, this UI change is awesome! Facebook did rollout a new UI and initially users were mad, but they then normalized and happily got back into broadcasting their personal data for free. Wait, Gmail, Yahoo and Facebook are free sites, but Big Bank is not. But Big Bank does not charge its users to use Internet Banking, right? We attempted to charge last decade but it backfired, so why should the users complain about the UI? It's GOOD for them! Good things must be shoved into their faces. Makes socialism such a fuzzy and 'noble' ideal... 

4. No need for stress testing, but when I do, I do it in PRODUCTION
Oh, stress testing stresses me up. The incessant whining of IT that our test environment is not up to par due to cost-saving measures to reduce IT expenditure really vexes me. What's wrong with saving cost? If I need to wait for stress testing to complete, I am not able to launch it quick, and big. Anyway, what's so performance-impacting than a UI change? It's like wearing a mask, right? You can still see, breathe, eat and hear when donning the mask. Oh, I love myself because I am so good at analogy.

Of course, in actual fact, it rarely boils down to a single person being the root cause to such incidents. Or maybe it really boils down to that. No one knows. I am not an insider in Big Bank. All I can say is, Bang2Bank.