Monday, September 28, 2020

BSF Genesis 2020/2021 - Bible References by Lesson

  1. Lesson 01 KJV | NIV | ESV
  2. Lesson 02 KJV | NIV | ESV
  3. Lesson 03 KJV | NIV | ESV
  4. Lesson 04 KJV | NIV | ESV
  5. Lesson 05 KJV | NIV | ESV
  6. Lesson 06 KJV | NIV | ESV
  7. Lesson 07 KJV | NIV | ESV
  8. Lesson 08 KJV | NIV | ESV
  9. Lesson 09 KJV | NIV | ESV
  10. Lesson 10 KJV | NIV | ESV
  11. Lesson 11 KJV | NIV | ESV
  12. Lesson 12 KJV | NIV | ESV
  13. Lesson 13 KJV | NIV | ESV
  14. Lesson 14 KJV | NIV | ESV
  15. Lesson 15 KJV | NIV | ESV
  16. Lesson 16 KJV | NIV | ESV
  17. Lesson 17 KJV | NIV | ESV
  18. Lesson 18 KJV | NIV | ESV
  19. Lesson 19 KJV | NIV | ESV
  20. Lesson 20 KJV | NIV | ESV
  21. Lesson 21 KJV | NIV | ESV
  22. Lesson 22 KJV | NIV | ESV
  23. Lesson 23 KJV | NIV | ESV
  24. Lesson 24 KJV | NIV | ESV
  25. Lesson 25 KJV | NIV | ESV
  26. Lesson 26 KJV | NIV | ESV
  27. Lesson 27 KJV | NIV | ESV
  28. Lesson 28 KJV | NIV | ESV
  29. Lesson 29 KJV | NIV | ESV
  30. Lesson 30 KJV | NIV | ESV

Sunday, June 09, 2019

wget on Windows

Overview

This is to document my steps to download all image (JPG) files along with PDF and regular HTML files instead of using the web browser, using only 1 command (wget).

Installation

Use Choco (https://chocolatey.org/). Follow installation instructions @ https://chocolatey.org/install

Then open a command prompt with administrative rights to install wget:
choco install wget

Usage

My target website (say abc.com) is protected by BASIC authentication. I am only interested in downloading files with extensions *.jpg, *.pdf & *.html. So I will create a directory to have the files placed i.e. c:\abc. Then, just run the commands below:
cd c:\abc 
wget --user-agent="Googlebot/2.1 (+https://www.googlebot.com/bot.html)" --http-user=user123 --http-password=coder4life -A "*.jpg,*.html,*.pdf" -r https://www.abc.com/folder123/ -l=0
where

--user-agent = User agent string to let the web server of target website to know about the kind of client/browser that is connecting. If not specified the value is "wget" which some web servers may block access

--http-user = BASIC username

--http-password = BASIC password (plain text)

-A = Inclusion list to download

-r = Tells wget to recursively get files (search the website for all possible paths/files)

-l = How "deep" should wget go. Default is 5, meaning from the URL https://www.abc.com/folder123/, wget can go until /folder123/1/2/3/4/5 and stop looking. The command above has value 0, which means "infinite" (until all possible paths are traversed)

Monday, April 24, 2017

North Korea: China or The USA's Opportunity?

Overview

North Korea's sabre-rattling tactics by launching missiles and conducting nuclear tests may be seen as an internal affair to boost confidence in Kim Jong-Un's regime, but with its neighbours being alarmed, and attracting the attention of the USA, the hermit kingdom is no longer insulated from world affairs. China, the emerging superpower, will be hard-pressed to act, while the USA, the current superpower, is also pondering on military action should things continue to escalate. Many see this geopolitical event as a problem. But this represent an opportunity for both sides.

Historical Background

The Korean peninsula has always been a buffer zone between its bigger neighbours; China to the west, Japan to the east, and Russia to the north. Chinese, Korean and Japanese cultures all share similarities, yet all have distinctives that separate them. In the early 20th century, China was so weakened by foreign intervention that Japan saw an opportunity to expand its territories westward. First, by annexing the Korean peninsula in 1910. Then, in 1931, Japan invaded and occupied Manchuria. China proper was then invaded in 1937.

The Chinese mainland territories captured by Japan were still controlled by them until 1945. After Europe was liberated by the Allies in May 1945, focus was shifted to Japan. USSR then agreed to enter the war against Japan. The Soviet Union invaded Japanese-occupied Manchuria in August 1945. At about the same time, the Americans dropped the atomic bombs on Hiroshima and Nagasaki. Japan then surrendered unconditionally.

The United Nations then developed plans for a trusteeship administration. The Soviet Union administered the peninsula north of the 38th parallel, while the United States administered the south. This resulted in the 1948 establishment of two separate governments, North Korea and South Korea. Both governments were molded in the image of their administrators.

In 1949, the Communist Party took control of China. The Soviet Union then welcomed China into its sphere of influence. Along with North Korea, the Soviets then have a buffer zone against the USA.

From this, after the Second World War, China needed to re-establish the Korean peninsula as a buffer zone against future invasions from the east. Even as the Japanese threat was eliminated by their defeat and surrender at the hands of the Allies, a new adversary suddenly emerged: The United States of America. Remember that the whole Korean peninsula was controlled by China. A southern Western-like state so close to the mainland gave the Chinese leaders restless nights.

When the Korean War broke out, China took a back seat at first, thinking that the North Korean Army, armed with Soviet weapons and trained by the Soviet army could do the job in unifying Korea. When the United Nations army was very close to capturing the entire peninsula, the Chinese Army suddenly mobilised and in a twist of events too long to discuss here, managed to get away with a ceasefire and retained the division of North & South Korea along the 38th parallel.

At that time, China's intervention was justified in the Communist world, having saved their smaller comrade from the jaws of defeat. A Korean peninsula half-controlled by another Communist regime is surely better than the entire peninsula controlled by the opposite camp. Chinese casualties were high but the fallen were celebrated as heroes for the great communist cause.

The Present Situation

North Korea's unbridled nuclear tests, along with missile launches from land and sea, is unnerving even for China, its greatest supporter. It is one thing to have this 'puppet' state occupy Western minds and shake the Japanese, but another if the provocations cause a military reaction to 'bring peace' to the region. It will be another Korean War, this time not of the communists' choosing. China now is not as dogmatic as China in the 1950s. This however does not eliminate the fact that she still needs a buffer zone.

Economics is a strong factor when going to war. The Japanese went all-out in the Second World War to capture resource-rich territories that they see as essential for their nation's benefit. Then, it was painful to have a lack of resources. Now, it is painful to expend resources in war. Both China and the US are seen as capable to intervene militarily, since both have similar interest to control the East Pacific regions.

Scenario A: The US Attacks, China Idles

As the incumbent superpower in the region, the US is still the protector for South Korea and Japan. A strike from North Korea to the 2 countries will surely pressure the US to respond. As the North Koreans have boasted on national TV that they are capable of challenging the US, the Trump administration can use this opportunity to eliminate this threat by military intervention.

A defeated North Korean regime can be a positive outcome, with which South Korea have the responsibility to rebuild its northern sibling. The US, through South Korea, can also offer assistance and this will spark demand for infrastructure and construction. The US can send a signal to the rest of the world of its status as a superpower & not to be messed with. It may also result in China being viewed as to have forsaken its neighbour for economic expediency, since ideology no longer is a strong factor in national decisions.

It will be a sign of weakness to the rest of the East Pacific nations, as China did not act to the threat so near its mainland, the likelihood of them really challenging Vietnam, the Philippines, Taiwan etc. over control of the nine-dash line (South China Sea region). This will boost the smaller nations' will to continue challenging China's claims of sovereignty.

Scenario B: The US Attacks, China Defends

This will surely spark internal outrage in China. Why send troops to defend North Korea in this time of economic uncertainty and at the cost of Chinese lifes just to maintain the Kim regime? It may boost the confidence of regional allies and 'neutral' nations, but internal backlash is something China could not afford.

If the outcome of the war is another stalemate, it will provoke further outrage as it will be compared to the first Korean War. This time, there's no strong ideological reason to justify shedding Chinese blood on foreign soil. Mao Zedong was the king in the party then, but Xi Jinping is no supreme commander now, though he is the leader.

Scenario C: The US Idles, China Attacks

This sounds preposterous, but in actual fact, is a possibility as history has shown. A region of instability will invite trouble from other nations that sees this as a 'low-hanging fruit'. That was how Japan annexed Korea in 1910. A weak China precipitated this. As the Kim regime is universally despised by the world, and internally-pressured to remain adored by its own people, a sudden invasion will give Kim Jong Un a heart attack and time to respond will be longer.

Militarily, this gives China the opportunity to gain experience by participating in combat within the same region. It can be a rehearsal for the invasion of Taiwan, though this requires dominance at sea. The Chinese army seen as liberating North Korea from the hegemony of the Kim regime will be a better morale-booster than defending the Kim regime.

China, emerging as the victor, will be able to strike a deal with South Korea in rebuilding the North. This will increase demand in the construction and infrastructure sector. This is surely a better way to use resources and manpower rather than building ghost cities in 'no man's land'. As happened in Germany, where the Western side had to pay for the Eastern side's debts and rebuilding costs, so will South Korea do the same for their Northern sibling.

This will be an eye-opener for South Korea as the US is no longer integral in its security and may look to China taking over the role. Perhaps there isn't a need for any nation to offer 'protection' if the Koreans view China as benevolent. But this may shake up Japan as China still has not forgotten World War II.

Scenario D: The US & China Attacks

This requires military coordination between both nations, which is a weak area since both nations regard each other with great suspicion, each trying to subvert rather than cooperate. A land invasion from China and an aerial assault by the US can be possible, as along as both militaries do not interfere with each other. Limited cooperation and coordination is still possible, as in the 2-side front in Europe against Nazi Germany by the Allies.

Conclusion

No doubt the US and China are challenging each other for influence in East Asia. In the Cold War era, choosing sides was easier, what more choosing a victor in World War II. China's emergence as a regional superpower is changing the status quo. Then, nations going to war with each other are not tightly coupled together economically/financially. When this coupling is severed, as happened between Japan and the US in the 1940s, war followed. Both the US and China are too interdependent with each other, but both also do not want to be seen as weak and losing ground. This complicates matters and given the circumstances, both nations will want to avoid going to war with each other.

If war is unavoidable, then Scenario C will favour China in the aftermath. China is one of the "Kings of the East". No power can take it lightly, even North Korea. I quipped to my South Korean friends of my next visit to "Korea" will be in Pyongyang. Perhaps this year will be the right year.

Tuesday, April 18, 2017

GoAccess: Visualize Your Web Traffic

If you want a visual representation of your web traffic, try GoAccess (https://goaccess.io/). You will never let it go.

I am just generating a report using static log files. GoAccess has many other features (real-time updates, console access, JSON/CSV-exporting etc.) that I haven't got the need to use (for now).

Tools Used:

- Mac OS 10.12
- GoAccess v1.2


Installation

brew install goaccess

Configuration

Retrieving logs from the web servers (Apache 2.4-based), sample output:
10.230.33.71 - - [26/Mar/2017:02:41:09 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 200 196 314
10.230.33.71 - - [26/Mar/2017:02:36:26 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 304 - 314
10.230.33.71 - - [25/Mar/2017:22:47:19 +0800] "GET /retail/mce/css/icons.data.svg.css HTTP/1.1" 304 - 314
10.230.33.71 - - [25/Mar/2017:17:46:35 +0800] "GET /retail/mce/css/icons.fallback.css HTTP/1.1" 304 - 314

Tip #1: To remove 'noise' if your web server is shared by multiple apps, use 'grep' to filter patterns that you are interested:
grep -i 'mypattern' access_log-20170328 > filtered_access_log-20170328

Tip #2: Although GoAccess has the ability to get input from multiple files, you can also combine all your log files via 'cat':
cat a.log b.log c.log > d.log

Log format settings from httpd.conf:
LogFormat "%h %l %u %t \"%r\" %>s %b %D" common

Note: The assumption was to take the Apache log format verbatim, since GoAccess supports the Apache log file format. After much trial and error, the log format for GoAccess is as below. The most frustrating issue was the response times getting rendered as 0.00us due to the inclusion of ">" in %>s. GoAccess only recognises %s.

Open file: /usr/local/Cellar/goaccess/1.2/etc/goaccess.conf, and set the following (adjust based on your own log patterns):
time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %l %u [%d:%t %^] "%m %U %H" %s %b %D

Save the file after editing, and run the following (assuming log filename is access_log):
goaccess access_log -o report.html -q

where
-o: Report file. For HTML report, specify a filename with *.html extension
-q: Ignore query string i.e., www.google.com/page.htm?query => www.google.com/page.htm

My output/report:



The man page (https://goaccess.io/man) has the full options available.

Side note: Why not AWStats (http://www.awstats.org/)? It was my 1st choice, but I was working on a Macbook, and the easiest route was GoAccess (just use brew to install). AWStats needed Perl and only had ready RPMs at its website's download page. Plus, GoAccess has a more impressive UI dashboard, and I needed to show some fancy stats to biz...

Thursday, June 02, 2016

Update Microsoft Windows if Windows Update seems to be 'stuck'

Sometimes old laptops/PCs have trouble updating via Windows Update after a new installation. Even after leaving the machine running, you will wake up in dismay knowing that you left the machine running without achieving anything.

Steps to do so:
  1. Stop Windows Update service (Run: services.msc)
  2. Go to: http://download.wsusoffline.net/
  3. Grab the latest download.
  4. Unzip, then there are 2 parts:
    1. Run UpdateGenerator.exe at folder root. Select appropriate Windows version (64-bit is x64). Then click Start button. Wait till it finishes downloading.
    2. Navigate to 'client' folder (e.g. wsusoffline\client). Run UpdateInstaller.exe. Then click Start button.
  5. Once done, there will be multiple restarts needed. After the dust is settled, can run Windows Update like normal.
Helpful link: http://superuser.com/questions/951960/windows-7-sp1-windows-update-stuck-checking-for-updates

Sunday, March 29, 2015

JCodec - Grabbing and saving picture frames from video

JCodec is an open source pure java implementation of video and audio codecs and formats (this statement was ripped from the website (http://jcodec.org/index.html). It's a nifty API, with a nice license (BSD), but sadly, as with open source software, the documentation can get a little bit stale.

The code listed on the website (as of Mar 29, 2015) to get a single frame works on v0.1.5, but as usual, being a person who wants to work with the latest and greatest piece of API, I got the v0.1.9 version from Maven (http://search.maven.org/#search%7Cga%7C1%7Cjcodec) and tried to run the code sample.

Alas, it didn't compile. After searching high and low for a remedy with the javadocs, and sidetracked to trying Apache Imaging (still in incubation), and took a look on OpenIMAJ (www.openimaj.org/), I still can't find a way to get a single frame with v0.1.9.

I finally got frustrated and just downloaded the v0.1.5 sources and see how it was implemented then. I realised that v0.1.9 removed the dependency on javax.imageio. That's why the example code didn't work. So I took some code out in v0.1.5 and reapplied to v0.1.9 just to test it out:


import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.File;

import javax.imageio.ImageIO;

import org.jcodec.api.FrameGrab;
import org.jcodec.common.FileChannelWrapper;
import org.jcodec.common.NIOUtils;
import org.jcodec.common.model.ColorSpace;
import org.jcodec.common.model.Picture;
import org.jcodec.scale.ColorUtil;
import org.jcodec.scale.Transform;
import org.junit.Test;

public class JCodecTests {

 @Test
 public void testSingleFrame_v019() throws Exception {
  int frameNumber = 150;
  Picture frame = FrameGrab.getNativeFrame(new File("D:/yk/mymovie.mp4"), frameNumber);

        Transform transform = ColorUtil.getTransform(frame.getColor(), ColorSpace.RGB);
        Picture rgb = Picture.create(frame.getWidth(), frame.getHeight(), ColorSpace.RGB);
        transform.transform(frame, rgb);
        BufferedImage bi = toBufferedImage(rgb);
        ImageIO.write(bi, "png", new File("frame_150.png"));
 }
}

It's a little bit 'exposed', but this would work, for now.

Lesson learnt: Go directly to the source! Long live open source!

Friday, March 28, 2014

Base64 in Java 8 - It's Not Too Late To Join In The Fun

Finally, Java 8 is out. Finally, there's a standard way to do Base64 encoding. For too long we have been relying on Apache Commons Codec (which is great anyway). Memory-conscious coders will desperately use sun.misc.BASE64Encoder and sun.misc.BASE64Decoder just to avoid adding extra JAR files in their programs, provided they are super sure of using only Sun/Oracle JDK. These classes are still lurking around in Java 8.

To try things out, I've furnished a JUnit test to show how to use the following APIs to encode:

  • Commons Codec: org.apache.commons.codec.binary.Base64
  • Java 8's new java.util.Base64
  • The sort-of evergreen internal code of Sun/Oracle's JDK: sun.misc.BASE64Encoder
package org.gizmo.util;

import java.util.Random;

import org.apache.commons.codec.binary.Base64;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import static org.junit.Assert.assertArrayEquals;

import sun.misc.BASE64Encoder;

public class Base64Tests {

 private static byte[] randomBinaryData = new byte[5000000];
 private static long durationCommons = 0;
 private static long durationJava8 = 0;
 private static long durationSun = 0;
 
 private static byte[] encodedCommons;
 private static byte[] encodedJava8;
 private static String encodedSun;
 
 @BeforeClass
 public static void setUp() throws Exception {
  
  //We want to test the APIs against the same data
  new Random().nextBytes(randomBinaryData);  
 }

 @Test
 public void testSunBase64Encode() throws Exception {
  
  BASE64Encoder encoder = new BASE64Encoder();

  long before = System.currentTimeMillis();

  encodedSun = encoder.encode(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationSun = after-before;
  System.out.println("Sun: " + durationSun);
 } 
 
 @Test
 public void testJava8Base64Encode() throws Exception {
  
  long before = System.currentTimeMillis();

  java.util.Base64.Encoder encoder = java.util.Base64.getEncoder();
  encodedJava8 = encoder.encode(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationJava8 = after-before;
  System.out.println("Java8: " + durationJava8);
 }
 
 @Test
 public void testCommonsBase64Encode() throws Exception {
  
  long before = System.currentTimeMillis();
  
  encodedCommons = Base64.encodeBase64(randomBinaryData);
  
  long after = System.currentTimeMillis();
  durationCommons = after-before;
  System.out.println("Commons: " + durationCommons);
 }

 @AfterClass
 public static void report() throws Exception {

  //Sanity check
  assertArrayEquals(encodedCommons, encodedJava8);
  System.out.println(durationCommons*1.0/durationJava8);
 }
}


What about the performance of these 3 ways? Base64 seems to be a small enough method so there are less ways to screw it up, but you'll never know what lies beneath the surface. From general timing (in the JUnit tests), it seems that the 3 methods can be arranged like this, from the fastest to the slowest: Java 8, Commons, Sun. A sample of the timing (encoding a byte array of size 5,000,000):

Sun: 521
Commons: 160
Java8: 37

Java 8's method ran 4x faster than Commons, and 14x faster than Sun. But this sample is just simplistic. Do try to benchmark for yourselves to come to your own conclusions.

So, which APIs to use? As any expert will tell you...it depends. If you have enough power to dictate that your code should only run on Java 8 and above, then by all means use the new java.util.Base64. If you just need to support multiple JDK versions and vendors, you can stick with Commons Codec or some other 3rd party API. Or wait until the older Javas to be out of circulation or usage, and rewrite your precious codebase. Or move on to another programming language.

Note: I did not even mention about using sun.misc.BASE64Encoder. Avoid it when possible. Perhaps one day this class will be removed in another (alos) version of JDK...it isn't present in other (heteros) JDKs by other vendors.

References: