GoAccess: Visualize Your Web Traffic

If you want a visual representation of your web traffic, try GoAccess (https://goaccess.io/). You will never let it go.

I am just generating a report using static log files. GoAccess has many other features (real-time updates, console access, JSON/CSV-exporting etc.) that I haven't got the need to use (for now).

Tools Used:

- Mac OS 10.12
- GoAccess v1.2

Installation

brew install goaccess

Configuration

Retrieving logs from the web servers (Apache 2.4-based), sample output:
10.230.33.71 - - [26/Mar/2017:02:41:09 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 200 196 314
10.230.33.71 - - [26/Mar/2017:02:36:26 +0800] "GET /retail/mce/img/icons/icon-close.png HTTP/1.1" 304 - 314
10.230.33.71 - - [25/Mar/2017:22:47:19 +0800] "GET /retail/mce/css/icons.data.svg.css HTTP/1.1" 304 - 314
10.230.33.71 - - [25/Mar/2017:17:46:35 +0800] "GET /retail/mce/css/icons.fallback.css HTTP/1.1" 304 - 314

Tip #1: To remove 'noise' if your web server is shared by multiple apps, use 'grep' to filter patterns that you are interested:
grep -i 'mypattern' access_log-20170328 > filtered_access_log-20170328

Tip #2: Although GoAccess has the ability to get input from multiple files, you can also combine all your log files via 'cat':
cat a.log b.log c.log > d.log

Log format settings from httpd.conf:
LogFormat "%h %l %u %t \"%r\" %>s %b %D" common

Note: The assumption was to take the Apache log format verbatim, since GoAccess supports the Apache log file format. After much trial and error, the log format for GoAccess is as below. The most frustrating issue was the response times getting rendered as 0.00us due to the inclusion of ">" in %>s. GoAccess only recognises %s.

Open file: /usr/local/Cellar/goaccess/1.2/etc/goaccess.conf, and set the following (adjust based on your own log patterns):
time-format %H:%M:%S
date-format %d/%b/%Y
log-format %h %l %u [%d:%t %^] "%m %U %H" %s %b %D

Save the file after editing, and run the following (assuming log filename is access_log):
goaccess access_log -o report.html -q

where
-o: Report file. For HTML report, specify a filename with *.html extension
-q: Ignore query string i.e., www.google.com/page.htm?query => www.google.com/page.htm

My output/report:

The man page (https://goaccess.io/man) has the full options available.

Side note: Why not AWStats (http://www.awstats.org/)? It was my 1st choice, but I was working on a Macbook, and the easiest route was GoAccess (just use brew to install). AWStats needed Perl and only had ready RPMs at its website's download page. Plus, GoAccess has a more impressive UI dashboard, and I needed to show some fancy stats to biz...

Running Apache/IHS on privileged ports (80, 443) as a non-root user: The 'sudo' way

Overview

For some out-of-this-world reason, your Unix administrator fears giving you the root access. But your web server (Apache or IBM HTTP Server) needs to bind to a privileged port e.g. 80, 443, and root privilege is needed. Using 'sudo', you can at least not bother your administrator to start/stop/restart anymore. He would either be happy that no one's calling to bug him anymore, or fear that someone like you has found a way to do something without his 'power'.

Steps below are for IBM HTTP Server (IHS) 8.5 running on AIX. Applicable for other versions & OSes with slight variations.

Assumptions

• IHS location: /opt/IBM/HTTPServer
• Using default httpd.conf @ /opt/IBM/HTTPServer/conf/httpd.conf
• HTTP port to listen/bind: 80
• User & group to delegate after startup: wasadmin:wasadmin
• Logs folder & files: /opt/IBM/HTTPServer/logs - access_log and error_log
• You have root access (for the time being)

Steps

1. 'Touch' the log files:

touch /opt/IBM/HTTPServer/logs/access_log
touch /opt/IBM/HTTPServer/logs/error_log

2. Make the following changes in httpd.conf (ensure user 'wasadmin' and group 'wasadmin' has been created already)

User wasadmin
Group wasadmin

3. Change ownership of IHS files:

chown -R wasadmin:wasadmin /opt/IBM/HTTPServer

4. Edit sudoers file (visudo), add the following line:

%wasadmin ALL = (root) NOPASSWD: /opt/IBM/HTTPServer/bin/apachectl *

This means any user in the 'wasadmin' group (prefixed with %) can control (start/stop) the IHS instance as root without needing to enter password. If you only want to allow the user 'wasadmin' to perform this, then remove the '%' to denote a user.

5. Ok, let's start IHS as user 'wasadmin'.

From root prompt:
sudo su - wasadmin

To check current user: id

Sample output:
uid=12(wasadmin) gid=203(wasadmin) groups=0(system)

Run as root without password:
sudo -u root /opt/IBM/HTTPServer/bin/apachectl start

To verify, fire up web browser and access the default webpage e.g. http://192.168.1.128 (port 80 is implied if not specified for HTTP).

Note: If you moved/deleted the log files, you need to touch them again using the non-root user as in Step #1 (e.g. wasadmin). Else the log files will be created with root ownership.