FlowViewer - Custom Graphs and Analysis

[Ref: OpenBSD 4.8 amd64, 4.9 amd64 & i386]

Now we have some analysis tools going, we have a basic workflow for pro-active investigation of what's happening on our network:-

  • Smokeping shows a spike in traffic latency on your link.
  • CUFlow assists in isolating various parts of the traffc including:
    • specific time period,
    • Networks and Servers (Network, Subnet)
    • Protocol and Ports (Services)
  • Now what ?

Enter, FlowViewer (for those who don't want to investigate at the command-line, flow-tools) and related tools FlowTracker, FlowGrapher.

FlowViewer gives you a GUI front-end for interrogate the database for quick analysis of the data. For our above scenario, you can quick drill through to see which hosts were generating the highest traffic at that time and to/from whom.

After the previous netflow analysis tools (flow-tools, flowscan, CUFlow) installing FlowViewer is rather simplified.

  • Install the Pre-requisites
  • Install Flowviewer
  • Configure Flowviewer

Pre-Requisites

Refer: Michael W. Lucas' book "Network Flow Analysis." which already identifies minimal prerequisites to get our FlowViewer up and running.

Binary tools I install from OpenBSD Packages, but perl modules may require installing from CPAN. PAN.

If you got here, without installing and testing the FlowScan/CUFlow, things should work but interpretation of data will really need more support from the above book.

The Package Binaries

  • RRDtool (rrdtool from packages, includes dependencies libiconv, jpeg)
  • gd 2.0.35 (gd from packages )
  • gd::graph (p5-GD-Graph from packages, includes dependencies p5-GD-TextUtil)
  • GDBM_File (p5-GDBM_File from packages, includes dependencies gdbm)

The Package Binaries

The binaries should be installable from the Ports/Packages. Check the website for what services require the binaries, whether you need them, but they are relatively easy to install so while we're learning, just install everything.

In summary, we need the gd related tools for graphing (e.g. FlowGrapher) and the RRD tool for Tracking, charting with tracking data.

Install

Ref: FlowViewer

As per the installation instructions ('cause I know you were reading it.)

  1. 'Un-tar' FlowViewer_3.X.tar in an appropriate directory such as your web server's cgi-bin directory
  2. Modify the contents of FlowViewer_Configuration.pm for your environment.
  3. If you are going to use FlowGrapher, you will need to install both the GD and the GD::Graph packages for Perl.
  4. If you are going to use FlowTracker, you will need to install the RRDtool package (at least version 1.2.12.) Create FlowTracker_Filters and FlowTracker_RRDtool subdirectories.
  5. Point your web browser at either of FlowViewer.cgi, FlowGrapher.cgi, or FlowTracker.cgi and go ...

Installation Tips

  1. If you do not embed your device name in the name of the directory that stores the raw flow data (e.g., you have only one device) set the @devices array to empty (i.e., @devices = "";) With version 3.3, you can now collect all devices to one directory, and sort by Exporter.
  2. Make sure that the FlowViewer 'reports', 'graphs', 'tracker', 'work', 'names', and 'log' (if you're logging) directories that you specify have adequate permissions (e.g., 0777) for the web server to write into them.
  3. Double check that you have configured the proper HTTP protocol (i.e., either HTTP (usually port 80), or HTTPS (usually port 443)).
  4. For FlowTracker make sure you install an RRDtool version that is 1.2.12 or later.

Get the Source

Get the current release from the FlowViewer homesite (http://ensight.eos.nasa.gov/FlowViewer/) and follow the basic install instructions that comes with the software.

curl -O http://ensight.eos.nasa.gov/FlowViewer/FlowViewer_3.4.tar

1. Copy to Install

Normally, /var/www/cgi-bin/FlowViewer, for our example below I'm using and base installation directory of:

PathDescription
/var/www/flowviewer Base Directory
/var/www/flowviewer/cgi CGI Scripts
/var/www/flowviewer/docs Documents
/var/www/flowviewer/saves SAVE Directory
/var/www/var/flowviewer Scratch/Work Directory

Using the above directory path construction, let's make all the appropriate directories:

basedir=/var/www/flowviewer
cgidir=$basedir/cgi
docs=$basedir/docs
workdir=/var/www/var/flowviewer

for d in saves docs cgi; \
    do mkdir -p $basedir/$d/; chmod -R a=rwx $basedir/$d/; done
for d in FlowGrapher FlowTracker FlowWorking; \
    do mkdir -p $docs/$d/; chmod -R a=rwx $docs/$d/; done
for d in names filters rrdtool exporter log; \
    do mkdir -p $workdir/$d/; chmod -R a=rwx $workdir/$d/; done

Untar the files and copy it into the CGI Scripts directory:

  • $cgidir (or) /var/www/flowviewer/cgi

Note: Using a 'personalised' path structure is 'nice' but it also means the real documentation doesn't always align well with this documentation. That's good, don't use this documentation as a cut-and-paste instruction set.

2. Modify FlowViewer_Configuration.pm

Make the changes appropriate for getting your site up and running, with your data etc.

One option of modifying the standard configuration, is to add your changes at the bottom of the configuration file, differentiating your changes.

File: FlowViewer_Configuration.pm

$FlowViewer_server       = "10.0.0.2";     # (IP address or hostname)

$trackings_title         = "My Company Title";
$user_logo               = "My.Company.Logo.jpg";  # For a nice look make your logo 86 pixels high
$user_hyperlink          = "http://www.example.com/";

@devices                  = ("sensorXY");
#@exporters               = ("sensorXY_ipaddress:sensorXY Title");

Set the Basic configuration of Service Title and what sensors are being reviewed.

For my configuration file, I've created 3 x new variables:

  • viewdocs
  • viewdocs_url
  • work

These are obviously relevant to this example, and simplify modifications.

File: FlowViewer_Configuration.pm

$basedir                 = "/var/www/flowviewer";
$baseurl                 = "/FlowViewer";
$workdir                 = "/var/www/var/flowviewer";
$viewdocs                = "$basedir/docs";
$viewdocs_url            = "$baseurl/docs/";
$cgi_bin_directory       = "$basedir/cgi";
$cgi_bin_short           = "$baseurl/cgi";

$reports_directory       = "$viewdocs";
$reports_short           = "$viewdocs_url";
$graphs_directory        = "$viewdocs/FlowGrapher";
$graphs_short            = "$viewdocs_url/FlowGrapher";
$tracker_directory       = "$viewdocs/FlowTracker";
$tracker_short           = "$viewdocs_url/FlowTracker";
$work_directory          = "$viewdocs/FlowWorking";
$work_short              = "$viewdocs_url/FlowWorking";
$save_directory          = "$basedir/saves";
$save_short              = "$baseurl/saves";
$names_directory         = "$workdir/names";
$filter_directory        = "$workdir/filters";
$rrdtool_directory       = "$workdir/rrdtool";

The above configuration specifies the local path where files are to be stored (X_directory), and the Web URL (X_short) matched to that directory.

The second configuration update is to set the location for the data.

File: FlowViewer_Configuration.pm

$flow_data_directory     = "/var/netflow";
$exporter_directory      = "/var/www/var/flowviewer/exporter";
$flow_bin_directory      = "/usr/local/bin";
$rrdtool_bin_directory   = "/usr/local/bin";

$log_directory           = "/var/www/var/flowviewer/log";

Make sure that the data and binary folders are correct for your configuration. The binaries on a regular install (using ports/packages) for OpenBSD should be as in the above.

During the initial startup of the CGI scripts, they will attempt to create the relevant reporting paths.

Copy your images to the new "working-directories"

cp $basedir/cgi/FlowViewer/*.png $docs
cp My.Company.Logo.jpg $docs

Graph Generation

To use FlowTracker and FlowGrapher, we start the 'perl' daemons to collate the data.

SCRIPTDIR=/var/www/flowviewer/cgi
LOGDIR=/var/www/var/flowviewer/log

PERL5LIB=${SCRIPTDIR} ${SCRIPTDIR}/FlowTracker_Collector \
    >> ${LOGDIR}/FlowTracker_Collector.log 2>&1 &
PERL5LIB=${SCRIPTDIR} ${SCRIPTDIR}/FlowTracker_Grapher \
    >> ${LOGDIR}/FlowTracker_Grapher.log 2>&1  &

Validate the command-lines work (common problems I've encountered are not having the right paths)

Your log file should have something like the below:

At 07/29/2015 09:05:52 started next collection.  Period: July 29, 2015 8:30:00 to July 29, 2015 8:35:00
0 trackings had a zero value. 2 trackings had a positive value.
At 07/29/2015 09:05:57 finished this loop. Update period: 1438133700  2 Trackings. Loop took: 5 seconds

With the above logs verified, you can add the command-lines to your /etc/rc.local to ensure it is started with each machine restart.

Apache

Customising your configuration, is basically choosing what paths you want to show on your browser. The classic example is a straight out alias using the Capitalisation of the application name FlowViewer.

File extract: /var/www/conf/httpd.conf

Alias /FlowViewer/cgi/ /var/www/flowviewer/cgi/
Alias /FlowViewer/docs/ /var/www/flowviewer/docs/
Alias /FlowViewer/saves/ /var/www/flowviewer/saves/

<Directory /var/www/flowviewer/cgi/>
    Options ExecCGI
    AllowOverride None
    Order allow,deny
    Allow from [list.of.ips.you.trust]
</Directory>
<Directory /var/www/flowviewer/docs/>
    Options Index
    AllowOverride None
    Order allow,deny
    Allow from [list.of.ips.you.trust]
</Directory>
<Directory /var/www/flowviewer/saves/>
    Options Index
    AllowOverride None
    Order allow,deny
    Allow from [list.of.ips.you.trust]
</Directory>

As noted in the above, working files are in the path /var/www/flowviewer/

And now, we can point our browser to the above site to start looking at reports:

http://collector-ip-address/FlowViewer/cgi/FlowViewer.cgi

Fortunately, that single online URL can connect you to other parts of the toolkit.

Chroot

Remember that OpenBSD's default Apache instance runs in the chroot(2) environment. To use FlowViewer, disable this environment using the "-u" option.

-u      By default httpd will chroot(2) to the ``ServerRoot'' path.  The
        -u option disables this behaviour, and returns httpd to the
        expanded "unsecure" behaviour.

If you're really paranoid, you could configure all the relevant tools to work within a chroot'd environment.

/var/www/var

Some directories may not have been, or permissions not useable.

Review the /var/www/log/error_log file

Workflow Sample

A simple, active monitoring day's activities could revolve such as the scenario below:

  • Isolate hosts/traffic in question using FlowViewer
  • View historic chart of host/traffic behaviour using FlowGrapher
  • Track continued behaviour of that host/traffic pattern using FlowTracker

FlowViewer

FlowViewer is good for getting snapshot views of traffic behaviour, whether the snapshot is 5 minutes or 10 days. It is a good way of isolating the hosts, patterns of heavy use such that we can isolate further points of investigation.

FlowGrapher

To view historical data on the identified traffic pattern, we can use FlowGrapher to generate a chart to give us a better view of previous behaviour.

FlowTracker

Love using this feature to justify what modifications may or may not be required on the infrastructure.

Relative Links

Other links that may be relevant, or give better clues.