Version: 5.1.0 - Updated on 21 April 2013
The report output can be in text (80 or 132 column output) or in HTML. The HTML report option is in table format and has additional functions over the ASCII report, such as bar chart summaries and specified highlighted entries.
You can process both standard logs and accounting logs.
It can be run on Unix/Linux or Windows servers and is designed to work with all versions of Firewall-1. It should work with any future releases unless there are major changes in logging.
The following log features are recognised:
If you find fwlogsum useful, please consider a donation to keep this project going.
The next example shows a similar report, but generated from an accounting log.
The source is available in the following formats:
If you are running Firewall-1 1.x or 2.x you can use an older version of fwlogsum that will work with those Firewall-1 versions. Note that this script has been frozen, so the new features in the current script will not be available.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
The other senario is where there is just one Firewall-1 host acting as the filtering host and management station. In this instance it is up to you whether you want to run perl and generate reports on your firewall. Generally it is better to transfer the Firewall-1 logs to an internal secure host and runs the reports from there.
After you have extracted the script, you need to make sure the perl path at the top of the script is set correctly. You will need to use perl 5.000 or higher.
Check the configuration paths for the Unix or NT environment (deping on which one you run). The Unix settings are set for a Solaris environment.
In order for the bar charts to display correctly, you will need to install the RGraph library from http://www.rgraph.net. The RGraph folder should be placed in the same location where the generated HTML reports exist.
If you want to use Geo IP lookups, you will need to install the Geo::IP Perl module and grab the latest GeoLite binary database from http://dev.maxmind.com/geoip/geolite
Change any of the following default settings in the configuration section to your personal preference.
$DNScachefile
Path to file to hold DNS cache.
$DNSexpire Expire DNS cache entries older than the specified number of days.
$Delimiter
If you have pre-existing logexport files or you generate logexport files for other uses that have a delimiter other than the default ;, set the delimiter you use here.
%Networks
This hash is used to map FW1 host and interface entries to a descriptive name. In a simple case of a FW1 gateway between the internet and private network, you could define something like:
%Networks = ( "FWHOST eth0" => "Internet Gateway",
"FWHOST eth1" => "Corporate Network"
);
In a more complicated FW1 environment with multiple FW1 hosts and interfaces, you can combine definitions. For example, if you had your corporate network behind multiple interfaces, you could define something like:
%Networks = ( "FWHOST eth0" => "Corporate Network",
"FWHOST eth1" => "Finance Network",
"FWHOST1 en0" => "DMZ",
"FWHOST1 en1" => "Outside world"
);
$Verbose
Setting this flag will display some processing messages about the number of lines processed.
$Interval
If verbose mode is set, the number of lines processed will be displayed at whatever interval you set this value to.
$TableBreak
If this variable is set to a value greater than 0, HTML tables will be broken into the specified number of lines. This has the advantage of being able to view the report as it is loading and prevents errors with browsers not being able to handle a single huge table.
$Type
This can be set to either: 'reject', 'drop', 'accept' or 'reject drop', depending on what you want the default report type to be.
$Header
This is an appropriate title header for the report type.
$MailHeader
This is the header used in the subject line if you mail reports.
$Sort
This is the sorting field, it can be set to: 'count', 'fwhost', 'rule', 'source', 'destination' or 'service'
$Top
This is set to the maximum number of entries you want in the summary reports.
$Format
This is set to either '80','132' or 'html' for the output format.
$ExcludeFWhost
You can choose to leave the FWHOST column from the reports if you run a single FW-1 host environment. This gives more space to the other columns when using the ASCII report output options.
$ResolveIP
If you have a slow naming service, you can get the script to resolve the IP addresses for you as it only needs to resolve addresses that appear in the report, which could potentially save a lot of time.
fw logexport command to output to a temporary file and generate the reports from that file. This saves multiple invocations of fw logexport which can take some time to generate the logs depending on their size.
-a --highlight Highlight specified lines -A --attackinfo Display attack info from SmartDefense -B --trenddir Trend data directory -bo --outbound Report only on outbound traffic -bi --inbound Report only on inbound traffic -c --width Column width. 80 or 132 chars -C --cachedns Cache DNS results -d --delimiter Delimiter for logexport fields (default: ;) -D --incdomsum Include domain summary in report -e --excludesvc Exclude specified service(s) from report -f --excludesrcsvc Exclude specified source services(s) from report -g --restrictcount Restrict entries with less than the specified count -G --geolookup Perform GEO IP Lookups -H --header Report header title -i --ignore Ignore specified entries (perl regexp) -l --logexport Read from specified logexport file (Standard or compressed) -L --fw1log Read from specified FW1 log file (Standard or compressed) -m --mail Mail report to specified user -n --excludeif Exclude specified FW interface/s from the report -o --output Output to specified file -p --incsrcport Include source port number in report -P --summaries Number of entries to appear in the summary (default: 10) -q --postresolveip Resolve IP addresses after filtering has been performed. -R --resolveip Resolve IP addresses (before filtering) -ra --rptaccepts Report only on accepted entries -rd --rptdrops Report only on dropped entries -rr --rptrejects Report only on rejected entries -rt --rptattacks Report only on attack entries -rx --rptdropsrejects Report only on dropped and rejected entries (Default) -S --summary Generate Summary only. -sa --sortattack Sort by attack type (only relevant for SmartDefense entries) -sc --sortcount Sort by count (default) -sd --sortdest Sort by destination address -sf --sortfw Sort by firewall host -sr --sortrule Sort by rule number -ss --sortsrc Sort by source address -sv --sortsvc Sort by service -t --includeonly Report only on specified entries (perl regexp) -T --time24 Display time summary as 24 hour clock -v --verbose Verbose mode -w --html Output in HTML -xb --xlateboth Report both normal address/port and translated address/port -xt --xlate Report just the translated address/port -y --svcname Convert port numbers to their name -Y --svcport Convert port names to their number
NOTE: You can use short flag options, long flag options or a combination of the two.
To report on all accepted packets, except http and gopher, and mail the results.
# fwlogsum -ra -e http,gopher -m fwreport@somewhere.com
To report on all accepted packets in 132 column mode, sorted by service
To read the data from an ASCII FW1 log produced with fw logexport
# fwlogsum -l fwlog.19981102
To report on dropped packets, sorted by destination and ignoring packets matching 'microsoft.com' and 'playboy.com'
# fwlogsum -rd -sd -i "microsoft\.com|playboy\.com"
To report all dropped and rejected packets in a series of compressed logexport files
# gzip -dc report.*.gz | fwlogsum -l - -rx
To report on both untranslated and translated adddresses/ports
# fwlogsum -xb
To report on accepted packets only matching telnet
To report on all dropped and rejected inbound traffic.
To produce a HTML output of the summary only for accepted packets.
To report on dropped and rejected packets including the source port, sorted by service, excluding smtp entries and displaying a 24 hour time summary.
# fwlogsum -w -rx -T -p -sv -e smtp
The following utilities are available with fwlogsum:
The utilities are included with the fwlogsum download, or can be downloaded separately:
Redesigned the HTML to be more modern. Bar graphs now use the excellent RGraph library.
Added the -G/--geolookup flag to perform country/city lookups based on IP address. This is done via the GeoLite database provided by www.maxmind.com
Cleaned up some of the redundant code.
Version 5.0.4 (16 Nov 2009)Added support for R70 export logs as the FW origin is logged in the origin_id field rather than orig field.
Processing speed improvements.
Version 5.0.3 (10 July 2007)
Fixed a bug that meant the excluded services summaries weren't being displayed correctly.
Version 5.0.2 (30 April 2004)
Fixed a bug that caused missing source/dst columns when using the -q flag.
Version 5.0.1 (6 April 2004)
Ignore blank lines otherwise it causes "unknown log entry" errors.
Version 5.0.0 (24 June 2003)
Removed the Perl formats. This will reduce the size of the HTML reports considerably as excessive whitespace is no longer needed.
Added the --attackinfo or -A flag to report on attack information from SmartDefense.
Added --rptattacks or -rt flag to report on only attack entries from SmartDefense.
Added --sortattack or -ra flag to sort by attack type (from SmartDefense).
Tweaked the stylesheet and various HTML bits.
Version 4.8.1 (1 March 2003)
Fixed a bug where the "Restrict entries not display" counter was incorrect.
Version 4.8.0 (30 January 2003)
Added the -q or --postresolveip flag to resolve IP addresses after filtering and matching. This can significantly speed up the report generation time.
Updated fwlogsum.cgi for the latest version and prettied the output.
Added new top level domains to the domains file.
Version 4.7.2 (6 December 2002)
Fixed the bug that I introduced with the bugfix in 4.7.1
Version 4.7.1 (5 December 2002)
Fixed a subtle but important bug that caused certain fields to contain the entry of the previous line if they had no value for that particular field.
Version 4.7.0 (29 November 2002)
Updated stylesheet so that the report can fit more on the screen.
Version 4.6.0 (15 March 2002)
Now handles FW1 alert entries by displaying them in bold/red font (HTML output only) and added an "alert type" summary.
There was some confusion with the -a flag called "highlight alert lines". This did have anything to do with the FW1 alert entries and was misleading. The word "alert" has been dropped from its description.
Now counts encrypted/decrypted entries.
Added a colour index in the HTML report.
(5 February 2002)
No code changes, but the licence has been changed to GPL.
Version 4.5.3 (23 January 2001)
Ignore non-log entries (eg: SecureRemote). This fixes a bug with the DNS cache not having an address to resolve.
Version 4.5.2 (10 January2001)
Cater for alert log entries.
Version 4.5.1 (20 December 2001)
Changed references from Mb to MB to correctly reflect Megabytes and not megabits.
Under some circumstances, icmp entries wouldn't be correctly formatted.
Version 4.5.0 (3 December 2001)
Added the -C or -cachedns flags to specify a DNS cache to be kept. This option will significantly increase the speed of the reports if you use fwlogsum to resolve IP addresses.
Version 4.4.2 (20 November 2001)
Fixed a bug where the match for the header line could be triggered by long URL's with a date/time parameter.
Version 4.4.1 (9 November 2001)
Fixed a bug where the --excludesvc was being ignored. Thanks to Paul Ford.
Version 4.4.0 (4 October 2001)
Added the -y -Y or -svcname -svcport flags to be able to convert port entries to names or vice versa.
The bar charts in the summaries are now proportional to the size of the cell.
Version 4.3.0 (26 July 2001)
Added the -n or --excludeif flags to specify certain FW interfaces to exclude. Thanks to Chuck Sterling
Version 4.2.0 (14 March 2001)
Finally added long options as I was running out of letters of the alphabet. Short options have been retained for backwards compatability.
Added -P or --summary flags to specify how many entries should appear in the summaries.
The report header date ranges are now split into date ranges for the log file and date ranges for the matched data. For example, you may have a log that runs from midnight to midnight, but the actual data in the report occurs between 5am-9am. The report will now display both date/time ranges.
Greatly simplified the HTML code by making use of style sheets.
Version 4.1.1 (8 February 2001)
Fixed the missing report generation date in the report header.
Version 4.1.0 (21 November 2000)
Added the ability to read from STDIN by specifying a - to the -l option. This allows you to use fwlogsum in a pipe:
gzip -dc log.* | fwlogsum -l - -o weekly_report.html
Version 4.0.2 (22 April 2000)
Fixed a bug that didn't process non default delimiters correctly.
Version 4.0.1 (11 April 2000)
The logexport delimiter is now quoted to prevent the shell from interpreting it. This bug is only relevant if fwlogsum runs the logexport command for you. Bug fix thanks to Graeme Miller
Verbose mode will now tell you what command or log file it is opening/running.
Version 4.0.0 (6 April 2000)
Accounting logs can now be processed with bandwidth output displayed in the summary section. A big thank you to Paul Nesfield for his input.
Option to store summary data in a dbm file so that long term trend reports can be generated. A simple script to retreive the top entries from the logs is provided.
Added the -H flag to specify a report header title.
Added the -o flag to write to an output flag.
Added Daily Usage to the summary section.
Every now and then it is possible to get log entries with no source, destination or service fields. For example, a switch doing a broadcast bootp request may have no source IP address. Previously these entries cuased fwlogsum to get the fields out of order. Empty fields now have a 'no-source', 'no-destination' or 'no-service' description.
Short log entries can now be processed. As short logs don't have rule numbers, the rule field is set to 'n/a'
Summary title headings are now more descriptive.
The bug with domain entries has really been fixed this time.
Multiple log files can now be concatinated without having to worry about whether the field order changes between the logs.
gen-fwlogsum has been renamed to fwlogsum.cgi
Version 3.9.0 (8 February 2000)
Fixed a bug that caused broadcast entries to be not displayed correctly and a bug that caused long domain names in the domains file to not be correctly processed.
Changed the handling of the logexport header line so that multiple logexport files can be concatinated without worrying about different field orders. (Oh how I wish Checkpoint could make logexport spit out a consistent field order).
Version 3.8.0 (4 February 2000)
Fixed a bug that caused gzipped logs to not be read correctly.
Added an option to ignore/include short log entries.
Version 3.7.0 (20 November 1999)
Fixed a bug that caused NAT (Valid Address) entries to not display correctly. Patch thanks to Martin Emmerich
Compressed logexport files are now uncompressed on the fly, saving on temporary space needed for uncompression.
Version 3.6.0 (21 August 1999)
Fixed/added the following items suggested by Martin Emmerich
Fixed bug in the check for Table Breaks if non HTML report is selected.
Fixed bug that caused the 24 Hour format summary to incorrectly display.
Added $FullSubjectLine switch to allow the mail subject to contain the report type.
Added content type when mailing HTML reports.
Updated gen-fwlogsum to recognise the -v flag.
Version 3.5.0 (3 June 1999)
Added -xb and -xt flags to display translated addresses/ports.
Added an option to break the HTML output into multiple tables to stop browsers barfing on a large single table.
The script will now only check for long logging on entries that match the report. This means if you are only interested in generating reports for dropped/rejected packets you coul turn short logging on for accepted packets.
Updated gen-fwlogsum to recognise the new flags and to fix a bug where it didn't recognise the -g flag.
Version 3.4.0 (23 Mar 1999)
Did anyway with the gateway interface definitions and replaced it with a network definition based on FW1 host and interface.
Improved the HTML layout.
Various neatening up in the code and comments.
The CGI script (gen-fwlogsum) is now included in the distribution.
Version 3.3.2 (23 Feb 1999)
Fixed bug that prevented multiple level domains from being recognised in the domain summary.
Version 3.3.1 (15 Jan 1999)
Fixed bug that incorrectly checks if long logging is enabled.
Version 3.3.0 (10 Jan 1999)
Added -D flag for domain summaries.
Verbose option now reports number of matched entries as well as lines processed.
The script now checks to see if long logging has been enabled. Any entries that have short logging will be rejected.
Added HTML anchors for each summary section, which can be utilitised by custom HTML table of contents.
Fixed bug with icmp entries displaying incorrectly in summary section.
Fixed bug when dealing with broadcast entries.
Fixed bug with time display with verbose option.
Version 3.2.0 (23 Dec 1998)
Made IP address resolution faster.
Verbose option now displays additional messages.
Some minor bug fixes.
Version 3.1.0 (19 Dec 1998)
Changed version number scheme to major release.minor release.bug fix release
Fixed various minor bugs to do with calculating gateway traffic.
Improved the HTML layout and added colour options for bar charts.
Added FW hosts summary. This can be useful for seeing the breakdown of matched data from each of the FW hosts in multi-FW1 environments.
Verbose option now reports number of lines being processed.
The following features/bugs were suggested/fixed by Lasse Holmqvist.
Added -R flag to have fwlogsum resolve IP addresses rather than during fw logexport. This can be useful if you have a slow name service as the script only needs to resolve addresses that appear in the report.
Added -x flag to exclude the FWHOST column in the report and leave more room for the other columns. Useful when using the 80 column ASCII report option.
Added -T flag to display time period summary as 24 hour summary.
Version 3.00b (6 Dec 1998)
Changed the method of processing logs from fw log output to fw logexport format. This means that the fields can come out in any order with any additional fields added and the script will pick up the relevant fields (assuming field names don't change). This has slowed the report generation down by quite a degree, but at least it is much more flexible.
Several flags were changed to be a little more consistant.
Tidied up HTML code.
The -a flag was added to allow highlighing of particular entries of interest in the HTML report output.
Options were added to set table colours and font attributes.
Version 2.16
Made compatible with NT perl.
Added extra comments in the configuration section.
Removed the -z compressed flag. Made -l and -L flags cater for compressed files as well.
Added the -g flag to restrict the size of reports by not displaying any entries with a count less than the specified number.
Changed the mail code to be more secure and robust.
Version 2.15
Added -L flag to read specified FW1 log files. Not to be confused with the -l flag to read ASCII log files.
Better error checking.
Code neatening and comments.
Clarified some of the descriptions.
Added the "Hourly summary" to follow any time based events. Eg: is a cronjob responsible for all the blocked snmp entries between 2am-3am
Added option to include custom HTML header/footer.
Fixed a problem with white space in the time.
Version 1.0 (May 1996) Original dates are lost in the mists of time, but the script is based on a script written by Cameron Humphries.
Please send any comments, bugs or suggestions to peter@ginini.com