+------------------------------------------------------------------------------------+ | PIX Logging Architecture v2.00 Beta 1: Installation, Configuration and Usage Guide | +-------------------------------+-------------------------+--------------------------+ | kris@logging-architecture.net | Wed, Nov 01, 2006 | version #2.00/1 | +-------------------------------+-------------------------+--------------------------+ +--------------------------------------------------+ | Table of Contents | +--------------------------------------------------+ | 1. Introduction | | 2. Features | | 3. Deployment Options | | 3.1 PLA Infrastructure Layout | | 3.2 PLA Centralized Deployment | | 3.3 PLA Distributed Deployment | | 3.4 PLA Traffic Requirements | | 4. PLA Installation | | 4.1 Installation Requirements | | 4.2 Installation Matrix | | 4.3 Installing Parsing Scripts | | 4.4 Installing Web-based Frontend | | 5. PLA Configuration | | 5.1 Configuring MySQL Database | | 5.2 Configuring Syslog | | 5.3 Configuring PIX Firewall | | 5.4 Configuring Apache | | 5.5 Configuring PLA Parsing Daemon | | 5.6 Configuring PLA Web-based Frontend | | 6. Using PIX Logging Architecture v2.00 | | 6.1 Basic Usage | | 6.2 Overview of PLA Menus | | 6.3 Log Viewing | | 6.4 Log Searching | | 6.5 Configuring Display Filtering | | 6.6 Configuring Traffic Descriptions | | 6.7 (a) Configuring Traffic Queries | | 6.7 (b) Running Traffic Queries | | 6.8 Configuring Parse Filtering | | 6.9 Executing Log Purges | | 7. PIX Logging Architecture Best Practices | | 7.1 Scaling the Deployment | | 7.2 Recommended Deployment Strategy | | 7.3 Parsing Fun | | 8. Additional Information | | 8.1 The People Behind PLA | | 8.2 Acknowledgements and Kudos | | 8.3 Support The PLA Project | | 8.4 PLA Resources | +--------------------------------------------------+ ==================================================== ==================================================== 1. INTRODUCTION The PIX Logging Architecture [PLA] provides a centralized method for correlating and displaying logs generated by Cisco PIX Firewalls. Cisco PIX supports logging to a syslog host. PLA uses this feature in order to parse these logs, based on their type (traffic, ids or informational) and then push them into a MySQL database. A web-based frontend queriesthe MySQL database for the entries and then displays them according to the various options available in the PLA web-based frontend. PIX Logging Architecture v2.00 is the second release of PLA, being released about 3 years after the original release, many changes and improvements have been made to the original code and many new features have been introduced in order to make PLA more suitable for different infrastructures. For those who were familiar with PLA v1.01, you'll get a whole realm of new features with this release whilst still maintaining the same look and feel. However, I've come along way with this project since 2003 so you'll find that some things have been changed since the original PLA v1.01 release. Amongst the major changes you'll find the following: * Log Parsing is done using a Perl-written daemon (no longer need to stop/start syslogd) * Log Message Parsing criteria are no longer defined in parsing scripts but stored within a table in the PLA database called "syslog_message" * Informational messages (i.e. PPPoE establishment, SSH/PDM logins, etc..) can now also be logged. * PLA Database has been modified. More fields are now logged (i.e. xlate[translations]) * PLA web based front end has seen the "statistics" section removed and replaced with a configuration section which allows you to set up traffic filters, traffic descriptions, user-defined search queries, parse filtering, database log purging and others. All these features are described in the features section and surely you'll explore and find out as you start using PIX Logging Architecture. 2. FEATURES The main purpose of PIX Logging Architecture is to provide a framework for treating, correlating and consulting logs of different devices. I've done successful PLA testing with the following components: * Cisco PIX OS 6.x * Cisco (FWSM) FireWall Services Module 3.1(3) However, PLA should also work correctly with Cisco FWSM 2.x and PIX OS 7.x versions. Parsing/Database Highlights: * Parsing of Cisco FWSM and PIX OS logs from the syslog files into the PLA database. * System Log message criteria used for parsing are stored in a table in the PLA database. * Parse filters can be defined using the web-based front end for traffic which should not be logged in the database. * Database entries can be purged based on certain criteria using the log purging option in the web-based front end. Web-based Front-end Highlights: * Web based front-end allows viewing of Traffic, IDS and Informational Logs. * Display details on Traffic, IDS and Informational logs and allow cross-searching in the database for similar logs. * Whois and DNS lookup of logged traffic. * PLA database can be searched based on various criteria for Traffic, IDS and informational logs. * User-defined queries can be created to define saved search criteria. * Traffic Display Filters can be created to filter out pre-defined traffic while displaying logs. * Traffic Descriptions can be created to identify traffic based on certain values. * Parse filters can be created to prevent certain traffic from being logged into the PLA database. * Log purging ensures that the database doesn't get too big. * On-the-fly Queries, Display Filters, Parse Filters and Traffic descriptions can be created from a log entry. 3. DEPLOYMENT OPTIONS 3.1 PLA INFRASTRUCTURE LAYOUT As previously mentioned, the PLA Infrastructure parses logs sent by a Cisco PIX Firewall, and then pushes them to a MySQL database. The parsing is done on the system defined as the syslog host for the Cisco PIX Firewall, the parsing script then uses the Perl DBI module to enter the logging information into the MySQL database. Two deployment infrastructures can be considered: - Centralized Deployment: all PLA components are located on the same system. - Distributed Deployment: various PLA components are located on different systems. 3.2 PLA CENTRALIZED DEPLOYMENT When using the PLA Centralized Deployment option, all PLA components are located on the same systems. This deployment could be used for various reasons, such as shortage of machines, centralized management policy. The following components are centralized on a single component: - syslog (with remote log reception activated) - MySQL database (with PLA database installed) - Apache Web Server - PLA Parsing Script - PLA Web-based Frontend The PLA Centralized Deployment architecture looks as follows: +---+ syslog +------------------------+ |PIX| ----------> | PLA Centralized System | +---+ +------------------------+ | (1) Parse Syslog | | (2) Populate Database | | (3) Provide Web-based | | frontend | +------------------------+ 3.3 PLA DISTRIBUTED DEPLOYMENT When using the PLA Distributed Deployment option, each PLA component or a group of PLA components are located on various systems. The PLA Distributed Deployment architecture looks as follows: +---+ syslog +------------------------+ +------------------------+ |PIX| ----------> | PLA Distributed System | | PLA Distributed System | +---+ | Logging Host | | DB + Web Frontend | +------------------------+ Perl::DBI +------------------------+ | (1) Parse Syslog | -----------> | (2) Populate Database | +------------------------+ | (3) Provide Web-based | | frontend | +------------------------+ Similarly, the database component can be seperated from the web-based front-end so that each component runs on a dedicated device. Such configurations may be optimal if you want to optimize the device resource utilisation. Moreover, certain security considerations may opt for such a distributed approach ensuring that DMZ devices are seperated from Internal systems and traffic only moves in a certain direction. 3.4 PLA TRAFFIC REQUIREMENTS As for traffic requirements, depending on your deployment strategy, certain openings may need to be made to your firewall rules to allow traffic between the various PLA components. The following traffic patterns exist: Status Source Destination Protocol/Service Description (MANDATORY) PIX Firewall PLA Logging Host Syslog (UDP/514) PIX Syslog to the log parser (MANDATORY) PLA Logging Host PLA Database Host Mysqld (TCP/3389) PIX log entries into database (MANDATORY) PLA Web FrontEnd PLA Database Host Mysqld (TCP/3389) PIX log queries from database (OPTIONAL) PLA Web FrontEnd DNS Server DNS (UDP/53) DNS Name Lookups (OPTIONAL) PLA Web FrontEnd ANY Whois (TCP/43) Whois Lookups (MANDATORY) PLA clients (browser) PLA Web Frontend HTTP(TCP/80) PLA Web-based FrontEnd Access HTTPS (TCP/443) 4. PLA INSTALLATION 4.1 INSTALLATION REQUIREMENTS The PLA Infrastructure requires certain components to be installed in order to function properly. The following components must be installed: * Syslogd * Perl (version >= 5.6.0) --> Perl is used to run the scripts associated with PLA. * Perl Modules --> Perl::DBI (version >= 1.20) --> Perl::CGI (version >= 1.19) --> DBD::Mysql (version >= 3.0008) --> Net::Whois::IP (version >= 0.50) --> Date::Manip (version >= 5.44) --> File::Tail (version >= 0.99.3) --> Socket (This should be standard in Perl) --> POSIX (This should be standard in Perl) * MySQL Database Server (version >= 3.23.42) * Apache Web Server I have not listed the exact installation steps for Perl, Perl Modules, MySQL Database Server and Apache. Information relative to these topics can be found at the following locations: * For Perl installations: http://www.cpan.org/misc/cpan-faq.html#How_install_Perl_source * For Perl Module installations: http://www.cpan.org/misc/cpan-faq.html#How_install_Perl_modules * For MySQL Database Server installations: http://dev.mysql.com/doc/ * For Apache Web Server Installations: http://httpd.apache.org/docs/ The PLA scripts are written in Perl, and should therefore work on any system supporting Perl. I'm personnally running this infrastructure on Linux, and I've had no problems so far. If you run this on any other platforms, i.e. ActivePerl on Windows or so, please send me an email (kris@logging-architecture.net) so I can add your platform to the compatibility list. 4.2 INSTALLATION MATRIX If you're opting for the distributed installation, it's important to know which components need to be installed where: Logging Host: * Syslogd * Perl * Perl Modules --> Perl::DBI --> DBD::Mysql --> File::Tail --> Socket (This should be standard in Perl) --> POSIX (This should be standard in Perl) Database Host: * MySQL Database Server Web Front-End Host: * Apache Web Server * Perl * Perl Modules --> Perl::DBI --> DBD::Mysql --> Perl::CGI --> Net::Whois::IP --> Date::Manip 4.3 INSTALLING PARSING SCRIPTS The parsing script is located in the "scripts/parsing/" directory of the PLA Software Package (the "tar.gz" file). This script must be installed on the system which received the system logs (syslog) and is used to identify which log entries should be pushed to the database, this script requires both Perl and Perl::DBI. Personally, I've put this script in an "/usr/sbin" directory. The parsing script is called "pla_parsed" and is a Perl script which runs in daemon mode so you just need to start it and it keeps running in the background. I've also included the rc.pla_parsed script which needs to be put into the /etc/init.d to automatically start the PLA Parse Daemon (pla_parsed) on system startup. If you move the pla_parsed script to another directory than "/usr/sbin" the "rc.pla_parsed" script will need to be updated. The script will need to be linked to the correct runlevel for it to started and stopped automatically. # ln -s /etc/init.d/rc.pla_parsed /etc/rc3.d/S99pla_parsed This script also requires a specific syslog configuration, which is shown in the Configuration section of this document. The "pla_parsed" script works as follows: 1. Connect to the Database and extract the syslog_message table (for identifying parsing criteria) 2. Connect to the Database and extract the parse_filter table (for identifying parsing filters) 3. Start reading the syslog messages file to which the PIX/FWSM logs are logged. 4. Match incoming message with syslog_message table extract and determine if a match exists 5. If a match exists, check whether the traffic matches a parse filter and if not, log the data to the PLA database. 6. If no match exists, determine whether to log the data to a flat temporary undefined traffic file. 4.3 INSTALLING WEB-BASED FRONTEND The web-based frontend is a collection of Perl/CGI scripts located in the "scripts/frontend/" directory of the PLA software package. These scripts query the MySQL database for information and entries pertaining to the Cisco PIX/FWSM logs. These scripts should be copied to a scripts executable directory on your web server. For example the "/cgi-bin/" directory on many web servers. In the configuration section of this document I've explained how to configure a directory to be script executable within Apache. For example, on my web server I've created a "pla2" directory in the main Apache HTML documents directory (DirectoryRoot - "/usr/local/apache2/htdocs") and set this to scripts executable. 5. CONFIGURATION 5.1 CONFIGURING MYSQL Since MySQL (the database) is the principal component empowering the PLA infrastructure, it needs to be configured properly. First of all, the PIX logging database needs to be created. A template for this database is located in the "scripts/" directory of the PLA Package and is called pix_database.sql. This file can easily be imported into the MySQL database as follows: root@mysql-host# cd /scripts root@mysql-host# mysql < pla_database.sql For a list of tables and information of the PIX Logging Architecture database layout, please refer to the PLA website. Secondly, a user must be created that is allowed full access only to the "pix" database. This user will be used both by the PLA Parsing script to push data into the database, and by the PLA Web-based Frontend to retrieve information from the database. The user can be created in the following manner: root@mysql-host# mysql -u root mysql mysql> GRANT ALL ON pix.* TO ''@'' IDENTIFIED BY ''; Remember to write down the user and password, you'll need to use these as variables in the PLA scripts. 5.2 CONFIGURING SYSLOG The syslog host will receive the alerts sent by the Cisco PIX/FWSM Firewall(s). First of all, the syslog daemon will need to be configured to listen for syslog messages from other hosts, in our case the Cisco PIX Firewall(s) on port 514/udp, this can be done by starting the syslogd in the following manner: "syslogd -m 0 -r" -> the "-r" option specified remote listening. Secondly, Syslog will need to be configured to redirect system log messages received from the Cisco PIX Firewall(s) to a separate file (I'm using "/var/log/.log"). In order to do this you'll have to reconfigure /etc/syslog.conf by adding the following line: "local6.* /var/log/.log" The "local6.*" facility is used here because I chose "logging facility 22" on the Cisco PIX Firewall, therefore this may need to be changed if you're using a different logging facility. The "/var/log/.log" file may also be different in your configuration, but do note that you will need to define this file as a variable($pix_log_file) in the PLA parsing script (pla_parsed). Note: Don't forget to restart syslogd after you've made these changes! 5.3 CONFIGURING CISCO PIX FIREWALL In order to get a Cisco PIX Firewall to be compatible with the PIX Logging Architecture, several configuration statements will have to be added on the Cisco PIX Firewall: A. Configuring System Logging pix(config)# logging on pix(config)# logging timestamp pix(config)# logging trap debugging pix(config)# logging facility pix(config)# logging host inside -> the "logging facility" defines where systems messages will be logged on the system host, I've set it to "22", which corresponds to logging facility local6 on a UNIX Syslog System. For more information about setting up PIX System Logging and Logging Facilities, refer to the following URL: http://www.cisco.com/en/US/products/hw/vpndevc/ps2030/products_tech_note09186a0080094030.shtml -> the "logging host inside" statement defines where to send the syslog messages, this should be the IP address of the system running the PLA Parsing Daemon (pla_parsed). -> if there are certain messages which you don't want to log, you can exclude them using the "no logging message " command. B. Configuring IDS Logging pix(config)# ip audit name info info action alarm pix(config)# ip audit name attack attack action alarm drop reset pix(config)# ip audit interface outside info pix(config)# ip audit interface outside attack pix(config)# ip audit info action alarm pix(config)# ip audit attack action drop -> this configuration enables the PIX to match IDS signatures and perform a drop action on them. In the above configuration, the interface is named "outside"; In your infrastructure, this interface may have another name. Note: It may be a good idea to synchronize the time on the Cisco PIX/FWSM using NTP. Note: Verify that your Cisco PIX/FWSM is correctly logging to your syslog server by checking the syslog's server's log messages file (i.e. "tail -f /var/log/.log" considering you've used the same nomenclature as the one described in the syslog configuration of this document). 5.2 CONFIGURING APACHE The Apache web server is used to house the scripts which power the PLA web-based front end. Whether you run Apache 1.x, 2.0.x or 2.2.x is your choice, however you may have to make some changes to the configuration in order to allow the scripts to correctly execute (in other words, the Apache needs to know that these scripts are executable and should thus show you the output of the scripts rather than the scripts code itself). If you're not very accustomed to the Apache web server, I suggest you install the PLA web-based frontend in the main Apache HTML documents directory which is usually located at the "/htdocs" directory. So you'll have to ensure all the contents of the PLA web based frontend "scripts/frontend/" (including the "images" subdirectory) is copied over to the Apache htdocs part. For the purpose of example we will create a "pla2" directory in the Apache default HTML documents directory (which we assume is located at "/usr/local/apache2/htdocs"): pla_webserver_host# mkdir -p /usr/local/apache2/htdocs/pla2 pla_webserver_host# cp -R pla_v2.00b1/scripts/frontend/* /usr/local/apache2/htdocs/pla2 Next, we'll configure Apache as follows: - /usr/local/apache2/htdocs/pla2 (EXECUTABLE DIRECTORY) - /usr/local/apache2/htdocs/pla2/images (NON-EXECUTABLE DIRECTORY) The following lines will need to be added to the "httpd.conf" file following the section. i.e. .. .. Options ExecCGI SetHandler cgi-script Options MultiViews -ExecCGI SetHandler default-handler 5.4 CONFIGURING PLA PARSING DAEMON The PLA Parsing Daemon (pla_parsed) script, located on the syslog host, will need to be configured with several parameters before PLA can work correctly. A. Configure Required Variables Several variables will need to be configured in the "pla_parsed" (PLA Parsing Daemon - /usr/sbin/pla_parsed) script: ## ## PLA REQUIRED Variables ## $mysql_db_host = ""; # Host running MySQL Database $mysql_db_port = ""; # Port on which MySQL is running (usually: 3306) $mysql_db_user = ""; # Username you configured in the MySQL database $mysql_db_pass = ""; # Password you configured in the MySQL database $mysql_db_name = ""; # MySQL PIX Logging Architecture Database Name (default: pix) $pix_log_file = " variable by the name of your context ofcourse!): ### ### DEFINE YOUR SYSLOG LAYOUT HERE! ### $regex_log_begin = "(.*):(.*) (.*) (.*) (.*) (.*) ((.*):(.*):(.*)) "; $var_pixhost=3; $var_pixmonth=4; $var_pixdate=5; $var_pixyear=6; $var_pixtime=7; $var_pixcontext=8; The following is what an FWSM 3.x log entry looks like: Sep 05 09:00:03 fwsm-pla-01 Sep 05 2006 09:01:30 PLACONTEXT : %FWSM-4-106100: access-list inbound_access permitted tcp external_if/1.1.1.1(22366) -> dmz_01/2.2.2.2(443) hit-cnt 1 (first hit) [0x91wasc80, 0x49dfsffdf] So the $regex_log_begin for the FWSM 3.x logs would look like: Sep 05 09:00:03 fwsm-pla-01 Sep 05 2006 09:01:30 PLACONTEXT C. Configure Optional Variables Several optional variables can be configured in the "pla_parsed" (PLA Parsing Daemon - /usr/sbin/pla_parsed) script: ## ## PLA Optional Variables ## $pla_suppress_unknown = "0"; # Defines whether to suppress unknown (not matching those in DB) Log Messages [ 0 - Do not suppress | 1 - Suppress ] $pla_unknown_writetofile = "1"; # Only applies when setting $pla_suppress_unkwown to "0". Determines whether to write the output to a file $pla_unknown_filename = "/var/log/pla_unknown_msgs.log"; # Sets the filename to write the unknown messages to. $pla_unknown_writetodb = "0"; # Only applies when setting $pla_suppress_unkwown to "0". Determines whether to write the output to the db (will be sent to the info_log table) D. Configure Automatic Startup The PLA Parse Daemon (pla_parsed) is a daemonized script thus it runs in the background. It can be started automatically at Operating System startup using the runlevel (rc) script "rc.pla_parsed". You just need to ensure that the pla_parsed script is started when the MySQL database server has already been started to ensure the PLA Parse Daemon can retrieve data from and write data to the PLA database. You'll have to link the startup script (i.e. /etc/init.d/rc.pla_parsed) to a runlevel and set it to be executable as follows: pix_parse_host# ln -s /etc/init.d/rc.pla_parsed /etc/rc3.d/S99pla_parsed pix_parse_host# chmod +x /etc/init.d/rc.pla_parsed 5.5 CONFIGURING WEB-BASED FRONTEND First of all, make sure the scripts in the "scripts/frontend/" directory of the PLA Package are copied to a directory configured to execute Perl/CGI scripts. Secondly, several variables need to be configured to allow the frontend to connect to the MySQL database. These variables can be found in the "conf.pl", which is also located within this directory: $mysql_db_host = ""; # Host running MySQL DB $mysql_db_port = ""; # Port running MySQL DB $mysql_db_user = ""; # MySQL DB Username $mysql_db_pass = ""; # MySQL DB Password 6. USING PIX LOGGING ARCHICTURE v2.00 6.1 BASIC USAGE The PIX Logging Architecture web based front-end can be reached by typing in the URL as defined on the web server with the "pix_traffic_logs" extension. So consider the application is installed on the 127.0.0.1 IP address in the pla2 directory then the browser needs to point to: http://127.0.0.1/pla2/pix_traffic_logs Using this URL, PIX Logging Architecture will open the main traffic logs page which display a resume of the PIX traffic logged events. 6.2 OVERVIEW OF PLA MENUS PIX Logging Architecture consists of a hierachical structure of menus allowing for easy navigation, use and configuration of the PLA web-based front-end. The following structure is used for the web-based front-end: +--+ PIX Traffic Logs (main menu) +--- PIX Traffic Log Details +--+ PIX IDS Logs (main menu) +--- PIX IDS Log Details +--- PIX Informational Logs (main menu) +--+ PIX Search Logs (main menu) +--- PIX Search Traffic Logs (sub menu) +--- PIX Search IDS Logs (sub menu) +--- PIX Search Informational Logs (sub menu) +--- PIX Traffic Queries (main menu) +--+ PLA Configuration (main menu) +--+ PIX Traffic Log Configuration (sub menu) +--- Display Filter Configuration (sub menu choice) +--- Traffic Description Configuration (sub menu choice) +--- Traffic/User-defined Query Configuration (sub menu choice) +--+ PIX IDS Logs Configuration (sub menu) +--- Display Filter Configuration (not active yet) +--+ PLA Global Configuration (sub menu) +--- Objects Configuration (not active yet) +--- Log Message Parsing Configuration (not active yet) +--- Incident Management (sub menu choice) +--- Log Purging Configuration (sub menu choice) +--- Parse Filter Configuration (sub menu choice) +--- PLA About (main menu) 6.3 LOG VIEWING The main functionality of PIX Logging Architecture is to allow logs to be parsed, correlated and available for viewing. PLA currently distinguishes between 3 types of logs; PIX Traffic Logs, PIX IDS Logs and PIX Informational Logs. Each of these logs has a dedicated log viewing page which allows the logged data to be consulted. A) PIX Traffic Logs Menu Item: (Main Menu) > "Traffic Logs" The PIX Traffic Logs view is used to display the PIX Traffic Logs. The general view shows the following columns in a list format: * Time * Log Resource (Firewall) * Log Action * Log Protocol * Source IP * Source Port * Destination IP * Destination Port * Log Flags On this page it is possible to select a few criteria including the date, log resource and whether the displayed information should contain the plain IP information or whether the addresses need to be resolved. Please note that traffic which has been filtered using a traffic display filter will not be shown in the "Traffic Logs" windows. In order to get more information about a certain log entry, click on the table row to display the "PIX Traffic Log Detail" page, which contains more specific information such as Xlate (Translation) information, Database Matches, and several Options. These options allow you to create filters, queries and descriptions on the fly. More information can be found later on in the traffic filters section of this document. B) PIX IDS Logs Menu Item: (Main Menu) > "IDS Logs" The PIX IDS Logs view is used to display the PIX IDS Logs. The general view shows the following columns in a list format: * Time * Log Resource (Firewall) * Log Protocol * Source IP * Destination IP * IDS Signature On this page it is possible to select a few criteria including the date, log resource and whether the displayed information should contain the plain IP information or whether the addresses need to be resolved. In order to get more information about a certain log entry, click on the table row to display the "PIX IDS Log Detail" page, which contains more specific information such as Database Matches, and several Options. At the time of this writing the only options available for the PIX IDS Log Detail is to link the Log ID to an incident. C) PIX Informational Logs Menu Item: (Main Menu) > "Informational Logs" The PIX Informational Logs view is used to display the PIX Informational Logs. Informational Logs include items such as SSH logins, config modifications, interfaces ups/downs and the like. The general view shows the following columns in a list format: * Time * Log Resource (Firewall) * Log Message * Log Description 6.4 LOG SEARCHING Besides showing a general view of the logs, it is also possible to carry out specific and granular searches against the database for entries which match the search criteria. PIX Logging Architecture v2.00 allows searching of PIX Traffic, IDS and Informational logs. A) PIX Traffic Logs Search Menu Item: (Main Menu) > "Search" (Sub Menu) > "Traffic Logs" PIX Traffic Logs can be searched based on a set of defined criteria. Within these searches a number of wildcards can be used with the "%" (percentage) character, which is the default SQL wildcard. The following lists all the fields which are allowed as search input: * Logging Resource * Traffic Action * Log Message * Source IP [wildcard allowed!] * Source Port [wildcard allowed!] * Destination IP [wildcard allowed!] * Destination Port [wilcard allowed!] * Log Protocol * Time Criteria: - Today - Specific Time Range - All Dates Please note that when you want to use an "any" wildcard, please leave the field blank (i.e. you don't need to fill in the "%" to match the Any condition). Some examples to clarify these searches and the usage of wildcards: +--+ You want to search all traffic from source 1.1.1.1 to destination 2.2.2.2 on port 80. +--- Source_IP=1.1.1.1 Source_Port= Destination_IP=2.2.2.2 Destination_Port=80 +--+ You want to search all traffic from source 1.1.1.x to destination 2.2.2.2 on port 80. +--- Source_IP=1.1.1.% Source_Port= Destination_IP=2.2.2.2 Destination_Port=80 +--+ You want to search all traffic from any to destination 2.2.2.2 on port 80. +--- Source_IP= Source_Port= Destination_IP=2.2.2.2 Destination_Port=80 B) PIX IDS Logs Search Menu Item: (Main Menu) > "Search" (Sub Menu) > "IDS Logs" PIX IDS Logs can be searched based on a set of defined criteria. Within these searches a number of wildcards can be used with the "%" (percentage) character, which is the default SQL wildcard. The following lists all the fields which are allowed as search input: * Logging Resource * Log Signature * Source IP [wildcard allowed!] * Destination IP [wildcard allowed!] * Log Protocol * Time Criteria: - Today - Specific Time Range - All Dates Please note that when you want to use an "any" wildcard, please leave the field blank (i.e. you don't need to fill in the "%" to match the Any condition). C) PIX Informational Logs Search Menu Item: (Main Menu) > "Search" (Sub Menu) > "Info Logs" PIX Informational Logs can be searched based on a set of defined criteria. Within these searches a number of wildcards can be used with the "%" (percentage) character, which is the default SQL wildcard. The following lists all the fields which are allowed as search input: * Logging Resource * Log Message * Informational Message Relation to a User-defined String [wildcard allowed!] - equals - does not equal - contains - does not contain - starts with - does not start with - ends with - does not end with * Time Criteria: - Today - Specific Time Range - All Dates 6.5 CONFIGURING DISPLAY FILTERING Sometimes you may have legal requirements from the governing industry bodies to log certain types of traffic, however you may not want them to show up when you're displaying your traffic logs because they're considered overhead traffic. Examples of such traffic may be outbound DNS, SMTP, HTTP requests which may "pollute" your log files since they are usually overall present in high numbers. The way a traffic filter works is by defining the criteria for traffic which should be filtered, for example * all outbound UDP DNS requests from your DNS server (10.10.10.10) should be filtered from the view Source_IP: 10.10.10.10 Source_Port: Destination_IP: Destination_Port:53 Protocol:UDP * all outbound UDP DNS requests from your DNS server (10.10.10.10) to a range of DNS servers (192.168.1.X) should be filtered from the view Source_IP: 10.10.10.10 Source_Port: Destination_IP:192.168.1.% Destination_Port:53 Protocol:UDP Ofcoure you don't have to worry about writing these filters in the text based form like I just did here, you'll have the web interface to configure them, but I'm just showing you how the evaluation of the criteria works and how you can use wildcards (either by leaving the fields blank for an "Any" match or by using the "%" wildcard character for partial matches). There's three ways of creating a traffic display filter: A) Creating a Traffic Filter Manually Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Display Filters" This page represents the main menu for the Traffic Display Filter configuration. It will list all your traffic filters give you a little menu on the top when you click on the plus-sign in front of the "Traffic Filter Options" link on the top. Using this menu you can create a new filter, search for certain variables within existing filters and show filters based on certain criteria. These latter two I'll let you find out later since they're quite trivial to use, so I'll focus on the Creation part. Creating a Traffic Display Filter manually involves several steps: 1) On the Traffic Display Filter configuration page, expand the "Traffic Filter Options" and click on the "Create" button 2) You'll now get a page with all blank form fields, some of which are input boxes, others are drop down menus. The following lists all the required form criteria and whether wildcards can be used: - Filter Name [Required] (Short description of the filter) (input text box) - Logging Resource (Allows you to select which firewalls logs the display filter should be active for) (choose one) + All Firewalls + - Logging Action (Allows you to select which traffic action the display filter should be active for) (choose one) + Any + Accept + Drop - Protocol (Allows you to select which protocol the display filter should be active for) (choose one) + TCP + UDP + ICMP - Source IP (Source IP for which the display filter should be active) (wildcards allowed!) (input text box) - Source Port (Source Port for which the display filter should be active) (wildcards allowed!) (input text box) - Destination IP (Destination IP for which the display filter should be active) (wildcards allowed!) (input text box) - Destination Port (Destination Port for which the display filter should be active) (wildcards allowed!) (input text box) - Filter Description (Longer description of the filter) (input text box) Click on the "Create" button to create the filter or on the "Reset" button to reset the form 3) Once you've created the filter you'll get a page with the summary of the filter you've just created. You've got three options from this page: - Clone This Filter (I'll come back to this later) - Edit This Filter (Allows you to modify the filter you just created) - Go Back to the Display Filter configuration page. 4) When you click on the "Back to the Display Filter" configuration page, you should see the list of all created traffic filters including the one you just created. When you create a filter it is enabled by default, which means that any traffic which matches the filter will not be displayed in the traffic logs view. However, it is possible to disable the filter by clicking on the "disable" button in the options section of the traffic display filter main page. Other options include "clone" (which will be discussed shortly), edit (which allows you to edit the specific filter) and remove (which will remove the traffic display filter). You can view the filter details by clicking on the filter name column (first column starting from the left). B) Creating a Traffic Filter With Cloning Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Display Filters" (Options Choice) > "[ clone ]" (within a selected traffic display filter in the "options" column) The purpose of Traffic Filter Cloning is to re-use criteria entered within another filter and cloning them in order to create a new filter. So basically if we have a filter "A" which has DNS traffic defined between a specific source and destination IP pair and you want to add another destination for that same source IP and same traffic (DNS), you can clone filter "A" to create filter "B". Using the process of cloning you don't have to create an entirely new filter because you can just copy over the criteria from another filter. Basically what a clone does is it gives you the same page as you would have for creating a new filter, however several of the parameters are already filled in according to a pre-defined template, being the traffic display filter you're cloning from. In addition, a clone will add the "CLONED:" keywords in front of the Filter Name and Filter Description fields so that you don't create filters with duplicate names. Once you've got the cloned filter create window open, you can edit any variables you want in order to create the new filter, in the same manner as you would edit an existing filter. Note: The same wildcards apply as for the "Creating a New Traffic Filter" section. Since creating a cloned traffic display filter involves mainly similar steps to creating a new filter, I will not repeat the entire process, however it is important to know that you can clone a traffic filter at three locations: * General Traffic Display Filter Configuration Page -> Under the "Options" column, click on "[ clone ]" for the desired row matching the traffic display filter you want to clone. * Specific Traffic Display Filter Viewing Page -> When you're at to the General Traffic Display Filter Configuration page, you can click on the Filter Name to open the specific details for the traffic display filter. At the Options tab on the bottom of the specific traffic display filter page you can select the "Clone Traffic Filter" option. * Specific Traffic Display Filter Creation Page -> When you've created a display filter you will get a summary with the details of the filter which has just been created. At the Options tab on the bottom of the specific traffic display filter page you can select the "Clone Traffic Filter" option. C) Creating a Traffic Filter From PIX Log Entry Menu Item: (Main Menu) > "Traffic Logs" (Select) > Select a Traffic Log which has been logged by the firewall. (Expand) > "Options" (Select) > "Display Filter" - "Create a display filter from PIX Log ID." While it is useful to be able to create Traffic Display Filters manually or cloning them. You're most likely creating the traffic display filters because of certain log entries you've seen. Therefore I've added the option to create a Traffic Display Filter straight from a PIX Traffic Log entry. This saves a lot of time since you don't have to worry about copy/pasting information across and it works in a similar way as cloning a Traffic Display filter in that it takes the information from the specific PIX Log Entry and allows you to edit it as you see fit. Note: The same wildcards apply as for the "Creating a New Traffic Filter" section. 6.6 CONFIGURING TRAFFIC DESCRIPTIONS Oftentimes you may come across traffic which may at first sight be unknown, especially some of the TCP/UDP high ports which are often used for Trojans as well as several legitimate applications (especially those very useful backup applications and the likes;). However, whilst traffic on port 80 and port 25 are well known services, an incoming scan on UDP/17500 may not be for example. That's why I've added the Traffic Descriptions functionality to PIX Logging Architecture, which is a feature that allows a user to define a description for a certain type of traffic. When the user clicks on the traffic details for this unknown traffic, each time it matches a traffic description, the description will be shown. Traffic Descriptions are accumulative, which means you can define multiple traffic descriptions for the same type of traffic and PIX Logging Architecture will then show you all of them. Traffic Descriptions are configured in a similar manner as Traffic Display Filters. There's three ways of creating a traffic description: A) Creating a Traffic Description Manually Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Descriptions" This page represents the main menu for the Traffic Description configuration. It will list all your traffic descriptions and give you a little menu on the top when you click on the plus-sign in front of the "Description Options" link on the top. Using this menu you can create a new description and search for certain variables within existing descriptions. Creating a Traffic Description manually involves several steps: 1) On the Traffic Description configuration page, expand the "Description Options" and click on the "Create" button 2) You'll now get a page with all blank form fields, some of which are input boxes, others are drop down menus. The following lists all the required form criteria and whether wildcards can be used: - Description Name [Required] (Short name of the description) (input text box) - Logging Resource (Allows you to select which firewalls logs the description should be active for) (choose one) + All Firewalls + - Logging Action (Allows you to select which traffic action the description should be active for) (choose one) + Any + Accept + Drop - Protocol (Allows you to select which protocol the description should be active for) (choose one) + TCP + UDP + ICMP - Traffic Type / Priority(Allows you to select what kind of traffic this is - just used for reference) (choose one) + Normal / Low + Normal / Medium + Normal / High + Anomalous / Low + Anomalous / Medium + Anomalous / High - Source IP (Source IP for which the description should be active) (wildcards allowed!) (input text box) - Source Port (Source Port for which the description should be active) (wildcards allowed!) (input text box) - Destination IP (Destination IP for which the description should be active) (wildcards allowed!) (input text box) - Destination Port (Destination Port for which the description should be active) (wildcards allowed!) (input text box) - Description Detail (Detailed information of the traffic description) (input text box) Click on the "Create" button to create the traffic description or on the "Reset" button to reset the form 3) Once you've created the description you'll get a page with the summary of the description you've just created. You've got three options from this page: - Clone Traffic Description (I'll come back to this later) - Edit Traffic Description (Allows you to modify the description you just created) - Go Back to the Traffic Description configuration page. 4) When you click on the "Back to the Traffic Description" configuration page, you should see the list of all created traffic descriptions including the one you just created. Options on this page include "clone" (which will be discussed shortly), edit (which allows you to edit the specific description) and remove (which will remove the traffic description). You can view the traffic description details by clicking on the description name column (first column starting from the left). B) Creating a Traffic Description With Cloning Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Descriptions" (Options Choice) > "[ clone ]" (within a selected traffic description in the "options" column) The purpose of Traffic Description Cloning is to re-use criteria entered within another description and cloning them in order to create a new description. So basically if we have a description "A" which has DNS traffic defined between a specific source and destination IP pair and you want to add another destination for that same source IP and same traffic (DNS), you can clone description "A" to create description "B". Using the process of cloning you don't have to create an entirely new description because you can just copy over the criteria from another description. Basically what a clone does is it gives you the same page as you would have for creating a new description, however several of the parameters are already filled in according to a pre-defined template, being the traffic description you're cloning from. In addition, a clone will add the "CLONED:" keywords in front of the Description Name and Description Detail fields so that you don't create descriptions with duplicate names. Once you've got the cloned description create window open, you can edit any variables you want in order to create the new description, in the same manner as you would edit an existing description. Note: The same wildcards apply as for the "Creating a New Traffic Description" section. Since creating a cloned traffic description involves mainly similar steps to creating a new description, I will not repeat the entire process, however it is important to know that you can clone a traffic description at three locations: * General Traffic Description Configuration Page -> Under the "Options" column, click on "[ clone ]" for the desired row matching the traffic description you want to clone. * Specific Traffic Description Viewing Page -> When you're at to the General Traffic Descriptions Configuration page, you can click on the Description Name to open the specific details for the traffic description. At the Options tab on the bottom of the specific traffic description page you can select the "Clone Traffic Description" option. * Specific Traffic Description Creation Page -> When you've created a traffic description you will get a summary with the details of the description which has just been created. At the Options tab on the bottom of the specific traffic descriptions page you can select the "Clone Traffic Description" option. C) Creating a Traffic Descriptions From PIX Log Entry Menu Item: (Main Menu) > "Traffic Logs" (Select) > Select a Traffic Log which has been logged by the firewall. (Expand) > "Options" (Select) > "Description" - "Create a description from PIX Log ID." While it is useful to be able to create Traffic Descriptions manually or cloning them. You're most likely creating the traffic descriptions because of certain log entries you've seen. Therefore I've added the option to create a Traffic Descriptions straight from a PIX Traffic Log entry. This saves a lot of time since you don't have to worry about copy/pasting information across and it works in a similar way as cloning a Traffic Description in that it takes the information from the specific PIX Log Entry and allows you to edit it as you see fit. Note: The same wildcards apply as for the "Creating a New Traffic Description" section. 6.7 (a) CONFIGURING TRAFFIC QUERIES Sometimes there's recurring traffic which you look for on a daily basis and you would like to run recurring searching for this type of traffic. While running a traffic logs search is not a bad idea, it requires you to input the parameters every time. Therefore I've created a Traffic Query, which is basically a "Set of Saved Search Parameters" which can then be given a name and ran at a later time. The advantage of using queries is that you do not need to re-type the parameters all ovah again with each recurring seach you do for similar traffic. Traffic Queries are configured in a similar manner as Traffic Display Filters. There's three ways of creating a traffic query: A) Creating a Traffic Query Manually Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Queries" This page represents the main menu for the Traffic Query configuration. It will list all your traffic queries and give you a little menu on the top when you click on the plus-sign in front of the "Query Options" link on the top. Using this menu you can create a new query, and search for certain variables within existing queries. Creating a Traffic Query manually involves several steps: 1) On the Traffic Query configuration page, expand the "Query Options" and click on the "Create" button 2) You'll now get a page with all blank form fields, some of which are input boxes, others are drop down menus. The following lists all the required form criteria and whether wildcards can be used: - Query Name [Required] (Short description of the Query) (input text box) - Logging Resource (Allows you to select which firewalls match the query) (choose one) + All Firewalls + - Logging Action (Allows you to select which traffic matches the query) (choose one) + Any + Accept + Drop - Protocol (Allows you to select which protocol matches the query) (choose one) + Any + TCP + UDP + ICMP - Log Message (Allow you to select which log message matches the query) (choose one) + Any + (i.e. PIX-3-313100,..) - Source IP (Source IP which matches the query) (wildcards allowed!) (input text box) - Source Port (Source Port which matches the query) (wildcards allowed!) (input text box) - Destination IP (Destination IP which matches the query) (wildcards allowed!) (input text box) - Destination Port (Destination Port which matches the query) (wildcards allowed!) (input text box) - Query Detail (Detailed information of the traffic query) (input text box) Click on the "Create" button to create the traffic query or on the "Reset" button to reset the form 3) Once you've created the query you'll get a page with the summary of the query you've just created. You've got three options from this page: - Clone Traffic Query (I'll come back to this later) - Edit Traffic Query (Allows you to modify the query you just created) - Go Back to the Traffic Query configuration page. 4) When you click on the "Back to the Traffic Query" configuration page, you should see the list of all created traffic queries including the one you just created. Options on this page include "clone" (which will be discussed shortly), edit (which allows you to edit the specific query) and remove (which will remove the traffic query). You can view the traffic query details by clicking on the query name column (first column starting from the left). B) Creating a Traffic Query With Cloning Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Traffic Logs" (Sub Menu Choice) > "Configure Query" (Options Choice) > "[ clone ]" (within a selected traffic query in the "options" column) The purpose of Traffic Query Cloning is to re-use criteria entered within another query and cloning them in order to create a new query. So basically if we have a query "A" which has DNS traffic defined between a specific source and destination IP pair and you want to add another destination for that same source IP and same traffic (DNS), you can clone query "A" to create query "B". Using the process of cloning you don't have to create an entirely new query because you can just copy over the criteria from another query. Basically what a clone does is it gives you the same page as you would have for creating a new query, however several of the parameters are already filled in according to a pre-defined template, being the traffic query you're cloning from. In addition, a clone will add the "CLONED:" keywords in front of the Query Name and Query Detail fields so that you don't create query with duplicate names. Once you've got the cloned query create window open, you can edit any variables you want in order to create the new query, in the same manner as you would edit an existing query. Note: The same wildcards apply as for the "Creating a New Traffic Query" section. Since creating a cloned traffic query involves mainly similar steps to creating a new query, I will not repeat the entire process, however it is important to know that you can clone a traffic query at three locations: * General Traffic Query Configuration Page -> Under the "Options" column, click on "[ clone ]" for the desired row matching the traffic query you want to clone. * Specific Traffic Query Viewing Page -> When you're at to the General Traffic Query Configuration page, you can click on the Query Name to open the specific details for the traffic query. At the Options tab on the bottom of the specific traffic query page you can select the "Clone Traffic Query" option. * Specific Traffic Query Creation Page -> When you've created a traffic query you will get a summary with the details of the query which has just been created. At the Options tab on the bottom of the specific traffic query page you can select the "Clone Traffic Query" option. C) Creating a Traffic Queries From PIX Log Entry Menu Item: (Main Menu) > "Traffic Logs" (Select) > Select a Traffic Log which has been logged by the firewall. (Expand) > "Options" (Select) > "Queries" - "Create a query from PIX Log ID." While it is useful to be able to create Traffic Queries manually or cloning them. You're most likely creating the traffic queries because of certain log entries you've seen. Therefore I've added the option to create a Traffic Queries straight from a PIX Traffic Log entry. This saves a lot of time since you don't have to worry about copy/pasting information across and it works in a similar way as cloning a Traffic Queries in that it takes the information from the specific PIX Log Entry and allows you to edit it as you see fit. Note: The same wildcards apply as for the "Creating a New Traffic Queries" section. 6.7 (b) RUNNING TRAFFIC QUERIES Menu Item: (Main Menu) > "Queries" Once the Traffic Queries have been created, they can be executed using at the "Queries" menu from the top menu bar. In order to execute the Traffic Query, you have to select the data for which the traffic query applies, then select the Query Name of the Traffic Query you wish to execute and select whether or not you would like to resolve the hostnames (by default no resolution is done). Click on the "GO" button in order to execute the query. When the Traffic Query has been executed, it will return a list of matching log entries and from this point on you can continue to use this PIX Logging Architecture page in the same way as you do with the Search Results or the Traffic Logs page. 6.8 CONFIGURING PARSE FILTERING In addition to Traffic Display filtering which filters out logged data from being displayed to the user, it is possible to also filter out traffic before it gets logged to the PLA database. There may be certain types of traffic which are overhead and you may not way to see logged, such as Nokia VRRP traffic, Multicast traffic, bulk HTTP, HTTPS, DNS and other type of traffic. Filtering out this traffic at the parsing stage ensures that the PLA database does not get "polluted" with unnecessarily logged traffic and helps maintain the database in keeping its size smaller. I've added Parse Filters exactly for this purpose. The way a parse filter works is by defining the criteria for traffic which should be filtered, for example * all outbound UDP DNS requests from your DNS server (10.10.10.10) should be filtered from being logged to the database: Source_IP: 10.10.10.10 Source_Port: Destination_IP: Destination_Port:53 Protocol:UDP * all outbound UDP DNS requests from your DNS server (10.10.10.10) to 192.168.1.1 and 192.168.1.2 and 192.168.1.3 should be filtered from being logged to the database: Source_IP: 10.10.10.10 Source_Port: Destination_IP:192.168.1.1 Destination_Port:53 Protocol:UDP Source_IP: 10.10.10.10 Source_Port: Destination_IP:192.168.1.2 Destination_Port:53 Protocol:UDP Source_IP: 10.10.10.10 Source_Port: Destination_IP:192.168.1.3 Destination_Port:53 Protocol:UDP As you can see in the above example, only a wildcard may be used by leaving the field blank and matching the entire field. Partial matching wildcards using the "%" (percentage) character are not allowed in the parse filters. NOTE: Please not that parse filters, unlike traffic filters, cannot use the "%" wildcard. Either the entire field needs to be "wildcarded" by leaving it blank or an exact and specific match needs to be defined. The reason I've done this is to ensure that all traffic excluded from the database is definitely the traffic you want to exclude. Although it may fill up the PLA database, it's always safer to log more and think you're logging less than logging less while you think you're logging more ;) NOTE: Once a new parse filter has been created it is not taken into account until the PLA Parse Daemon (pla_parsed) is restarted! There's three ways of creating a parse filter: A) Creating a Parse Filter Manually Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Global Configuration" (Sub Menu Choice) > "Configure Parse Filters" This page represents the main menu for the Parse Filter configuration. It will list all your parse filters and give you a little menu on the top when you click on the plus-sign in front of the "Parse Filter Options" link on the top. Using this menu you can create a new filter, search for certain variables within existing filters and show filters based on certain criteria. These latter two I'll let you find out later since they're quite trivial to use, so I'll focus on the Creation part. Creating a Parse Filter manually involves several steps: 1) On the Parse Filter configuration page, expand the "Parse Filter Options" and click on the "Create" button 2) You'll now get a page with all blank form fields, some of which are input boxes, others are drop down menus. The following lists all the required form criteria and whether wildcards can be used: - Filter Name [Required] (Short description of the filter) (input text box) - Logging Resource (Allows you to select which firewalls logs the parse filter should be active for) (choose one) + All Firewalls + - Logging Action (Allows you to select which traffic action the parse filter should be active for) (choose one) + Any + Accept + Drop - Protocol (Allows you to select which protocol the parse filter should be active for) (choose one) + TCP + UDP + ICMP - Source IP (Source IP for which the parse filter should be active) (ONLY LEAVE BLANK WILDCARDS ALLOWED!) (input text box) - Source Port (Source Port for which the parsefilter should be active) (ONLY LEAVE BLANK WILDCARDS ALLOWED!) (input text box) - Destination IP (Destination IP for which the parse filter should be active) (ONLY LEAVE BLANK WILDCARDS ALLOWED!) (input text box) - Destination Port (Destination Port for which the parse filter should be active) (ONLY LEAVE BLANK WILDCARDS ALLOWED!) (input text box) - Filter Description (Longer description of the filter) (input text box) Click on the "Create" button to create the filter or on the "Reset" button to reset the form 3) Once you've created the filter you'll get a page with the summary of the filter you've just created. You've got three options from this page: - Clone Parse Filter (I'll come back to this later) - Edit Parse Filter (Allows you to modify the filter you just created) - Go Back to the Parse Filter configuration page. 4) When you click on the "Back to the Parse Filter" configuration page, you should see the list of all created parse filters including the one you just created. When you create a filter it is enabled by default, which means that any traffic which matches the filter will not be logged to the PLA database. However, it is possible to disable the filter by clicking on the "disable" button in the options section of the parse filter main page. Other options include "clone" (which will be discussed shortly), edit (which allows you to edit the specific filter) and remove (which will remove the parse filter). You can view the filter details by clicking on the filter name column (first column starting from the left). B) Creating a Parse Filter With Cloning Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Global Configuration" (Sub Menu Choice) > "Configure Parse Filters" (Options Choice) > "[ clone ]" (within a selected parse filter in the "options" column) The purpose of Parse Filter Cloning is to re-use criteria entered within another filter and cloning them in order to create a new filter. So basically if we have a filter "A" which has DNS traffic defined between a specific source and destination IP pair and you want to add another destination for that same source IP and same traffic (DNS), you can clone filter "A" to create filter "B". Using the process of cloning you don't have to create an entirely new filter because you can just copy over the criteria from another filter. Basically what a clone does is it gives you the same page as you would have for creating a new filter, however several of the parameters are already filled in according to a pre-defined template, being the parse filter you're cloning from. In addition, a clone will add the "CLONED:" keywords in front of the Filter Name and Filter Description fields so that you don't create filters with duplicate names. Once you've got the cloned filter create window open, you can edit any variables you want in order to create the new filter, in the same manner as you would edit an existing filter. Note: As with the "Creating a New Parse Filter" section, only the Leave Blank wildcard works here. Since creating a cloned parse filter involves mainly similar steps to creating a new filter, I will not repeat the entire process, however it is important to know that you can clone a parse filter at three locations: * General Parse Filter Configuration Page -> Under the "Options" column, click on "[ clone ]" for the desired row matching the parse filter you want to clone. * Specific Parse Filter Viewing Page -> When you're at to the General Parse Filter Configuration page, you can click on the Filter Name to open the specific details for the parse filter. At the Options tab on the bottom of the specific parse filter page you can select the "Clone Parse Filter" option. * Specific Parse Filter Creation Page -> When you've created a parse filter you will get a summary with the details of the filter which has just been created. At the Options tab on the bottom of the specific parse filter page you can select the "Clone Parse Filter" option. C) Creating a Parse Filter From PIX Log Entry Menu Item: (Main Menu) > "Traffic Logs" (Select) > Select a Traffic Log which has been logged by the firewall. (Expand) > "Options" (Select) > "Parse Filter" - "Create a parse filter from PIX Log ID." While it is useful to be able to create Parse Filters manually or cloning them. You're most likely creating the parse filters because of certain log entries you've seen. Therefore I've added the option to create a Parse Filter straight from a PIX Traffic Log entry. This saves a lot of time since you don't have to worry about copy/pasting information across and it works in a similar way as cloning a Parse filter in that it takes the information from the specific PIX Log Entry and allows you to edit it as you see fit. Note: As with the "Creating a New Parse Filter" section, only the Leave Blank wildcard works here. 6.9 EXECUTING LOG PURGES When running PLA in an environment with a high amount of traffic you can easily start filling up the PLA database. In order to keep tabs on the size of the PLA database, I've added the Log Purging feature, which allows users to delete PLA log entries older than a certain date. While it is not possible to manually specify a log purging policy, several policies have been pre-defined which can be used to purge the PLA database. Menu Item: (Main Menu) > "Configuration" (Sub Menu) > "Global Configuration" (Sub Menu Choice) > "Configure Log Purging" The General Log Purging page can be found when clicking on the "Configure Log Purging" link under the "Global Configuration" section of the "Configuration" menu. The "Configure Log Purging" page lists the various log purging policies which can be selected by clicking on them. The following log purging policies exist for Traffic, IDS and Informational logs: * Older than 7 days * Older than 14 days * Older than 21 days * Older than 1 month * Older than 2 months * Older than 3 months * Older than 6 months * Older than 1 year * Older than 2 years * Older than 3 years Once you've clicked on one of the log purging policies, you will be taken to the log purging confirmation page, which shows various details such as the date prior to which logs will be purged, the number of records that will be purged and which require you to confirm two check boxes before being able to successfully carry out the purging of the logs. The following information is provided on the log purging confirmation page: * Purging Policy Name * Purging Policy Description * Purging Policy Target (traffic_log, ids_log or info_log) * Purging All Logs Newer Than: * Purging All Logs Older Than: * Number of Log Records Matching the Purging Criteria: The following check boxes need to be checked in order to allow the log purging to take place: * I wish to purge the PLA Database according to the Purge Policy "". * I accept the responsibility for the permanent deletion of certain records from within the PLA Database. I just added these confirmation check boxes so that someone wouldn't accidentally purge lets say.. the entire PLA database ;) After these check boxes have been checked, you can click on the "Purge Logs" button to start the log purging process. NOTE: Purging the logs may take a while and eventually time out (HTTP timeout to the PLA web-based front end) due to the size of the database. 7. PIX LOGGING ARCHITECTURE BEST PRACTICES 7.1 SCALING THE DEPLOYMENT When considering a PIX Logging Architecture deployment it is important to properly scale the machines running the infrastructure. Ofcourse the usage of the machines depends a great deal on the number of logs you're going to have, the number of queries you're gonna throw at the database and so-forth. However here's some general considerations regarding the deployment of PIX Logging Architecture v2.00: * PLA Parsing Daemon The PLA Parsing Daemon is responsible for extracting the log entries from the system log messages file and push them to the database. A lot of the work of the PLA Parsing Daemon is performed in memory so you'll have to ensure that the system running the PLA Parsing Daemon has an adequate amount of memory. For medium to large deployments I'd recommend between 256MB and 1024MB of memory. As for the CPU, I've had it running on P3 and P4 machines without much of a problem. Disk space is not necessarily a concern on the PLA Parsing Daemon since you can set up a log rotation policy for syslog which will "remove" the old system log files. * PLA Database The PLA Database is the most demanding component of the PLA infrastructure, it goes without say that enough disk space needs to be on the PLA Database server in order to house the database. To create an average statistic, each log entry takes up about 0.25Kb. So if you've got a network with 1.000.000 daily log entries into the PLA database then you're looking at a daily growth of the PLA database of +/- 400MB. So if you're willing to retain your logs for 90 days you're looking at 36GB of data. So ensure that whatever the capacity of your disk, it is enough to handle the disk space constraints introduced by PLA. Moreover, the PLA Database server is also in charge of handling the SQL queries, which can oftentimes be quite resource-intensive depending on the granularity of the request. You'll have to forsee a good amount of memory on this server anywhere between 512MB and 1-2GB depending on the types of the queries. As for CPU, I'd recommend a P4 or Dual Core CPU. * PLA Web-Based Front End The PLA Web-based Front End Server is the least resource intensive component out of all of 3 PLA components and does not have any special requirements. NOTE: Rather than separating each of these components on a dedicated server, you can also install all of these components on the same system. 7.2 RECOMMENDED DEPLOYMENT STRATEGY Whilst it is perfectly possible to install the different components of the PIX Logging Architecture infrastructure and let the logs come in, this may not be the most efficient approach. You may have traffic logged that you don't want to log. You may be logging certain log messages which are not necessarily useful to you, etc.. etc.. I've done several PIX Logging Architecture v2.00 deployments, both on small as well as on large networks and it's sometimes a very painful thing to find out that you're getting 100 log entries a second into the database and then need to start filtering. After a while your browser stalls when you're trying to load 5-6MB PLA web-front end pages and the like. Thus to minimize such "nightmares", I'd like to share the following recommended deployment steps with you (which I've learned the hard way;): 1) Install all 3 PLA components (PLA Parsing Daemon, PLA Database, PLA Web-Based Front End) and their required supporting software (Apache/MySQL/Perl/Perl Modules) 2) Start running the PLA Database and the PLA web-based front end - DO NOT activate the PLA Parsing Daemon yet 3) If you already know which specific traffic you do not want to log, start creating Parse Filters for this traffic. 4) If you already know which specific traffic you want to log but not display, start creating Traffic Display Filters for this traffic. 5) If available, a trending of network traffic top services and top connections would be a plus, if not you'll have to handle in a "best effort" methodology, but try and find out what overhead traffic may be transitting your networks, and create very broad filters for these types of traffic. Typical "high overhead traffic filters" are: - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 80 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 443 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 8080 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 25 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 110 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 53 - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 139 (that's always an evil one;) - Source_IP: ANY | Source_Port: ANY | Destination_IP: Any | Destination_Port: 445 (another evil one perhaps?;) 6) Activate Syslogd Remote Log reception on the Logging Host and configure the PIX/FWSM to log to this host - DO NOT activate the PLA Parsing Daemon yet. 7) Let Syslogd run for a predefined time period (1 hour is usually always a good sampling already) and look through the log file, ensure that the log messages which are indicated in the log file match those defined as parsing criteria in the "syslog_messages" table of the PLA database. For a list of all log messages which are logged or specifically excluded, please refer to the PIX Logging Architecture v2.00 web site. Also create traffic display filters and parse filters for any traffic which you may have noticed in the log file and do not want to log or display. 8) Create Traffic Queries for known and unknown traffic so you can easily identify these types of traffic. 9) Activate Syslogd Remote Log reception once more and start running the PLA Parsing Daemon (pla_parsed) 10) Connect to the PLA web-based front end and ensure that messages are entered correctly into the database. 11) Continue configuring Traffic Display Filters and Parse Filters as you see the accumulation of log traffic. 7.1 PARSING FUN The PIX Logging Architecture Parsing Daemon (pla_parsed) uses a predefined set of criteria to be able to match incoming logs to known parsing formats and thus extrapolate the required information from the log file. The way this is done is using a "syslog_messages" table within the PLA database which contains all the known log formats and defines what should be logged and should not be logged. The purpose of this section is to explain how parsing is done so that if you need to change anything to the PLA database parsing you'll know what to look for. No web-based front end is available for editing the syslog_messages table and thus you'll have to get your hands dirty inside the PLA database to be able to create some new log messages. The "syslog_message" table consists of the following fields, as shown in the following "describe syslog_message" output: +--------------------+-----------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------------+-----------------+------+-----+---------+----------------+ | field_id | int(8) unsigned | | PRI | NULL | auto_increment | | message_id | varchar(20) | YES | | NULL | | | message_type | int(8) | YES | | NULL | | | message_regex | varchar(255) | YES | | NULL | | | message_time | varchar(32) | YES | | NULL | | | message_action | varchar(32) | YES | | NULL | | | regex_src_ip | int(8) unsigned | YES | | NULL | | | regex_dst_ip | int(8) unsigned | YES | | NULL | | | regex_src_pt | int(8) unsigned | YES | | NULL | | | regex_dst_pt | int(8) unsigned | YES | | NULL | | | regex_xlate_src_ip | int(8) unsigned | YES | | NULL | | | regex_xlate_dst_ip | int(8) unsigned | YES | | NULL | | | regex_xlate_src_pt | int(8) unsigned | YES | | NULL | | | regex_xlate_dst_pt | int(8) unsigned | YES | | NULL | | | regex_idsmsg | int(8) unsigned | YES | | NULL | | | regex_proto | int(8) unsigned | YES | | NULL | | | regex_flags | int(8) unsigned | YES | | NULL | | | regex_time | int(8) unsigned | YES | | NULL | | | regex_sig | int(8) unsigned | YES | | NULL | | | regex_msg | int(8) unsigned | YES | | NULL | | | regex_iface | int(8) unsigned | YES | | NULL | | +--------------------+-----------------+------+-----+---------+----------------+ 21 rows in set (0.00 sec) PLA basically supports 3 types of log messages ("message_type" field) defined by the adjacent message types: - Traffic Logs (Message Type: 1) - IDS Logs (Message Type: 2) - Informational Logs (Message Type: 0) The PLA message types 1 & 2 work quite straightforward and require most of all fields within the syslog_message table to be filled out with the correct information for parsing the messages. For PLA message type 0 things works slightly different but we will come back on this later on. The PLA Parsing Daemon connects to the PLA database upon startup and downloads the entire syslog_message table into memory. For each new incoming message, the PLA parsing daemon will match the message_id (i.e. PIX-6-302015) of the incoming log entry with the ones it has extracted from the database. If it matches the database it's going to check the value for message_type, if it's 1 (Traffic Logs) or 2 (IDS Logs), the PLA parsing daemon will continue to parse all the information in accordance with the regular expressions (RegEx) condition specified in the "message_regex" field of the syslog_message table. Just to give an example on how this works, I'll give a practical example. Lets say a log message comes in of with the message_id PIX-6-302015. Below is the message followed by a table which lists the syslog_message field name, syslog_message field value for this field. Oct 28 20:31:29 fwext-dmz-01.dmz.counterstrike.mil Oct 28 2006 20:30:21: %PIX-6-302015: Built outbound UDP connection 6914 for outside:82.131.5.64/1254 (82.131.5.64/1254) to inside:10.1.10.8/48733 (80.200.224.122/2159) +--------------------+------------------------------------------------------------------------------------------------+ | Field | Value | +--------------------+------------------------------------------------------------------------------------------------+ | field_id | 4 | | message_id | PIX-6-302015 | | message_type | 1 | | message_regex | Built outbound (.*) connection .* for .*:(.*)/(.*) .(.*)/(.*). to .*:(.*)/(.*) .(.*)/(.*).. | | message_time | %b %d %Y %H:%M:%S | | message_action | ACCEPT | | regex_src_ip | 6 | | regex_dst_ip | 2 | | regex_src_pt | 7 | | regex_dst_pt | 3 | | regex_xlate_src_ip | 8 | | regex_xlate_dst_ip | 4 | | regex_xlate_src_pt | 9 | | regex_xlate_dst_pt | 3 | | regex_idsmsg | 0 | | regex_proto | 1 | | regex_flags | 0 | | regex_time | 0 | | regex_sig | 0 | | regex_msg | 0 | | regex_iface | 0 | +--------------------+------------------------------------------------------------------------------------------------+ So if we were to take the log message shown above and fill in the variables, i.e. when evaluating the log message using the regular express we come up with the following results: message_regex = "Built outbound UDP connection 6914 for outside:82.131.5.64/1254 (82.131.5.64/1254) to inside:10.1.10.8/48733 (80.200.224.122/2159)" message_action = ACCEPT regex_src_ip (position #4 in message_regex) = 10.1.10.8 regex_dst_ip (position #2 in message_regex) = 82.131.5.64 regex_src_pt (position #7 in message_regex) = 48733 regex_dst_pt (position #3 in message_regex) = 1254 regex_xlate_src_ip (position #8 in message_regex) = 80.200.224.122 regex_xlate_dst_ip (position #4 in message_regex) = 82.131.5.64 regex_xlate_src_pt (position #9 in message_regex) = 2159 regex_xlate_dst_pt (position #3 in message_regex) = 1254 regex_proto = 1 (UDP) Based on this extrapolated information, the PLA Parsing Daemon then generates a log entry which it will push to the database, unless the specific log entry matches a known parse filter. So if you wish to write your own parsing criteria you can write them as shown above and thus keep PLA up to date. I myself will do my best to keep the PLA database as up to date as possible. Currently I've got a total number of 23 Traffic and IDS log formats defined for PIX and FWSM together. NOTE: Remember that when you want to define Traffic or IDS log criteria, be specific and include the Regex statement followed by the proper positions for the different variables. Next, lets look at how PLA treats the message_type 0. Message Type 0 serves a double purpose: - Define what should be logged into the Informational Logs table. - Define what should not be logged at all. Two conditions exist: 1) IF message_type = 0 and message_action = 1 then the log should be included as an informational log entry. For example: PIX log message PIX-6-603109 (which informs about PPPoE status) - adheres to this condition (message_type=0 / message_action=1) Thus the following message: Oct 29 06:55:15 fwext-dmz-01.dmz.counterstrike.mil Oct 29 2006 06:54:07: %PIX-6-603109: Teardown PPPOE Tunnel, tunnel_id = 0, remote_peer_ip = 80.200.224.1 Will be logged as an informational log message with the following information field: "Teardown PPPOE Tunnel, tunnel_id = 0, remote_peer_ip = 80.200.224.1" 2) IF message_type = 0 and message_action = 0 then the log should be excluded and not even considered by PLA Parse Daemon. This condition is valid for ALL message types (Traffic Logs, IDS Logs, Informational Logs) regardless of their nature. This has been added to provide the ability to users to exclude certain log messages from being logged based on the message id (i.e. PIX-6-305011). Excluding specific log entries can be a critical part of keeping the PLA database down to a reasonable size. For example, PIX Log Message PIX-6-305011 (which describes Port Address Translation) for dynamic port openings is not logged to the PLA database because it has been excluded. 8. ADDITIONAL INFORMATION 8.1 THE PEOPLE BEHIND PIX LOGGING ARCHITECTURE I'd like to take this opportunity to introduce the people who have worked on making PIX Logging Architecture a reality. * Kris(tof) Philipsen - PLA Author / Developer [kris@logging-architecture.net] [http://kris.sequenced.org] Just a quick intro.. I'm Kris Philipsen, Senior Security Consultant at Cybertrust, Inc. [http://www.cybertrust.com] and based in their Luxembourg, Europe office. My interest in Cisco PIX, FWSM and everything that has to do with Cisco has started a long time ago now which is the reason why I'm running a PIX firewall here at my house and was having, like many other people, tremendous problems with Cisco's log strategy so I decided to change some things and make my (and hopefully everybody elses) life a little easier by bringing the PIX Logging Architecture project to life. As far as Cisco goes, I'm currently CCNP, CCSP and am working towards my CCIE Security for which I've passed the written exam so far. * Peter Boutzev - PLA Developer [peter@logging-architecture.net] [http://airfair.dnsalias.org] I'm gonna take this opportunity to introduce Peter. A great friend and colleague! Peter Boutzev is Senior Security Consultant at Cybertrust, Inc. [http://www.cybertrust.com] and based in their Luxembourg, Europe office. Peter, just like Kris, is also very interesting in everything that has to do with Cisco stuff as well as open source software. I definitely want to thank Peter for all of the great efforts and time he's put into helping me co-develop PLA and coming up with new and exciting features! It's an honor to work with you Peter. I'd also like to give mad kudos to Peter for passing the CCIE Security lab exam this year.. so Peter's now officially part of the CCIE Security club.. congrats and hope to follow you there soon! 8.2 ACKNOWLEDGEMENTS AND KUDOS Thanks to all people who have been wonderful in helping improve PIX Logging Architecture and making it into what it is today. I'd like to especially thank my friend Christophe who pre-beta tested PIX Logging Architecture v2.00 in the corporate infrastructure of the company he works for and has been great in giving me feedback on the FWSM testing and has come up with some ideas for new features which have currently been integrated into PLA v2.00. I'd also like to thank Viviane, the woman of my life who has put up with me while I've been working on designing, coding and improving PIX Logging Architecture. I love you! 8.3 SUPPORT THE PIX LOGGING ARCHITECTURE PROJECT If you like the PIX Logging Architecture project and would like to support us there's several ways of doing this: * Help Improve PIX Logging Architecture Help us improve PIX Logging Architecture by giving your feedback, comments and bug reports so we can get on making PLA a better piece of software. * Link to the http://www.logging-architecture.net website. We've got a small logo which you can use to link to us using the following HTML code: PIX Logging Architecture * Keep The PIX Logging Architecture Project Running PIX Logging Architecture is a software which is being provided for free and can be used without any obligations or monetary compensation. Nevertheless, costs (such as hosting as well as hardware investments) are involved in running the PIX Logging Architecture project and keeping it as up to date as possible and support the latest software and hardware technologies which we try to maintain through the means of donations and sponsored ads clicks. Your click to our sponsors as well as your donations to this project can make the difference in keeping the site and it's project up and running! Please refer to the http://www.logging-architecture.net/pla2/ website for more information on this. 8.4 PIX LOGGING ARCHITECTURE RESOURCES Besides this Installation, Configuration and Usage guide for PIX Logging Architecture, I've also put 4 mailing lists up regarding various topics relating to PIX Logging Architecture: * pixla-announce * pixla-bugs * pixla-comments * pixla-support Please refer to the following page if you're interested in subscribing to these mailing lists: http://www.logging-architecture.net/mailinglists.html