Author Archives: apomeroy

Installation notes for Logger 6 on CentOS

Posted on by

[Update 2016/04/15]:  Installing Logger 6.2 on CentOS 7.1

CentOS (or RHEL) 7 changed a number of things in the OS for command and control, such as the facility to control services – for example, rather than “service” the command is now “systemctl”.  Below I outline a “quickstart” way to get HPE ArcSight Logger 6.2 installed on CentOS 7.1 (minimal distribution). Of course you want to read the Logger Installation Guide, Chapter 3 “Installing Software Logger on Linux” for the complete instructions and be sure you understand the commands I suggest below before you run them. No warranties here, just suggestions.  😉

  1. Do a base install of CentOS (or RHEL) 7.1, minimal packages.  I often suggest adding in Compatibility Libraries, however for this Logger 6.2 install, I just used the base install.  Ensure /tmp has at least 5GB of free space and /opt/arcsight has at least 50GB of usable space – I’d suggest going with at least:
    • /boot – 500MB
    • / – 8GB+
    • swap – 6GB+
    • /opt – 85GB+
  2. Ensure some needed (and helpful) utilities are installed, since the minimal distribution does not include these and unfortunately the Logger install script just assumes they are there .. if they aren’t, the install will eventually fail (such as no unzip binary).
    • yum install -y bind-utils pciutils tzdata zip unzip
    • Unlike my ESM install, for Logger, I left SELinux enabled and things appear to be working alright, but your mileage may vary.  If in doubt, disable it and try again.  To disable, edit /etc/selinux/config and set the mode to “disable” (or at least to “permissive”)
    • Disable the netfilter firewall (again, at some point I’ll update this with the rules needed to leave netfilter enabled).
    • systemctl disable firewalld; systemctl mask firewalld
    • Install and configure NTP
    • yum install -y ntpdate ntp
    • (optionally edit /etc/ntp.conf to select the NTP servers you want your new Logger system to use)
    • systemctl enable ntpd; systemctl start ntpd
    • Edit /etc/rsyslog.conf and enable forwarding of syslog events to your friendly neighborhood syslog SmartConnector (optional, but otherwise how do you monitor your Logger installation?) .. you can typically just uncomment the log handling statements at the bottom of the file and fill in your syslog SmartConnector hostname or IP address. Note the forward statement I use only has a single at sign – indicating UDP versus TCP designated by two at signs:
    • $ActionQueueFileName fwdRule1 # unique name prefix for spool files
      $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
      $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
      $ActionQueueType LinkedList # run asynchronously
      $ActionResumeRetryCount -1 # infinite retries if host is down
      # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
      #*.* @@remote-host:514
      *.* @10.10.10.5:514
    • Restart rsyslog after updating the conf file
    • systemctl restart rsyslog
    • Optionally add some packages that support trouble shooting or other non-Logger functions you run on the Logger server, such as system monitoring
    • yum install -y mailx tcpdump
  3. Update the maximum number of processes and open files our Logger software can use:
    Backup the current settings:
    cp /etc/security/limits.d/20-nproc.conf /etc/security/limits.d/20-nproc.conf.orig
    Drop in new config file (assuming you have copy/pasted the following settings into /root/20-nproc.conf):
    cp 20-nproc.conf /etc/security/limits.d/20-nproc.confContents of the /etc/security/limits.d/20-nproc.conf file becomes:
    # Default limit for number of user's processes to prevent
    # accidental fork bombs.
    # See rhbz #432903 for reasoning.
    * soft nproc 10240
    * hard nproc 10240
    * soft nofile 65536
    * hard nofile 65536
    root soft nproc unlimited
    Reboot to enable the new settings.
  4. Add an unprivileged user “arcsight” to own the application and run as:
    groupadd -g 1000 arcsight
    useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
    passwd arcsight
  5. Ensure the *parent* directory for the Logger software exists. Standard locations for installation of ArcSight products should be /opt/arcsight, so for example, we’re going to install our Logger software at /opt/arcsight/logger.
    cd /opt
    mkdir /opt/arcsight
  6. Run the Logger installation binary as “root” user
    • ./ArcSight-logger-6.2.0.7633.0.bin
  7. After the installation script completes successfully, you should be able to login to the console via a web browser https://<hostname>
    Default username “admin” with default password “password”. You’ll be forced to change the admin password on login.
  8. If you are going to install any SmartConnectors on the system hosting your Logger, check out my post regarding required libraries for CentOS and RedHat, before you try to run the Linux SmartConnector install. This includes any Model Import Connectors (MIC) or forwarding connectors (SuperConnectors).

 

[Update 2016/03/11]: Starting with SmartConnector 7.1.7 (I think, might be a rev or two earlier), there are a couple more libraries that are needed to successfully install the SmartConnector on Linux. Include libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64
yum install libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64

These notes describe an installation of HP ArcSight Logger 6.0.1 on a CentOS 6.5 virtual machine.

For a test install of Logger 6, I built a CentOS vm with the following parameters:
Basic install from the CentOS 6.5 Minimum ISO
1 CPU with 2 cores
4GB memory
80GB virtual disk
1 bridged network adapter
Disk partition sizes:
root fs 6GB, swap 4GB, /home 2GB, /opt/arcsight 50GB, /archive 10GB, free space approximately 15GB

As soon as the system was up, I commented out the archive filesystem (will be re-mounted under the /opt/arcsight/logger directory)
vi /etc/fstab

Installed the bind-utils package so I could use dig and friends, then did a full yum update:
yum install bind-utils ntp
yum update

This turns the system into CentOS 6.6, but that’s still a supported system for Logger, so all’s good.

Next we prepare the system for Logger software install by adding a user and changing some of the system configuration.

Add a non-root user to own and run the Logger application:
groupadd -g 1000 arcsight
useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
passwd arcsight

Install libraries that Logger depends on:
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686
yum install zip unzip

Update the maximum number of processes and open files our Logger processes can have:
cp 90-nproc.conf /etc/security/limits.d/90-nproc.conf

Contents of the /etc/security/limits.d/90-nproc.conf file becomes:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
*          soft    nproc     10240
*          hard    nproc     10240
*          soft    nofile    65536
*          hard    nofile    65536
root       soft    nproc     unlimited

Turn off services we don’t need and turn on the ones we do need. Later we will write some iptables rules so we can turn the firewall back on when we’re done.

chkconfig iptables off
service iptables stop
chkconfig iscsi off
service iscsi stop
chkconfig iscsid off
service iscsid stop
ntpdate name-of-ntp-server-you-trust
chkconfig ntpd on
service ntpd start

All of these steps are packaged up here in centos-setup.shl:
groupadd -g 1000 arcsight
useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
passwd arcsight
cp 90-nproc.conf /etc/security/limits.d/90-nproc.conf
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686
yum install zip unzip
chkconfig iptables off
service iptables stop
chkconfig iscsi off
service iscsi stop
chkconfig iscsid off
service iscsid stop
ntpdate 0.centos.pool.ntp.org
chkconfig ntpd on
service ntpd start

Turns out since we need 3+GB of free space in /tmp, I needed to extend the root filesystem .. I only allocated 2GB to begin with. Extend the root logical volume (lv_root) by adding 1,000 Physical Extents (4MB each):

Boot into rescue mode .. do NOT mount linux partitions, then drop to a shell

vgs
vgchange -a y vg_swlogger1
lvextend -l +1000 /dev/vg_swlogger1/lv_root
e2fsck -f /dev/vg_swlogger1/lv_root
resize2fs /dev/vg_swlogger1/lv_root

Now reboot and confirm there is at least 4GB of free space in /tmp. Could also have mounted a ram filesystem, but this will do as I’m conserving my memory on the host.

Upload the Logger installer binary and also the license file to the system into root’s home directory (or where you have space).

As root, run the Logger software install:
chmod u+x ArcSight-logger-6.0.0.7307.1.bin
./ArcSight-logger-6.0.0.7307.1.bin

Word of advice .. if doing this in a vm, run the install from the vm console since it’s possible the vm will be busy enough a remote ssh session could get disconnected – and the install will not complete properly.

After the install, we should be able to open a browser by navigating to https://name-of-vm-here

Sign in as arcsight / password then navigate to the System Administration section to change the admin password.

Creating event replay files for ArcSight SmartConnectors

Posted on by

The ArcSight connector framework includes the capability to record event replay files from inbound event streams, regardless of the type of event data. This is enormously useful for development and testing individual of use cases, demonstrations and training. The following article is based on ArcSight SmartConnector version 7.0.7.

Events are replayed back to the target destinations by selecting some variety of previously recorded replay files using an ArcSight Test SmartConnector. Either multiple event files or a consolidated file can be used with the Test Alert connector. Since the Test Alert connector is a standard SmartConnector, multiple destinations can be configured, such as to Enterprise Security Manager (ESM) and/or Logger. As event files are replayed back into the target(s), the timestamp can be the original or can be overridden to the current time. This enables historical analysis as well as event data appropriate for any time sensitive rules or use cases.

Create Replay File Directly From Connector

1. Shut down Connector Service.
2. Open the .../current/user/agent/agent.properties file, add following two properties to agent.properties file:

agent.component.count=36
agent.component[35]=com.arcsight.agent.loadable._RecordComponent

agent.properties replay configuration

3. Start Connector Service again

The Connector will start capturing events being sent to ESM, writing the output to .../current/replayagent/{agent-id}.sessions

4. Stop the Connector when you are done capturing events
5. Open the agent.properties file again and remove or comment out the lines added in step 2, then restart the connector again
6. Rename the .sessions file to .events and copy it to the …/current directory of the Testalert SmartConnector and start (or restart) the Test Alert SmartConnector.
7. Start Test Alert to replay the file.

Testalert Connector

Once the replay file or files are selected, the events can be replayed into the system with a specified Event Per Second (EPS) rate

replay-event-rate

Optimizing the Collection and Replay

By default, the Test Alert SmartConnector will replay the recorded events with a current timestamp.  Where it is desirable to replay the events with the original timestamp, the connector can be configured through the normal connector reconfiguration (…/current/bin/runagentsetup.sh)

One of the disadvantages of this approach is apparent if using this method to collect sample event data that would not normally be directed to your ESM instance. On the source SmartConnector, the destination can be set to be a CSV file – enabling the ability to turn very large event feeds directly into .event files without using any ESM storage and processing capacity.

Replaying Events with Original Timestamps

To enable replay of the recorded events with their original timestamps, edit the .../current/user/agent/agent.properties file and add or uncomment out the following lines:

agents[0].preserveagenttime=true
agents[0].preservedetecttime=true

When the Test Alert SmartConnector starts again, the events will be replayed with original timestamps.
 

Enabling Single Line Logging from pfSense Firewalls to ArcSight

Posted on by

While pfSense firewall offerings are based on the BSD packet filter (pf) functions and offer excellent performance and value, the current implementation my customers are running (2.1.5) outputs firewall rule logs in two syslog lines.  The skilled developers that maintain pfSense have indicated that in the 2.2 release they will likely move to a single line log format that is friendly to machine parsers, however I needed to provide some ArcSight parsing of this log data immediately.

Two approaches are possible here.  I could write a FlexConnector parser that expects and parses two consecutive syslog lines into one in order to pull out the relevant source, destination addresses, ports and device handling information (pass, block, reject, interface, protocol, flags, etc).  The other option I discovered was what I implemented since my customer was willing to allow a system patch to be applied to the firewalls that outputs log entries on a single line.  This will make a FlexConnector parser a breeze to create.

After looking through the pfSense support forums, I found this post by jimp (one of the developers) that provided a patch to enable single line logging.  To install the patch, you need to install the System Patch installation package on the firewall, since it’s not there by default.

pfSense-systemPatchPackage

Package install on pfSense Firewall

 

Enables installation of patches

Enables installation of patches

 

 

From the Package Manager, install the System Patch package.  This enables downloading and automating the patch installation process.  This is a safe way to apply patches, since it will validate the patch can be applied and more importantly backed out if needed.

Select the Patches option in the System menu to start the patch application process. At this point you can (and probably should) just enter the URL to the patch versus manually entering the patch diff file in the text dialog box that comes up. Entering the URL (http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff) gives the ability to Fetch the contents of the patch (double click on the Fetch option).  pfSense-patchThe System Patch package will go download the patch at the given URL and validate it.  If the validation is successful (it can apply and also roll back the patch), then the Test and Apply links should automatically show up.  If not, only the Test link will show up .. click on the Test link to validate the patch can be applied and backed out.  This should happen automatically so the Apply link shows up.  After application of the patch there will be apfSense-rawLogFormatOption new option that shows up in the Status > System Logs > Settings pane that allows the raw logs to be written out on a single line.pfSense-block

Now the logs are coming in on a single line, creation of an ArcSight FlexConnector is simple.  I first directed logs from the firewall to a standard syslog daemon SmartConnector (7.0.5) and enabled Preserve Raw so I can grab pfSense-raw-logthe original event and write the parser to grab all the relevant fields I’m looking to correlate.

Using the ArcSight FlexConnector tool kit I was able to get it to suggest a starting point for the regex patterns to parse the single line logs.  Launched the regex GUI with arcsight regex in any connector’s current/bin directory. I copied a FlexConnector config file from an iptables parser I wrote to use to start from .. and produced a quick and dirty mapping of the pfSense events with this parser in user/agent/flexagent/syslog/pfsense.subagent.sdkrfilereader.properties.  Note this is a work in progress, so no categorization map files have been provided yet.  That will come soon.

 

Building a Highly-Available ArcSight SmartConnector Cluster with Pacemaker

Posted on by

Cost Effective SmartConnector HA

This paper describes the use of open source clustering software used to build a low-cost, reliable, high availability environment on CentOS Linux in which to run both passive and active SmartConnectors, providing automated failure recovery.

Introduction

At current time there is no inherent High-Availability capability for ArcSight SmartConnector installations other than HA management of connectors through multiple Connector Appliances. Once events have been acquired by a SmartConnector, the store-and-forward architecture provides a reliable event handling ecosystem, but the problem is what to do when a specific SmartConnector, or the system it is running on, fails. Traditionally customers would procure and employ hardware load balancers in front of SmartConnector Connector Appliances or Connector Concentrators, although that only really deals with passive connectors, such as syslog, SNMP or other listeners. Active connectors such as Windows, Database readers, etc would require a manual failure recovery in order to restore the service of event collection. Although customers can use commercial clustering technology, such as Veritas Cluster Server, those tools can require substantial capital investment. This paper describes the use of open source clustering software used to build a low-cost, reliable, high availability environment in which to run both passive and active SmartConnectors, providing active failure recovery and service continuance. This configuration is not endorsed or supported by HP Enterprise Security Products and is provided for informational purposes only.

This package includes documentation and scripts to setup a cluster from scratch in an automated manner. Access to cluster packages in CentOS or local customer provided repositories is needed by the setup scripts. Users of this package need to obtain a Linux binary of the HP ArcSight SmartConnector software – it is not included. The result of the included quickstart script will be a functional cluster with a syslog SmartConnector running and able to fail-over to a partner node in the case of primary node failure. The two cluster nodes must have at least two (2) network segments, although all traffic to/from the event sources can be on any customer network that is reachable via standard IPv4 routing – the cluster does not operate in-line but rather as a distinct IP node on the customer network.

Assuming a relatively fast connection to the Internet, or internal servers, for access to the CentOS software repositories, the quickstart script can complete the cluster setup in less than 15 minutes, but one should expect to take a day to review the cluster configuration, commands and proper operating procedures. Recovery from incorrect cluster commands or operations will almost assuredly require a cluster outage for re-configuration, resync or worse, backup/recovery. Given the relative low cost of simple 1U servers, it is strongly recommended that two pairs of nodes are used to create a test cluster and production cluster. Modest VMware or other virtual servers can be used to implement the test environment. TCP/UDP protocol ports that are used are specific to the unique cluster IP addresses, so there should not be any collisions – although care must be taken to choose unique multicast addresses for the cluster communication provided by corosync. This is not done automatically by the quickstart scripts.

Feed back is welcomed, both success stories and problems/bugs that are encountered, but users need to self-support any implementations. The current maintainer is Allen Pomeroy (a at pomeroy dot us)

Download the Whitepaper and Cluster setup scripts in this zip file: BuildingAHASmartConnectorCluster-2.0.6

Attributes of Successful People

Posted on by

Posted by one of my friends on LinkedIn .. Attributes of Successful People

Successful people:

  • Have a sense of gratitude
  • Compliment others
  • Forgive
  • Give others appropriate credit for victories
  • Read everyday
  • Share information and ideas
  • Embrace change
  • Keep a to-do and/or project list
  • Set goals and develop life plans
  • Continuously learn
  • Keep a journal
  • Accept responsibility for their failures
  • Operate from a transformational perspective

Outbound network traffic with multiple interfaces

Posted on by

Why does Red Hat Enterprise Linux 6 invalidate / discard packets when the route for outbound traffic differs from the route of incoming traffic?

Issue Description
Why does Red Hat Enterprise Linux 6 invalidate / discard packets when the route for outbound traffic differs from the route of incoming traffic?
Why does Red Hat Enterprise Linux 6 differ from Red Hat Enterprise Linux 5 in handling asymmetrically routed packets?

Solution posted at access.redhat.com/site/solutions/53031

Red Hat Enterprise Linux (RHEL) 6 Resolution

Temporary change
To accept asymmetrically routed (outgoing routes and incoming routes are different) packets set “rp_filter” to 2 and restart networking, by running the following commands:

echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

Persistent change
To make this behaviour persistent across reboots, modify /etc/sysctl.conf and make the following change prior to reboot:

net.ipv4.conf.default.rp_filter = 2

Root Cause

RHEL6 (unlike RHEL5) defaults to using ‘Strict’ Reverse Path Forwarding (RPF) filtering.

Comments
The sysctl net.ipv4.conf.default.rp_filter selects the default RPF filtering setting for IPv4 networking. (It can be overridden per network interface through net.ipv4.interfacename.rp_filter).

Both RHEL6 and RHEL5 ship with a default /etc/sysctl.conf that sets this sysctl to 1, but the meaning of this value is different between the RHEL6 and the RHEL5 kernel.

Microsoft Shortcuts and Notes

Posted on by

Searching within Outlook 2010

Useful keyword searches:

category:=”Red Category”
read:no Items that have not been read. You can also use read:false to get the same results.
category:business Items that are categorized as business.
messagesize:enormous Items whose size is larger than 5 megabytes

Windows 7
Launch command window based on specific folder

Open command window here

Make your own Reduce File Size presets for PDF export

Posted on by

Within Preview there is a filter that can be used to reduce the size of PDF files (think of PDF files that are 600 DPI high resolution).  Unfortunately it produces very poor quality images to the point of being unusable. Fortunately there is a way to create and install your own custom quartz filters for use in Preview that give large file size reductions while maintaining good quality.

After some googling, I found a perfect article that explains why the default Mac OS X Reduce File Size filter produces terrible quality images .. and how to fix that:

http://hints.macworld.com/article.php?story=20120629091437274

The filter, which is just a XML file, can be edited with any text or programming editor then saved to the  /System/Library/Filters  directory with a unique filename.  The Reduce File Size (Good) filter is what I use .. rather than posting as a code block and messing around with escaping the XML so the code displays correctly, the file is available for [download here].

Simply download the contents of this file, ensure it is renamed to a .qfilter file, then copy into the system filter directory (so it is available for all users). I chose to use /System/Library/Filters/Reduce File Size Good.qfilter. You may need to be a Mac OS X Administrator to write this file into the shared system library folder. At this point, in Preview you can use this filter to reduce large scanned PDF files by almost a magnitude of order.

Here is the text of the original post:

 
Make your own Reduce File Size presets for PDF export
Jul 05, ’12 07:30:00AM Contributed by: zpjet

I was never satisfied with results of “Reduce File Size” Quartz filter when trying to make some PDFs smaller before sending them by e-mail. It made them too small, and the graphics were fuzzy.

I eventually found where these filters are:

/System/Library/Filters

I was delighted to find out they’re XML files easily editable with TextEdit (or any other text editor). I also found why this particular filter makes quite unusable PDFs, as these parameters were just too low:

Compression Quality 0.0
ImageSizeMax 512

So I copied this file to my Desktop, and then made two more copies of it, and called them Reduce File Size Good, Better and Best. Then I changed the parameters of each file to 0.25, 0.5 and 0.75 for Compression Quality, and used these three values for ImageSizeMax:

842 (that’s A4 at 72dpi)
1684 (A4 at 144dpi)
3508 (A4 at 300dpi)

Finally, I changed the default string for the Name key at the end of each file to reflect the three settings, so they display the names I have given them in the menu.

Then I copied them to a /Library/Filters folder I created (for some reason, ~/Library/Filters doesn’t work in Lion) and now when I open a picture or PDF in Preview, I have the option of four different qualities for reduced file sizes.

As an example, I have a JPEG of scanned A4 invoice at 300dpi and it’s 1.6MB. When exporting to PDF in reduced size, the file is only 27 KB and it’s quite unusable – very fuzzy and hard to read. The Good one is much easier to read, slightly fuzzy and still only 80 KB. Better is 420 KB and clear, and the Best is 600 KB and almost as good as the original even on a laser printer.

Apple Airport Utility 5.6 on Mountain Lion (10.8.4)

Posted on by

Apple has been continuously dumbing down the AirPort Utility to the point where their default AirPort Utility (version 6.x) is virtually useless.  Settings such as syslog destination, NTP settings, etc can only be set with AirPort Utility 5.6 – problem is it cannot be loaded on Mountain Lion (10.8.4).  Discussion here at https://discussions.apple.com/thread/4172563?start=15&tstart=0 shows how to install AirPort Utility 5.6.1 on ML without clobbering version 6.

In case Apple removes that thread (written by Douglas Urner), here it is:

Just in case you’d like to use AirPort Utility 5.6.1 on Mountain Lion (and probably Lion as well), here’s how to install it:

  1. Download the disk image (you can find it here:http://support.apple.com/kb/DL1536).
  2. Mount the disk image and drag the install package (AirPortUtility.pkg) to your desktop.
  3. Fire up Terminal and prepare to show off…
  4. Make a temporary directory and cd into it: mkdir tmp ; cd tmp
  5. Extract the Payload file from the install package with xar, here’s the command: xar -x -f ~/Desktop/AirPortUtility.pkg Payload
  6. The result will be a directory named AirPortUtility.pkg (just like the file, but now you can move into it to get the files you want). Inside will be a file called Payload that is a compressed archive of AirPort Utility.app.
  7. So our next move is to extract the app. Here’s the command: gzcat AirPortUtility.pkg/Payload | tar -xf –
  8. When it finished there will be three new folders Applications, Library, and System. Your nice new copy of AirPort Utility 5.6.1 will be in the Utilities folder inside of the Applications folder. Use Finder to rename it (assuming you want to keep version 6 as well) then drag it to your Applications/Utilities folder.
  9. The other two folders hold the AirPort Base Station Agent and its supporting files. I’m not sure if you need/want these or not. As best I can figure the agent does two things: it checks for updates for AirPort Utility and it monitors AirPort base stations for problems. You probably already have a version running as it comes with the system and it seems to know how to talk to both versions of AirPort Utility (I got nagged about updating).
  10. The final step is to launch AirPort Utility and confirm that it works. You’ll probably want to go into preferences and turn off the option to check for updates. If all is good you can remove the temporary directory: cd .. ; rm -rf tmp (or drag it into the trash with Finder).

That pretty much does it. I hope it helps somebody out there.