How To Increase ArcSight ESM Command Center GUI Timeout

In the appliance versions of most ArcSight products, there is the ability to set the user session timeout period. Typically this defaults to somewhere between five (5) and 15 minutes – good for a default but incredibly annoying for any real user.  In ArcSight Enterprise Security Manager (ESM), there is no such GUI configuration that allows modification of the user session timeout – so this is what has worked for me:

Set ArcSight Command Center (ACC) timeout greater than 900 seconds (15 minutes) – set to 28800 seconds (8 hours)
vi /opt/arcsight/manager/config/server.properties
service.session.timeout=28800
/sbin/service arcsight_services stop all
/sbin/service arcsight_services start all

Default is 600 seconds = 5 minutes.

In 6.5, 6.5.1 and 6.8 you also need to add the following for the Logger interface in ESM:

vi /opt/arcsight/logger/userdata/logger/user/logger/logger.properties
server.search.timeout=28800
/sbin/service arcsight_services stop all
/sbin/service arcsight_services start all

Default is 600 seconds = 5 minutes.

Yes, eight (8) hours may seem like a long time, so chose what is appropriate for your site.  :)

Installation notes for Logger 6 on CentOS

[Update 2016/04/15]:  Installing Logger 6.2 on CentOS 7.1

CentOS (or RHEL) 7 changed a number of things in the OS for command and control, such as the facility to control services – for example, rather than “service” the command is now “systemctl”.  Below I outline a “quickstart” way to get HPE ArcSight Logger 6.2 installed on CentOS 7.1 (minimal distribution). Of course you want to read the Logger Installation Guide, Chapter 3 “Installing Software Logger on Linux” for the complete instructions and be sure you understand the commands I suggest below before you run them. No warranties here, just suggestions.  ;-)

  1. Do a base install of CentOS (or RHEL) 7.1, minimal packages.  I often suggest adding in Compatibility Libraries, however for this Logger 6.2 install, I just used the base install.  Ensure /tmp has at least 5GB of free space and /opt/arcsight has at least 50GB of usable space – I’d suggest going with at least:
    • /boot – 500MB
    • / – 8GB+
    • swap – 6GB+
    • /opt – 85GB+
  2. Ensure some needed (and helpful) utilities are installed, since the minimal distribution does not include these and unfortunately the Logger install script just assumes they are there .. if they aren’t, the install will eventually fail (such as no unzip binary).
    • yum install -y bind-utils pciutils tzdata zip unzip
    • Unlike my ESM install, for Logger, I left SELinux enabled and things appear to be working alright, but your mileage may vary.  If in doubt, disable it and try again.  To disable, edit /etc/selinux/config and set the mode to “disable” (or at least to “permissive”)
    • Disable the netfilter firewall (again, at some point I’ll update this with the rules needed to leave netfilter enabled).
    • systemctl disable firewalld; systemctl mask firewalld
    • Install and configure NTP
    • yum install -y ntpdate ntp
    • (optionally edit /etc/ntp.conf to select the NTP servers you want your new Logger system to use)
    • systemctl enable ntpd; systemctl start ntpd
    • Edit /etc/rsyslog.conf and enable forwarding of syslog events to your friendly neighborhood syslog SmartConnector (optional, but otherwise how do you monitor your Logger installation?) .. you can typically just uncomment the log handling statements at the bottom of the file and fill in your syslog SmartConnector hostname or IP address. Note the forward statement I use only has a single at sign – indicating UDP versus TCP designated by two at signs:
    • $ActionQueueFileName fwdRule1 # unique name prefix for spool files
      $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
      $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
      $ActionQueueType LinkedList # run asynchronously
      $ActionResumeRetryCount -1 # infinite retries if host is down
      # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
      #*.* @@remote-host:514
      *.* @10.10.10.5:514
    • Restart rsyslog after updating the conf file
    • systemctl restart rsyslog
    • Optionally add some packages that support trouble shooting or other non-Logger functions you run on the Logger server, such as system monitoring
    • yum install -y mailx tcpdump
  3. Update the maximum number of processes and open files our Logger software can use:
    Backup the current settings:
    cp /etc/security/limits.d/20-nproc.conf /etc/security/limits.d/20-nproc.conf.orig
    Drop in new config file (assuming you have copy/pasted the following settings into /root/20-nproc.conf):
    cp 20-nproc.conf /etc/security/limits.d/20-nproc.confContents of the /etc/security/limits.d/20-nproc.conf file becomes:
    # Default limit for number of user's processes to prevent
    # accidental fork bombs.
    # See rhbz #432903 for reasoning.
    * soft nproc 10240
    * hard nproc 10240
    * soft nofile 65536
    * hard nofile 65536
    root soft nproc unlimited

    Reboot to enable the new settings.
  4. Add an unprivileged user “arcsight” to own the application and run as:
    groupadd -g 1000 arcsight
    useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
    passwd arcsight
  5. Ensure the *parent* directory for the Logger software exists. Standard locations for installation of ArcSight products should be /opt/arcsight, so for example, we’re going to install our Logger software at /opt/arcsight/logger.
    cd /opt
    mkdir /opt/arcsight
  6. Run the Logger installation binary as “root” user
    • ./ArcSight-logger-6.2.0.7633.0.bin
  7. After the installation script completes successfully, you should be able to login to the console via a web browser https://<hostname>
    Default username “admin” with default password “password”. You’ll be forced to change the admin password on login.
  8. If you are going to install any SmartConnectors on the system hosting your Logger, check out my post regarding required libraries for CentOS and RedHat, before you try to run the Linux SmartConnector install. This includes any Model Import Connectors (MIC) or forwarding connectors (SuperConnectors).

 

[Update 2016/03/11]: Starting with SmartConnector 7.1.7 (I think, might be a rev or two earlier), there are a couple more libraries that are needed to successfully install the SmartConnector on Linux. Include libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64
yum install libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64

These notes describe an installation of HP ArcSight Logger 6.0.1 on a CentOS 6.5 virtual machine.

For a test install of Logger 6, I built a CentOS vm with the following parameters:
Basic install from the CentOS 6.5 Minimum ISO
1 CPU with 2 cores
4GB memory
80GB virtual disk
1 bridged network adapter
Disk partition sizes:
root fs 6GB, swap 4GB, /home 2GB, /opt/arcsight 50GB, /archive 10GB, free space approximately 15GB

As soon as the system was up, I commented out the archive filesystem (will be re-mounted under the /opt/arcsight/logger directory)
vi /etc/fstab

Installed the bind-utils package so I could use dig and friends, then did a full yum update:
yum install bind-utils ntp
yum update

This turns the system into CentOS 6.6, but that’s still a supported system for Logger, so all’s good.

Next we prepare the system for Logger software install by adding a user and changing some of the system configuration.

Add a non-root user to own and run the Logger application:
groupadd -g 1000 arcsight
useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
passwd arcsight

Install libraries that Logger depends on:
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686
yum install zip unzip

Update the maximum number of processes and open files our Logger processes can have:
cp 90-nproc.conf /etc/security/limits.d/90-nproc.conf

Contents of the /etc/security/limits.d/90-nproc.conf file becomes:
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
*          soft    nproc     10240
*          hard    nproc     10240
*          soft    nofile    65536
*          hard    nofile    65536
root       soft    nproc     unlimited

Turn off services we don’t need and turn on the ones we do need. Later we will write some iptables rules so we can turn the firewall back on when we’re done.

chkconfig iptables off
service iptables stop
chkconfig iscsi off
service iscsi stop
chkconfig iscsid off
service iscsid stop
ntpdate name-of-ntp-server-you-trust
chkconfig ntpd on
service ntpd start

All of these steps are packaged up here in centos-setup.shl:
groupadd -g 1000 arcsight
useradd -u 1000 -g 1000 -d /home/arcsight -m -c "ArcSight" arcsight
passwd arcsight
cp 90-nproc.conf /etc/security/limits.d/90-nproc.conf
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686
yum install zip unzip
chkconfig iptables off
service iptables stop
chkconfig iscsi off
service iscsi stop
chkconfig iscsid off
service iscsid stop
ntpdate 0.centos.pool.ntp.org
chkconfig ntpd on
service ntpd start

Turns out since we need 3+GB of free space in /tmp, I needed to extend the root filesystem .. I only allocated 2GB to begin with. Extend the root logical volume (lv_root) by adding 1,000 Physical Extents (4MB each):

Boot into rescue mode .. do NOT mount linux partitions, then drop to a shell

vgs
vgchange -a y vg_swlogger1
lvextend -l +1000 /dev/vg_swlogger1/lv_root
e2fsck -f /dev/vg_swlogger1/lv_root
resize2fs /dev/vg_swlogger1/lv_root

Now reboot and confirm there is at least 4GB of free space in /tmp. Could also have mounted a ram filesystem, but this will do as I’m conserving my memory on the host.

Upload the Logger installer binary and also the license file to the system into root’s home directory (or where you have space).

As root, run the Logger software install:
chmod u+x ArcSight-logger-6.0.0.7307.1.bin
./ArcSight-logger-6.0.0.7307.1.bin

Word of advice .. if doing this in a vm, run the install from the vm console since it’s possible the vm will be busy enough a remote ssh session could get disconnected – and the install will not complete properly.

After the install, we should be able to open a browser by navigating to https://name-of-vm-here

Sign in as arcsight / password then navigate to the System Administration section to change the admin password.

Outbound network traffic with multiple interfaces

Why does Red Hat Enterprise Linux 6 invalidate / discard packets when the route for outbound traffic differs from the route of incoming traffic?

Issue Description
Why does Red Hat Enterprise Linux 6 invalidate / discard packets when the route for outbound traffic differs from the route of incoming traffic?
Why does Red Hat Enterprise Linux 6 differ from Red Hat Enterprise Linux 5 in handling asymmetrically routed packets?

Solution posted at access.redhat.com/site/solutions/53031

Red Hat Enterprise Linux (RHEL) 6 Resolution

Temporary change
To accept asymmetrically routed (outgoing routes and incoming routes are different) packets set “rp_filter” to 2 and restart networking, by running the following commands:

echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

Persistent change
To make this behaviour persistent across reboots, modify /etc/sysctl.conf and make the following change prior to reboot:

net.ipv4.conf.default.rp_filter = 2

Root Cause

RHEL6 (unlike RHEL5) defaults to using ‘Strict’ Reverse Path Forwarding (RPF) filtering.

Comments
The sysctl net.ipv4.conf.default.rp_filter selects the default RPF filtering setting for IPv4 networking. (It can be overridden per network interface through net.ipv4.interfacename.rp_filter).

Both RHEL6 and RHEL5 ship with a default /etc/sysctl.conf that sets this sysctl to 1, but the meaning of this value is different between the RHEL6 and the RHEL5 kernel.

Libraries needed to install ArcSight SmartConnectors on RedHat Enterprise Linux and CentOS

[Update 2016/03/11]:

Starting with SmartConnector 7.1.7 (I think, might be a rev or two earlier), there are a couple more libraries that are needed to successfully install the SmartConnector on Linux. Include libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64
yum install libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64

[Update 2014/02/04]:
Simpler syntax for the install, using yum to do the automatic dependency processing, and .. a update for CentOS 6.4 64-bit. I believe RHEL 6.4 64-bit would also need these libraries. This worked for installing ArcSight SmartConnector 6.0.7 on CentOS 6.4 64-bit.

glibc.i686
libX11.i686
libXext.i686
libXi.i686
libXtst.i686

You could install like:
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686

[Original post]
While installing an ArcSight SmartConnector 6.0.2 on RedHat Enterprise Linux 6.2 64-bit, the initial install runs successfully, however the connector configuration never kicks off, then install just claims it is done. runagentsetup.sh fails with Error occurred during initialization of VM .. java/lang/NoClassDefFoundError: java/lang/Object .. obviously a pretty major Java error.

Turns out there are some additional libraries that need to be loaded in addition to what is listed in the documentation.

Some research leads me to believe there were some base libraries that may be missing from the vanilla RHEL 6.2 64 bit install. Basic Server + Desktop configuration was selected and all libraries referenced in the ESM 6.0c Install Guide and SmartConnector User Guide were installed. Tracing through all the dependencies created this exact list of of libraries that are required to be installed on RHEL 6.2 64 bit:

glibc-2.12-1.47.el6.i686.rpm
glibc-2.12-1.47.el6.x86_64.rpm
glibc-common-2.12-1.47.el6.x86_64.rpm
libX11-1.3-2.el6.i686.rpm
libX11-1.3-2.el6.x86_64.rpm
libX11-common-1.3-2.el6.noarch.rpm
libXau-1.0.5-1.el6.i686.rpm
libXau-1.0.5-1.el6.x86_64.rpm
libxcb-1.5-1.el6.i686.rpm
libxcb-1.5-1.el6.x86_64.rpm
libXext-1.1-3.el6.i686.rpm
libXext-1.1-3.el6.x86_64.rpm
libXi-1.3-3.el6.i686.rpm
libXi-1.3-3.el6.x86_64.rpm
libXtst-1.0.99.2-3.el6.i686.rpm
libXtst-1.0.99.2-3.el6.x86_64.rpm
nss-softokn-freebl-3.12.9-11.el6.i686.rpm
nss-softokn-freebl-3.12.9-11.el6.x86_64.rpm

Note the specific X libraries versus the generic list as shown in the connector user guide. What was interesting about these is that they did NOT all install when doing a wildcard rpm install, and additionally did not report any failures. After some trial and error, on my system it appears the 32 bit X libraries needed to be installed individually for some reason. You may want to use rpm -q -a to verify each of the libraries successfully installed. Once all the above libraries were installed, the connector installation worked as expected.

A tarball with the libraries can be downloaded from here.

Extract the libraries, change into the resulting directory, then you can use the following brute force syntax to determine which libraries are not installed and install them:

rpm -ivh `ls | while read rpmfile; do rpm -q \`basename $rpmfile .rpm\`; done | egrep 'not installed' | awk '{print $2}' | xargs`

Seagate Disk Replacement and RAID1 mdadm Commands

So I’ve had to replace a Seagate disk yet again and spent a frustrating amount of time on their website looking for a link to their warrenty replacement page >> http://www.seagate.com/support/warranty-and-replacements/

At least this time, I’m using Linux software RAID for a RAID1 mirroring configuration. When I determined there was a disk failure, I used the following mdadm commands to remove the bad drive:

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[0](F) sdb1[2]
5139084 blocks [2/1] [U_]
md1 : active raid1 sda2[0](F) sdb2[2]
9841585344 blocks [2/1] [U_]
unused devices:

– Fail and remove all /dev/sdb partitions (/dev/sdb1, /dev/sdb2)
# mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
# mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1
# mdadm --manage /dev/md1 --fail /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md1
# mdadm --manage /dev/md1 --remove /dev/sdb3
mdadm: hot removed /dev/sdb3

– Shutdown and replace the bad disk (assuming you have been able to replace with the exact disk)
– Copy the partition table from the surviving disk
# sfdisk -d /dev/sda | sfdisk /dev/sdb

– Re-attach the partitions from /dev/sdb to the RAID1 mirrors:
# mdadm --manage /dev/md0 --add /dev/sdb1
# mdadm --manage /dev/md1 --add /dev/sdb2

You should now see the md devices syncing up by:
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda1[1] sdb1[2]
5139084 blocks [2/1] [U_]
[======>.......] recovery = 49.3% ...

Once the sync completes, install grub on both the drives again:
# grub
grub> root (hd0,0)
grub> setup (hd0)
grub> root (hd1,0)
grub> setup (hd1)

Great reference pages:
http://serverfault.com/questions/483141/mdadm-raid-1-grub-only-on-sda
https://blogs.it.ox.ac.uk/jamest/2011/07/26/software-raid1-plus-grub-drive-replacement/

Unix, Linux and Mac OS X Notes

Here’s some notable command syntax I use. You can also select the Notes category and you’ll get more specific topics such as Linux LVM and Mac OS X commands.

rsyslog options

Forward syslog events to external host via UDP:
– edit /etc/rsyslog.conf .. add a stanza like the example at the end of the file .. a single @ = UDP forward, @@ = TCP forward

$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
*.* @10.0.0.45:514

– restart the rsyslog daemon
systemctl restart rsyslog.service
or
service rsyslog restart

Mac OS X syslog to remote syslog server

Forward syslog events on Mac OS X 10.11 to external syslog server via UDP or TCP:
– edit /etc/syslog.conf .. add a line at the end of the file .. a single @ = UDP forward, @@ = TCP forward

*.* @10.0.0.45:514
# remote host is: name or ip:port, e.g. 10.0.0.45:514, port optional

– restart the OS X syslog daemon
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

Write ISO image to USB on Mac

– plug in USB to Mac
– lookup disk number
sudo diskutil list
– unmount the USB
sudo diskutil unmountDisk /dev/disk2
– copy ISO image to USB
sudo dd if=CentOS.iso of=/dev/disk2

NIC MAC change

Changing MAC address of NIC
– RedHat stores this in: /etc/sysconfig
networking/devices/ifcfg-eth?
networking/profiles/default/ifcfg-eth?
hwconf
You need to edit the hwaddr in /etc/sysconfig/hwconf and HWADDR in the other locations (some are links).

ssh tunneling of syslog traffic

– Example SSH configuration for tunneling a syslog TCP stream from a remote server back to a local node:

Remote node has TCP client process (rsyslog) running, we want it to write to a local TCP port (15514/tcp), and have that local port forward to the local node we have initiated the ssh connection from to a syslog daemon listening on port 1514/tcp:

Remote node rsyslog.conf:
@@localhost:15514

Event flow is through ssh on the remote node, listening on 15514/tcp and forwarding to the local node via ssh tunnel launched on the local node:
$ ssh -R 15514:localhost:1514 remotehostusername@remote.hostname.domain

To complete the picture, we probably want some sort of process on the local node to detect when the ssh connection has been lost and (1) re-establish the ssh connection, (2) restart rsyslog on the remote host to re-establish the connection from the remote rsyslog daemon to the ssh listener on port 15514/tcp.

YUM Software Repository

– Manually add DVD location/repository by:

35.3.1.2. Using a Red Hat Enterprise Linux Installation DVD as a Software Repository

To use a Red Hat Enterprise Linux installation DVD as a software repository, either in the form of a physical disc, or in the form of an ISO image file.

1. Create a mount point for the repository:
mkdir -p /path/to/repo

Where /path/to/repo is a location for the repository, for example, /mnt/repo. Mount the DVD on the mount point that you just created. If you are using a physical disc, you need to know the device name of your DVD drive. You can find the names of any CD or DVD drives on your system with the command cat /proc/sys/dev/cdrom/info. The first CD or DVD drive on the system is typically named sr0. When you know the device name, mount the DVD:
mount -r -t iso9660 /dev/device_name /path/to/repo
For example: mount -r -t iso9660 /dev/sr0 /mnt/repo

If you are using an ISO image file of a disc, mount the image file like this:
mount -r -t iso9660 -o loop /path/to/image/file.iso /path/to/repo
For example: mount -r -o loop /home/root/Downloads/RHEL6-Server-i386-DVD.iso /mnt/repo

Note that you can only mount an image file if the storage device that holds the image file is itself mounted. For example, if the image file is stored on a hard drive that is not mounted automatically when the system boots, you must mount the hard drive before you mount an image file stored on that hard drive. Consider a hard drive named /dev/sdb that is not automatically mounted at boot time and which has an image file stored in a directory named Downloads on its first partition:

mkdir /mnt/temp
mount /dev/sdb1 /mnt/temp
mkdir /mnt/repo
mount -r -t iso9660 -o loop mount -r -o loop /mnt/temp/Downloads/RHEL6-Server-i386-DVD.iso /mnt/repo

2. Create a new repo file in the /etc/yum.repos.d/ directory:
The name of the file is not important, as long as it ends in .repo. For example, dvd.repo is an obvious choice. Choose a name for the repo file and open it as a new file with the vi text editor. For example:

vi /etc/yum.repos.d/dvd.repo

[dvd]
baseurl=file:///mnt/repo/Server
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

The name of the repository is specified in square brackets — in this example, [dvd]. The name is not important, but you should choose something that is meaningful and recognizable. The line that specifies the baseurl should contain the path to the mount point that you created previously, suffixed with /Server for a Red Hat Enterprise Linux server installation DVD, or with /Client for a Red Hat Enterprise Linux client installation DVD. NOTE: After installing or upgrading software from the DVD, delete the repo file that you created to get updates from the online sources.

IP Networking

– Manually add IPv4 alias to interface by:
ip addr add 192.168.0.30/24 dev eth4
– Manually remove that IPv4 alias to interface by (note the subnet mask):
ip addr del 192.168.0.30/32 dev eth4
– Manually add route for specific host:
route add -host 45.56.119.201 gw 10.20.1.5

pcap files

– Split large pcap file by using command line tool that comes with Wireshark editcap:
editcap -c 10000 infile.pcap outfile.pcap

tcpdump options

Display only packets with SYN flag set (for host 10.10.1.1 and NOT port 80):
tcpdump 'host 10.10.1.1  &&  tcp[13]&0x02 = 2  &&  !port 80'

Mac OS X (10.7)

sudo /usr/sbin/sysctl -w net.inet.ip.fw.enable=1
sudo /sbin/ipfw -q /etc/firewall.conf
sudo ifconfig en0 lladdr 00:1e:c2:0f:86:10
sudo ifconfig en1 alias 192.168.0.10 netmask 255.255.255.0
sudo ifconfig en1 -alias 192.168.0.10
sudo route add -net 10.2.1.0/24 10.3.1.1

rpm commands:

List files in an rpm file
rpm -qlp package-name.rpm

List files associated with an already installed package
rpm --query –-filesbypkg package-name
How do I find out what rpm provides a file?
yum whatprovides '*bin/grep'
Returns the package that supplies the file, but the repoquery tool (in the yum-utils package) is faster and provides more output as well as do other queries such as listing package contents, dependencies, reverse-dependencies.

sed commands:

Remove specific patterns (delete or remove blank lines):
sed '/^$/d'
sed command matching multiple line pattern (a single log line got split into two lines, the second line beginning with a space):
cat syslog3.txt | sed 'N;s/\n / /' > syslog3a.txt
– matches the end of line (\n) and space at the beginning of the next line, then removes the newline

awk commands:

Print out key value pairs KVP separated by =:
awk /SRC=/ RS=" "
Print out source IP for all iptables entries that contain the keyword recent:
cat /var/log/iptables.log | egrep recent | awk /SRC=/ RS=" " | sort | uniq
Sum column one in a file, giving the average (where NR is the automatically computed number of lines in the file):
./packet_parser analyzer_data.pcap | awk '{print $5}' | sed -e 's/length=//g' | awk 'BEGIN {sum=0} { sum+=$1 } END { print sum/NR }'
Find the number of tabs per line – used to do a sanity check on tab delimited input files
awk -F$'\t' '{print NF-1;}' file | sort -u

sort by some mid-line column

I wanted to sort by the sub-facility message name internal to the dovecot messages, so found the default behavior of sorting by space delimited columns works.

sort -k6 refers to the sixth column with the default delimiter as space.
sort -tx -k1.20,1.25 is an alternative, where ‘x’ is a delimiter character that does not appear anywhere in the line, and character position 20 is the start of the sort key and character position 25 is the end of the sort key.

This sorts by the bold column:
$ sort -k6 dovecot.txt
Oct 7 09:09:31 server1 dovecot: auth: mysql: Connected to 10.30.132.15 (db1)
Oct 7 09:34:03 server1 dovecot: auth: sql(user1@example.com,10.30.132.15): Password mismatch
Oct 7 09:33:36 server1 dovecot: auth: sql(someuser@example.com,10.30.132.15): unknown user
Oct 7 09:15:27 server1 dovecot: imap(user1@example.com): Disconnected for inactivity bytes=946/215256
Oct 7 09:21:11 server1 dovecot: imap(user2@example2.com): Disconnected: Logged out bytes=120/12718

dos2unix equivalent with tr

tr -d '\15\32' < windows-file.csv > unix-file.csv

Fedora 16 biosdevname

– Fedora 16 includes a package called “biosdevname” that sets up strange network port names (p3p1 versus eth0) .. since I don’t particilarly care if my ethernet adapter(s) is(are) in a particular PCI slot, remove this nonsense by:

yum erase biosdevname

– to take total control of network interfaces back over (edit /etc/sysconfig/network-scripts/ifcfg-eth?)

– remove NetworkManager

yum erase NetworkManager
chkconfig network on

WordPress notes for pomeroy.us

Production site is www.pomeroy.us
Development site is dev.pomeroy.us

Assumptions:
– webserver root directory is /var/web
– production node is called prod
– development node is called dev
– WordPress database is called wpdb

Procedure to copy production WordPress instance to the development node:
1. Copy webserver www root dir via a tarball
tar czf prod-20110808.tgz /var/web

2. Dump the WordPress database to a MySQL dmp file:
mysqldump -u$mysqluser -p$mysqlpass wpdb | \
 gzip -c > prod-20110808.dmp.gz

3. Copy these two backup files to the dev node:
scp prod-20110808* user@dev:.

On the development node:
4. Unpack the webserver tarball:
mv /var/web /var/web.previous
cd /
tar xzvf prod-20110808.tgz

5. Drop the WordPress database and restore the new version:
mysql> drop database wpdb;
mysql> create database wpdp;
$ gunzip prod-20110808.dmp.gz
$ mysql -u$mysqluser -p wpdb < prod-20110808.dmp

6. Update the WordPress 'siteurl' and 'home' options to point to the development node:
update wp_options set option_value='http://dev.pomeroy.us' where option_name='siteurl';
update wp_options set option_value='http://dev.pomeroy.us' where option_name='home';

Should be all done!

Building a new PVR

<Updated Aug 18, 2011 after a successful PVR rollout>

Technology has evolved since the last MythTV PVR I built, as chronicled here.  Here’s the latest techniques and tech that I’ve used to (start) build(ing) my current PVR. I’ll update this article as I go, as there’s been some bumps along the way, so completion of the project has been slower than I anticipated.

Requirements for my new PVR include:

  • Linux operating system for cost and flexibility reasons
  • Quiet! Fan-less operation if at all possible, external power supply ok
  • Small form factor, black case to fit in with my current home theater gear
  • Video capture with MPEG-2 hardware acceleration to help keep the CPU needed as small as possible, in an expansion card format for the most compact physical footprint .. additionally there must be at least two independent tuners
  • Analog tuners, but would be good if they were capable of digital for when I eventually move to digital/HD
  • IR receiver and transmitter capability for easy remote control and ability of the PVR to use my current set-top box as a source (gives me all the cable company movies and channels that are not available via the basic cable connection
  • Ability to schedule at least 10 shows and retain 5 episodes of each show .. also ability to schedule based on show name alone
  • Ability to perform post-recording processing, such as removing commercials or changing formats
  • Should be able to use a pre-packaged distribution for most if not all of the functions .. I know it’s a home-brew, but I’m tired of messing with individual packages, firmware, and custom codes to make it work. Using a distribution package makes it easier to maintain through updates.
  • Want to purchase the parts from the same supplier if possible (ended up using newegg.ca)

Since I already run MythTV, it was an obvious starting point and given I don’t have an affinity to a specific Linux distribution, I looked at Mythbuntu and Mythdora since I’m familiar with and already run both Ubuntu and Fedora distributions.

After downloading the Mythbuntu 10.10 ISO disk image, I discovered I didn’t have my USB DVD drive, so I wanted to create a bootable USB flash disk.  I followed the excellent instructions at https://help.ubuntu.com/community/Installation/FromUSBStick and successfully burned a bootable Mythbuntu disk on a 2GB USB flash disk via a Ubuntu VM running on my MacBook Pro.


Continue reading

MySQL Notes

MySQL Command Line and Configuration Notes

Drop tables with wildcard:

There are multiple ways to specify MySQL credentials, this is not the best, but simply an example of how to drop tables using a wildcard pattern. In this case, command line history such as .bash_history will store your MySQL username and password plaintext, and an extended process listing will also reveal both username and password. When run from the command line like this, the SQL commands and the credentials are not stored in the MySQL history file (.mysql_history).  On closed (private) systems, the risk is low, especially if you clean up after these maintenance activities by purging the command histories.

mysql -u user -p password database -e "show tables" | grep "table_pattern_to_drop_" | awk '{print "drop table " $1 ";"}' | mysql -u user -p password database

Resetting WordPress user passwords

Resetting WordPress 3.0 user passwords can be done directly within MySQL through the following procedure.  This assumes your installation of WordPress stores user passwords in the wp_users table as MD5 hashes and the unique site prefix for all WordPress tables in MySQL is _x.

Connect to the database via your favorite GUI (phpMyAdmin, Navicat) or command line with either the WordPress role account or any other MySQL user account with select and update privileges on the WordPress database:

update wp_x_users set user_pass = MD5('123abc890') where user_login = 'administrator';

This will update the password for user ‘administrator’ to ‘123abc890’.  Once this has completed, either flush the wp_x_users table or exit the tool used to access the database to cause the updates to be committed.  Sign into WordPress with the new password and optionally change the password via the user interface.

Accessing Ubuntu desktop from Mac Snow Leopard

Accessing my Ubuntu 9.04 Gnome desktop from the built in Mac OS X 10.6.2 VNC viewer took a bit of tweaking on the Ubuntu Gnome side. I have an OpenVPN SSL tunnel between the Mac and the Ubuntu desktop, however a SSH tunnel could also be used to protect the VNC session. In this post, I’ll just cover the VNC server setup assuming a secure connection between the Mac and the desktop.

Initially I followed the guidance at sanity, inc.”How to OS X Leopard Screen Sharing with Linux“, on Ubuntu I installed tightvnc:

apt-get install tightvncserver

Then tested it out by starting up the vnc server on the Ubuntu system as the user I want to run the remote session as:

tightvncserver -geometry 1024x700 -depth 24 :1

As tightvncserver starts up the VNC service, it will check for a .vncpasswd file in the user home directory. If it doesn’t exist, you will be prompted for a password to use to protect the remote session.  Note VNC is not designed to be used for multi-user remote access.
On the Mac, rather than use Bonjour to automatically discover the Ubuntu screen sharing service, I just referred to the VNC session directly within Finder which invokes the built in VNC viewer. Enter the VNC session password when prompted and the Ubuntu desktop is displayed. connect-to-server Within Finder, either use Go -> Connect to Server or Apple-K to bring up the Connect to Server window.  The server address is the URL that points to the Ubuntu VNC instance vnc://10.10.1.2:5901 where the port is 5900 + the display number specified when starting up the tightvncserver (5901).

This all worked fantastic, except for the keyboard mapping within Gnome – it was scrambled.  After googling several possible solutions, the only one that was successful for me was to disable the keyboard plugin in Gnome

Amit Gurdasani wrote on 2008-04-28: #51

I’ve also encountered this issue with TightVNC and the hardy release. My solution was to capture the xmodmap -pke output as ${HOME}/.Xmodmap at the login screen (DISPLAY=:0 XAUTHORITY=/var/lib/gdm/:0.Xauth sudo xmodmap -pke > ${HOME}/.Xmodmap). When gnome-settings-daemon starts up and finds an .Xmodmap, it asks if it should be loaded — I answer yes. As a side effect, if gnome-settings-daemon were to be restarted without the .Xmodmap, it’d scramble the keyboard layout again. With an .Xmodmap in place, it’ll load the .Xmodmap every time.

Due to another issue (#199245, gnome-settings-daemon crashing with BadWindow every time a window is mapped), I disabled the keyboard plugin using gconf-editor, at /apps/gnome_settings_daemon/plugins/keyboard. Since it’s not being loaded, I suspect it might not garble the layout even if I remove the .Xmodmap now.

So maybe disabling the keyboard plugin is a better fix.

On the Ubuntu system, invoke the Gnome configuration editor (gconf-editor on command line), then navigate to apps -> gnome_settings_daemon -> plugins -> keyboard uncheck the Active keyword.  Kill the VNC daemon and relaunch it – problem fixed.

pkill vnc
tightvncserver -geometry 1024x700 -depth 24 :1

Various methods exist to automatically start and kill the VNC server, but for now this will do it for me.

IMAP mailstore migration .. again

So last weekend, I discovered that Spamhaus decided it would be a good idea to place all of the public IP addresses for Slicehost (my Linux VPS hoster) into their Spamhaus block list (SBL). This covered both my slice in Dallas and the one in St. Louis – meaning an impressive chunk of inbound mail to my domains was being trashed by the sending MTA and an even bigger chunk of my outbound mail was being outright rejected since the sending IP’s were on the SBL.  Slicehost worked hard to convince Spamhaus to recind the blocklist, so the Slicehost IP’s got moved over to the less-nasty-but-you’re-still-probably-a-spamming-dirtbag Policy Block list (PBL) assuming affected IP owners would request to be removed from that list.

Sample query to see if you’re on any Spamhaus block list:  http://www.spamhaus.org/query/bl?ip=10.11.12.13

It seems it’s time to relinquish the care and feeding of my own Postfix mail system and turn to a hosted solution.  This means I need to migrate about 5GB of IMAP store to another site (again).  Last time I did a wholesale migration, I used imapsync to make the transition painless. In the code example below, an SSL connection to the IMAPS server at imap-server.sourcedomain.com is made with username@sourcedomain.com and the password stored in the plaintext file secret1. An SSL connection is made to the target system (which happens to be the server on which the imapsync tool is running, but could just as easily be another IMAPS server somewhere on a network accessible to the host where imapsync is running). The –delete and –expunge1 arguments will clean the successfully moved messages from IMAP store #1 .. so be sure you have your messages on the target successfully! Imapsync can be run iteratively to ensure you have got all the messages from your source.


/usr/bin/imapsync \
--host1 imap-server.sourcedomain.com \
--ssl1 \
--authmech1 LOGIN \
--user1 username@sourcedomain.com --passfile1 secret1 \
--host2 127.0.0.1 --user2 username@targetdomain.com --passfile2 secret2 \
--ssl2 \
--delete --expunge1 \
--buffersize=128

And one can use the

--dry

option to just test the process but not actually move any of the messages.

So that’s it – I’m about half way though migrating my current IMAP stores over to a hosted mail solution, so that I don’t need to keep up with the increasing level of care and feeding that running your own mail service requires.  Before I get too many darts about that .. I first started running my own personal MTA in 1995, adding spam and av filtering over time, and adding substantial redundancy (servers, sites, storage) so I could rely on it and fix things that broke as I had time rather than right when they broke (which was always at a bad time).  My new hosted solution takes over from two VPS servers running Postfix, Spamassassin, ClamAV, Greylisting with the IMAP store replicated across data centers in different states (15 minute rsyncs).  So soon, the (hopefully) last Allen Pomeroy owned and operated MTA can be turned off, while I get to work on fun stuff, rather than figuring out why my email is bouncing.  :-)

Update 2012/12/17:

Sometimes manual manipulation of your mailstore via IMAP is needed, so here’s how I deleted a large number of folders I had trashed and were being synced to my new system from the old.  Kinda clunky, since I didn’t get the scripted version to work (just used a copy/paste in an interactive bash session), but got the job done for now.

Connect to the IMAP server using SSL:
openssl s_client -crlf -connect imap.emailsrvr.com:993

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Server ready director6.mail.ord1a.rsapps.net

Log in with your email credentials:
0 login user@domain.com Password

0 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS QUOTA] Logged in

List the folders you want to remove:
0 list "" "Trash.*"

That didn’t return the list I was expecting, so I listed all folders
0 list "" "*"

… and realized the source mail system adds “INBOX” on the front of the folder names, so then this command worked to list the folders to be deleted:
0 list "" "INBOX.Trash.*"

I copied the output and edited it to insert the folder name into a delete command:
0 delete "INBOX.Trash.Folder1"
0 delete "INBOX.Trash.Folder2"
0 delete "INBOX.Trash.Folder3"

0 OK Delete completed.
0 OK Delete completed.
0 OK Delete completed.

Finish off the session by logging out:
0 logout

* BYE Logging out
0 OK Logout completed.
closed

Building a web security lab (with VMware Fusion)

Problem: VMware machines load boot loader immediately, no BIOS banner, so can’t get into BIOS to alter boot settings.
Solution: Edit the vm’s .vmx file and add the line:

bios.bootDelay = "5000"

which adds a 5000 millisecond (5 second) delay to the boot, or add:

bios.forceSetupOnce = "TRUE"

to make the VM enter the BIOS setup at the next boot.

Problem: VMware Fusion 3.0 doesn’t give a way to edit the virtual network settings via the GUI.
Solution: To change the subnet used by the NAT or HostOnly networks, go root in Mac OS X and edit

/Library/Application Support/VMware Fusion/networking

and set the following lines to the subnets desired:

answer VNET_1_HOSTONLY_SUBNET 192.168.35.0
answer VNET_8_HOSTONLY_SUBNET 10.10.1.0

To add additional custom isolated host only VLANs, also edit the networking file and add additional VNET definitions. There can apparently only be 8 VLANs with VLAN 1 and 8 already pre-defined.

answer VNET_2_DHCP no
answer VNET_2_HOSTONLY_NETMASK 255.255.255.0
answer VNET_2_HOSTONLY_SUBNET 10.10.21.0
answer VNET_2_VIRTUAL_ADAPTER yes
answer VNET_3_DHCP no
answer VNET_3_HOSTONLY_NETMASK 255.255.255.0
answer VNET_3_HOSTONLY_SUBNET 10.10.22.0
answer VNET_3_VIRTUAL_ADAPTER yes
answer VNET_4_DHCP no
answer VNET_4_HOSTONLY_NETMASK 255.255.255.0
answer VNET_4_HOSTONLY_SUBNET 10.10.23.0
answer VNET_4_VIRTUAL_ADAPTER yes

Now create your vm with as many network interfaces as you have separate VLANs (vnet) then edit the node.vmx vm configuration file and change the interfacename.connectionType to custom, and define the VLAN (vnet) that interface will attach to:

#ethernet0.connectionType = "nat"
ethernet0.connectionType = "custom"
ethernet0.vnet = "vmnet3"

Also realize that VMware will take the .1 host address on each vmnet – so you cannot assign .1 to any of your VMs.

Problem: Ubuntu 9.10 persistent network configuration (stores the MAC address of network adapters), so if you copy a machine, by default Ubuntu will setup a new logical adapter (eth1) since the MAC address has changed (when you answer I Copied It in VMware).
Solution: Tell VMware you copied the machine, so it will chose a unique MAC address. Boot Ubuntu into single user mode (another article on that to follow) then edit the MAC address associated with eth0.

sudo vi /etc/udev/rules.d/70-persistent-net.rules

find the stanza of the network interface in question (NAME=”eth0″) and set the following ATTR tag to the new MAC address:

ATTR{address}=="new-mac-address-here"

Sifting through Checkpoint FW1 logs

Recently I found myself in the unhappy position of needing to sift through slightly more than a billion Checkpoint Firewall-1 log lines, looking for specific patterns of access. The problem was that many of the exported fwm log files had differing column positions and there had been many ruleset changes over the course of 11 months worth of log data. Many of the excellent FW1 log summarization tools (such as Peter Sundstrom’s fwlogsum) didn’t handle the hundreds of files and differing column positions.

The final scripted solution was processing over 11,000 lines/second .. and still took over 23 hours for the first run.

Log file exports via fwm logexport can have variable column positioning, except for record ID number “num”, which is *always* column number one.  I see three viable alternatives to the changing column position in the ASCII log files exported via fwm – so we can automate the log processing:

  • Export the FW1 log file to ASCII via
    fwm logexport -i fw1-binary-logfile -o fw1-ascii-logfile.txt -n -p
    1. Parse the header line (line #1) of every log file and dynamically map (rearrange) the columns to a pre-determined standard in memory before further processing (painful, expensive)
    2. Tell Checkpoint fwm to export in a fixed column ordering
        create
        logexport.ini
        and place in
        $FWDIR/conf directory
        eg. fwmgmtsrv:
        C:\WINDOWS\FW1\R65\FW1\conf
        logexport.ini:
        [Fields_Info]
        included_fields = num,date,time,orig,origin_id,type,action,alert,i/f_name,
        i/f_dir,product,rule,src,dst,proto,service,s_port,xlatesrc,xlatedst,
        nat_rulenum,nat_addtnl_rulenum,xlatesport,xlatedport,user,
        partner,community,session_id,ipv6_src,ipv6_dst,
        srckeyid,dstkeyid,CookieI,CookieR,msgid,elapsed,
        bytes,packets,start_time,snid,ua_snid,d_name,id_src,ua_operation,
        sso_type_desc,app_name,auth_domain,uname4domain,wa_headers,
        result_desc,r_dest,comment,url,redirect_url,enc_desc,e2e_enc_desc,
        auth_result,attack,log_sys_message,
        rule_uid,rule_name,service_id,resource,reason,cat_server,
        dstname,SOAP Method,category,ICMP,message_info,
        TCP flags,rpc_prog,Total logs,
        Suppressed logs,DCE-RPC Interface UUID,Packet info,
        message,ip_id,ip_len,ip_offset,fragments_dropped,during_sec
    3. Use OPSEC LEA tools to extract event log records instead of export via fwm logexport

    Once the ASCII log files are available for processing, my fw1logsearch.pl script can be used to find complex patterns of interest.  Any matching records found by fw1logsearch will be output with an initial FW1 header line so that fw1logsearch can be used iteratively, to build very complex search criteria.  fw1logsearch can also write out a discard file allowing completely negative logic searches resulting in 100% of the input data separated into a match file and a didn’t match file.  Some examples of how I’ve used it are shown here:

    gunzip -c fwlogs/2009*gz | \
    fw1logsearch.pl --allinclude \
    -S '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
    -d '10\.1\.1[1359]\.|10\.2\.1[01]\.|192\.168\.2[245]\.' \
    -p '^1310$|^1411$|^1812$|^455' | \
    fw1logsearch.pl -S '192\.168\.22\.14$|10\.2\.11\.12$' |\
    fw1logsearch.pl --allexclude \
    -S '^192\.168\.24\.12$' -P '^1310$' --rejectfile 192-168-24-12-port-1310.txt

    Line by line:
    1. Unzip the compressed ASCII log files, feed them to the first instance of fw1logsearch.pl
    2. First fw1logsearch – all conditions must be true for any events to match
    Source address must NOT be in any of the following regex ranges:
    10.1.11.* 10.1.13.* 10.1.15.* 10.1.19.*
    10.2.10.* 10.2.11.*
    192.168.22.* 192.168.24.* 192.168.25.*
    Destination address must be in one of the same following regex ranges.
    Service (destination port) must be one of:
    Exactly port: 1310, 1411, 1812, or any port starting with 455
    No protocol is specified, so it will match either TCP or UDP

    fw1logsearch.pl will output any matching events to stdout, including a FW1 log header line, so the next instance of fw1logsearch.pl continues filtering the result set.

    3. The second fw1logsearch.pl specifies Source Address must not be any of the following
    192.168.22.14

    10.2.11.12

    4. The last fw1logsearch.pl excludes port 1310 from 192.168.24.12, and puts all those records into a separate reject file, while writing the other records to stdout.

    This script has been used to process over 4 billion records within the project I wrote it for – and precisely found all the use of particular business cases I needed to modify.  The result was zero outages and no unintended business interruption.

    Basic syntax/help file:

    Usage:  fw1logsearch.pl
    [-a|–incaction|-A|–excaction <action regex>]
    [-p|–incservice|-P|–excservice <dst port regex>]
    [-b|–incs_port|-B|–excs_port <src port regex>]
    [-s|–incsrc|-S|–excsrc <src regex>]
    [-d|–incdst|-D|–excdst <dst regex>]
    [-o|–incorig|-O|–excorig <fw regex>]
    [-r|–incrule|-R|–excrule <rule-number regex>]
    [-t|–incproto|-T|–excproto <proto regex>]

    [–dnscache <dns-cache-file>]
    [–resolveip]
    [–allinclude]
    [–allexclude]
    [–rejectfile <file>]
    [–debug <level>]

    fw1logsearch.pl will search a fwm logexport text file for regex patterns specified for supported columns (such as service, src, dst, rule, action, proto and orig).

    Include and exclude regex matches may be specified on the same line, although they both will include (print) a line or exclude (reject) a line based on single matches.  Allinclude or Allexclude must be specified to force a match
    only on all specified column regex patterns.

    Regex patterns can be enclosed with single quotes to include characters that are special to the shell, such as the ‘or’ (|) operator.

    Header will be output only if there are any matching lines.

    Example invocations:
    $ cat 2008-07-07*txt | \
    fw1logsearch.pl \
    -p ’53|domain’ \
    -d ‘192.168.1.2|host1|10.10.1.2|host2’ \
    -o ‘192.168.2.3|10.10.2.4|10.10.4.5’ \
    -S ‘64.65.66.67|32.33.34.35|10.10.*|192.168.*’ \
    –resolveip
    Will require destination port (service) to be 53, destination IP to be any of 192.168.1.2, host1, 10.10.1.2, or host2  the reporting firewall (origin) to be any of 192.168.2.3, 10.10.2.4, or 10.10.4.5  and the source IP must not be
    any of 64.65.66.67, 32.33.34.35, 10.10.*, or 192.168.*  Any lines that match this criteria, will display and the orig, src, and dst columns will use the default DNS cache file (dynamically built/managed) to perform name resolution, replacing the IP addresses where possible.

    Include regex patterns:
    -a  –incaction    Rule action (accept, deny)
    -b  –incs_port    Source port (s_port)
    -p  –incservice   Destination port (service)
    -s  –incsrc       Source IP|hostname
    -d  –incdst       Destination IP|hostname
    -o  –incorig      Reporting FW IP|hostname
    -r  –incrule      Rule number that triggered entry
    -t  –incproto     Protocol of connection

    Exclude regex patterns:
    -A  –excaction    Rule action (accept, deny)
    -B  –excs_port    Source port (s_port)
    -P  –excservice   Destination port (service)
    -S  –excsrc       Source IP|hostname
    -D  –excdst       Destination IP|hostname
    -O  –excorig      Reporting FW IP|hostname
    -R  –excrule      Rule number that triggered entry
    -T  –excproto     Protocol of connection

    Other options:
    –debug {level} Turn on debugging
    –dnscache      Specify location of DNS cache file to be used with
    the Resolve IPs option
    –resolveip     Resolve IPs for orig, src, and dst columns AFTER filtering
    –rejectfile    Write out all rejected lines to a specified file

    Download fw1logsearch.pl

    How to build a MythTV PVR on Fedora Core 7

    <Notes In Progress – many of these steps have been automated in scripts, I’m in the process of updating this doc to show those steps and include the scripts>

    Fresh install of OS and MythTV on n43

    Created 2007/09/05 – last revised 2008/01/06

    I needed to upgrade MythTV to 0.20.2 due to the demise of Zap2It schedules, but I didn’t have another system which matched the hardware used on my MythTV PVR.  So I installed a spare 120GB EIDE and started from scratch to build another MythTV instance from scratch.  Once I was satisfied the new instance would pass the SAT (spouse approval test), I used LVM to move everything over to the 320GB SATA disk which currently contains the old (production) MythTV sw and configuration.

    This describes that build and migration process. 

    Hardware (n43):

    Antec Fusion HTPC case

    antec-fusion

    AMD Sempron processor (about 1.6GHz)
    512MB memory – good enough for single tuner and OS
    Hauppage PVR-350 standard definition capture card

    Integrated on to mainboard:
    Audio:
    ALC883 PCM nVidia MCP51 controller – kernel module snd-hda-intel (high definition audio)
    Video:
    nVidia C51 – Quadro NVS 210S / GeForce 6150LE
    nVidia EIDE and SATA controller – …

    1. Install OS

    Seems to be lots of hits on Fedora 7 and MythTV, as well the reading I’ve done on Fedora 7 seems to show it can be easily kept up to date (via yum) – and it has the OS clustering capabilities as part of the base now, which I’ll use when I split the current single system making it the back end and adding a silent (diskless) front end.

    Downloaded and burnt Fedora 7 i386 DVD. For future options, extracted and burnt boot.iso … also see notes on how to install Fedora 7 via boot.iso and NFS. (notes to be added)

    Disabled SATA hdd in BIOS (could have unplugged it, but easier to just disable via sw).
    Used DVD drive in n43 to install Fedora 7 on the temporary EIDE drive.
    Select packages for install:
    MySQL Server
    Web Server

    See scripts for automated Fedora 7 OS setup and package install (setup1.shl, setup2.shl) (scripts to be added)

    About 25 min off DVD for base load

    Setup (first time boot):
    Firewall – allow SSH and HTTP, otherwise no inbound services other than ESTABLISHED,RELATED are needed at this point. Will open MySQL and ICMP for monitoring purposes later. When this system becomes the MythTV backend, will have to add MythTV ports (see FAQ).
    SELinux – disable, will add SE configuration at some future point.
    NTP – use default Fedora 7 NTP service configuration, time sync is obviously very important (unless you don’t want your recordings to start/end at the right times).

    MythTV seems to heavily use KDE, so although Gnome is default, may need to use KDE. I selected Gnome this time. And KDE this time. And RatPoison is a compact window manager which may be easier to configure for mythtv. Finally I’m using fvwm2 .. more on that later.

    Update /etc/hosts
    192.168.2.143 mythtv.networkforensics.org mythtv n43

    Manually set the interface speed/duplex (gigabit interface doesn’t do well in autonegotiate – poor performance, but no interface errors). Will come back and setup an init script.
    # ethtool -s eth0 speed 100 duplex full

    Add ATrpms repository into yum configuration:
    NOTE: other ATrpm yum configurations on the net don’t work!
    – add the following into /etc/yum.conf
    [atrpms]
    name=Fedora Core $releasever – $basearch – ATrpms
    baseurl=http://dl.atrpms.net/f$releasever-$basearch/atrpms/stable
    gpgkey=http://ATrpms.net/RPM-GPG-KEY.atrpms
    gpgcheck=1

    Import ATrpms key
    # rpm –import http://ATrpms.net/RPM-GPG-KEY.atrpms

    Disable ATrpms repository, so we only get mythtv packages from it:
    – add the following to the atrpms section just added to the /etc/yum.conf:
    enabled=0

    Update the packages to current using the standard Fedora repositories

     

    When following the Fedora / MythTV HOWTO (http://wilsonet.com/mythtv), they use variable KVER which is just uname -r

     

    # echo “export KVER=\\`uname -r\\`” >> /etc/profile.d/kver.sh

     

    Do yum upgrade to get latest kernel and system

    # yum upgrade

    464MB 265 packages

     

    <odd>

    Kernel panic during reboot after upgrade

    Searched on “2.6.22 fedora 7 kernel panic noapic” – lots of suggestions but doing a single boot again, while interrupting the grub boot and adding noapic to the end of the kernel boot line seemed to fix it. Now the grub.conf looks like (note the vga=791 arg):

    default=0
    timeout=5
    splashimage=(hd0,0)/grub/splash.xpm.gz
    hiddenmenu
    title Fedora (2.6.22.4-65.fc7)
    root (hd0,0)
    kernel /vmlinuz-2.6.22.4-65.fc7 ro root=/dev/VolGroup00/LogVol00 rhgb quiet vga=791
    initrd /initrd-2.6.22.4-65.fc7.img
    title Fedora (2.6.21-1.3194.fc7)
    root (hd0,0)
    kernel /vmlinuz-2.6.21-1.3194.fc7 ro root=/dev/VolGroup00/LogVol00 rhgb quiet vga=791
    initrd /initrd-2.6.21-1.3194.fc7.img

     

    Add window manager fvwm here – comes from Fed
    ora repo

    # yum install fvwm2

     

     

    2. No mouse cursor in Gnome

    Seems the nVidia graphics are broken somehow. Must disable hardware cursor?

     

    <URL>

    ok add this line to your xorg.conf

    Option “HWCursor” “off”

    so it looks something like this

    Code:

    Section "Device"
      BoardName    "GeForce 6600/GeForce 6600 GT"
      BusID        "1:0:0"
      Driver       "nvidia"
      Identifier   "Device[0]"
      Option       "HWCursor" "off"
      Screen       0
      VendorName   "NVidia"
    EndSection

    logged out from Gnome, causes an X restart then the mouse cursor showed up properly.

     

     

    3. Set Monitor and Resolution

    In gnome, I manually set the monitor, it could not autodetect the Viewsonic Optiquest V75. Restarted X, had to set the resolution (it defaulted to way too high of a setting)

     

    HOLD:

    As per Fedora MythTV setup guide, install the nVidia drivers:

    # yum -y install nvidia-graphics9755-kmdl-$KVER
    # yum -y install nvidia-graphics9755-libs nvidia-graphics9755

     

    Actually, this is done now by copying in a revised xorg.conf

    [root@mythtv grub]# cat /etc/X11/xorg.conf.1024×768-monitor-only-V75-BEST

    # Xorg configuration created by system-config-display
    Section “ServerLayout”
    Identifier “single head configuration”
    Screen 0 “Screen0” 0 0
    InputDevice “Keyboard0” “CoreKeyboard”
    EndSection
    Section “InputDevice”
    Identifier “Keyboard0”
    Driver “kbd”
    #Option “XkbModel” “pc105”
    Option “XkbModel” “pc101”
    Option “XkbLayout” “us”
    EndSection
    Section “Monitor”
    Identifier “Monitor0”
    ModelName “Monitor 1280×1024”
    HorizSync 31.5 – 79.0
    VertRefresh 50.0 – 90.0
    Option “dpms”
    EndSection
    Section “Device”
    Identifier “Videocard0”
    Driver “nv”
    Option “HWCursor” “off”
    EndSection
    Section “Screen”
    Identifier “Screen0”
    Device “Videocard0”
    Monitor “Monitor0”
    DefaultDepth 24
    SubSection “Display”
    Viewport 0 0
    Depth 24
    Modes “1024×768” “800×600” “720×400” “640×480” “640×400” “640×350”
    EndSubSection
    EndSection


    Turn off screen saver

    Disable unnecessary services

    services=”avahi pcps bluetooth”
    for service in $services; do chkconfig $service off; service $service stop; done

    avahi – DNS service discovery
    pcps – smart card daemon
    bluetooth

     

    4. Set mysql to start on boot

    chkconfig mysqld on

     

    5. Start up MySQL

    service mysqld start

     

    6. Set MySQL root password

    mysql –uroot

    mysql> grant all on *.* to root@localhost identified by “rootpassword” with grant option;

    mysql> grant all on *.* to root@n43 identified by ‘rootpassword’ with grant option;

     

    7. Create MythTV database

    Test out mysql connection, user, password

    # mysql -uroot -prootpassword

    mysql>

     

    Run MythTV database setup

    # mysql -uroot -prootpassword < /usr/share/mythtv/sql/mc.sql

     

    /usr/share/mythtv/sql/mc.sql:
    CREATE DATABASE if not exists mythconverg;
    GRANT ALL ON mythconverg.* TO mythtv@localhost IDENTIFIED BY “mythtv”;
    FLUSH PRIVILEGES;
    GRANT CREATE TEMPORARY TABLES ON mythconverg.* TO mythtv@localhost
    IDENTIFIED BY “mythtv”;
    FLUSH PRIVILEGES;
    ALTER DATABASE mythconverg DEFAULT CHARACTER SET latin1;

    8. Install MythTV Suite

     

    # yum –enable=atrpms install mythtv-suite

    122 packages, 105MB

     

    Create directory for recordings

    # mkdir /storage/recordings

    # chown mythtv:mythtv /storage/recordings

     

    9.
    Install ivtv drivers and firmware for PVR-350

    # yum –enable=atrpms install ivtv-firmware
    # yum –enable=atrpms install ivtv-kmdl-$KVER

     

     

    10. Update modprobe.conf to enable TV Out on PVR-350

     

    # load ivtv-fb for PVR-350 output
    install ivtv /sbin/modprobe –ignore-install ivtv; /sbin/modprobe ivtv-fb

     

    Manually load ivtv

    # /sbin/depmod -a
    # /sbin/modprobe ivtv

     

    Manually tried to load ivtv-fb – segfaulted … see the part of the howto on modifying grub boot loader…

     

    We’re going to make little modification to the kernel boot line in your grub.conf file that should force the ivtv frame buffer to load on /dev/fb1, as well as allow the ivtv-fb module to be loaded and unloaded. Without doing this, unloading the ivtv-fb module would probably crash your system. To the end of all ‘kernel /vmlinuz…’ lines in /boot/grub/grub.conf, append ‘vga=791’, then reboot your system. This tells the kernel to load a frame buffer for your video card at 1024×768, 16-bit color. I use this all the time myself, simply so I can see more when I’m not in X. I’d always done this on my 350-equipped box without even thinking about it, which could explain some of why I’ve not run into some of the problems other folks have…

     

    Note video device:

    [root@mythtv ~]# ls -l /dev/video*

    lrwxrwxrwx 1 root root 6 2007-09-09 18:17 /dev/video -> video0

    crw——- 1 root root 81, 0 2007-09-09 18:17 /dev/video0

    crw——- 1 root root 81, 16 2007-09-09 18:17 /dev/video16

    crw——- 1 root root 81, 24 2007-09-09 18:17 /dev/video24

    crw——- 1 root root 81, 32 2007-09-09 18:17 /dev/video32

    crw——- 1 root root 81, 48 2007-09-09 18:17 /dev/video48

    [root@mythtv ~]#

     

    From dmesg:

    ivtv0: Registered device video0 for encoder MPEG (4 MB)

    ivtv0: Registered device video32 for encoder YUV (2 MB)

    ivtv0: Registered device vbi0 for encoder VBI (1 MB)

    ivtv0: Registered device video24 for encoder PCM audio (1 MB)

    ivtv0: Registered device radio0 for encoder radio

    ivtv0: Registered device video16 for decoder MPEG (1 MB)

    ivtv0: Registered device vbi8 for decoder VBI (1 MB)

    ivtv0: Registered device vbi16 for decoder VOUT

    ivtv0: Registered device video48 for decoder YUV (1 MB)

     

     

    11. Test out PVR-350 TV Out

    As per https://help.ubuntu.com/community/MythTV_Edgy_hardware_pvr-350_TV-out

     

    Try to display the TV test pattern by putting the saa7127 module into test mode:

    # /sbin/rmmod saa7127
    # /sbin/modprobe saa7127 test_image=1

     

    Works!

    Resume normal operation:

    # rmmod saa7127
    # modprobe saa7127

     

     

    Test video capture

    # /usr/bin/v4l2-ctl -i 0

     

     

    12. Manually compile ivtv module for X

    Had to manually compile ivtv driver for x to enable tv out .. due to some 2.6.22 issue.

     

    As per README in ivtv x driver package – must install xorg sdk to allow compile:

    # yum install xorg-x11-server-sdk

     

    Then compile the new ivtv xdriver:

    # sh ./configure

    # make

    # make install

     

    Copy into xorg directory:

    # cp /usr/local/lib/xorg/modules/drivers/ivtv_drv.so /usr/lib/xorg/modules/drivers

    if gdm failed, ps -ef , then kill it to restart

     

    copy in new xorg.conf (with TV Out section) and do <ctrl><alt><backspace> to restart x server

    [root@mythtv ~]# cat /etc/X11/xorg.conf.tvout

    # XFree86 4 configuration created by pyxf86config

    Section “ServerLayout”

    Identifier “Default Layout”

    Screen 0 “Screen0” 0 0

    InputDevice “Mouse0” “CorePointer”

    InputDevice “Keyboard0” “CoreKeyboard”

    EndSection

    Section “Files”

    # RgbPath is the location of the RGB database. Note, this is the name of the

    # file minus the extension (like “.txt” or “.db”). There is normally

    # no need to change the default.

    # Multiple FontPath entries are allowed (they are concatenated together)

    # By default, Red Hat 6.0 and later now use a font server independent of

    # the X server to render fonts.

    RgbPath “/usr/X11R6/lib/X11/rgb”

    # ModulePath “/usr/X11R6/lib/modules/extensions/nvidia”

    # ModulePath “/usr/X11R6/lib/modules/extensions”

    # ModulePath “/usr/X11R6/lib/modules”

    FontPath “unix/:7100”

    EndSection

    Section “Module”

    Load “dbe”

    Load “extmod”

    Load “fbdevhw”

    Load “glx”

    Load “record”

    Load “freetype”

    Load “type1”

    EndSection

    Section “InputDevice”

    # Specify which keyboard LEDs can be user-controlled (eg, with xset(1))

    # Option “Xleds” “1 2 3”

    # To disable the XKEYBOARD extension, uncomment XkbDisable.

    # Option “XkbDisable”

    # To customise the XKB settings to suit your keyboard, modify the

    # lines below (which are the defaults). For example, for a non-U.S.

    # keyboard, you will probably want to use:

    # Option “XkbModel” “pc102”

    # If you have a US Microsoft Natural keyboard, you can use:

    # Option “XkbModel” “microsoft”

    #

    # Then to change the language, change the Layout setting.

    # For example, a german layout can be obtained with:

    # Option “XkbLayout” “de”

    # or:

    # Option “XkbLayout” “de”

    # Option “XkbVariant” “nodeadkeys”

    #

    # If you’d like to switch the positions of your capslock and

    # control keys, use:

    # Option “XkbOptions” “ctrl:swapcaps”

    # Or if you just want both to be control, use:

    # Option “XkbOptions” “ctrl:nocaps”

    #

    Identifier “Keyboard0”

    Driver “keyboard”

    Option “XkbRules” “xfree86”

    #Option “XkbModel” “pc105”

    Option “XkbModel” “pc101”

    Option “XkbLayout” “us”

    EndSection

    Section “InputDevice”

    Identifier “Mouse0”

    Driver “mouse”

    Option “Protocol” “IMPS/2”

    Option “Device” “/dev/input/mice”

    Option “ZAxisMapping” “4 5”

    Option “Emulate3Buttons” “no”

    EndSection

    Section “InputDevice”

    # If the normal CorePointer mouse is not a USB mouse then

    # this input device can be used in AlwaysCore mode to let you

    # also use USB mice at the same time.

    Identifier “DevInputMice”

    Driver “mouse”

    Option “Protocol” “IMPS/2”

    Option “Device” “/dev/input/mice”

    Option “ZAxisMapping” “4 5”

    Option “Emulate3Buttons” “no”

    EndSection

    Section “Monitor”

    Identifier “NTSC Monitor”

    HorizSync 30-68

    VertRefresh 50-120

    Mode “720×480”

    # D: 34.563 MHz, H: 37.244 kHz, V: 73.897 Hz

    DotClock 34.564

    HTimings 720 752 840 928

    VTimings 480 484 488 504

    Flags “-HSync” “-VSync”

    EndMode

    EndSection

    Section “Device”

    Identifier “Hauppauge PVR 350 iTVC15 Framebuffer”

    #Driver “ivtvdev”

    # 2007/09/09 ACP – changed to ivtv

    Driver “ivtv”

    ### change fb1 to whatever your card grabbed

    Option “fbdev” “/dev/fb1”

    Option “ivtv” “/dev/fb1”

    ### change the BusID to whatever is reported by lspci,

    ### converted from hex to decimal

    BusID “PCI:4:6:0” # lspci says 00:08.0

    ### More examples

    #BusID “PCI:0:10:0” # lspci says 00:0a.0

    #BusID “PCI:1:14:0” # lspci says 01:0e.0

    #BusID “PCI:0:5:1” # lspci says 00:05.1

    EndSection

    Section “Screen”

    Identifier “Screen0”

    Device “Hauppauge PVR 350 iTVC15 Framebuffer”

    Monitor “NTSC Monitor”

    DefaultDepth 24

    DefaultFbbpp 32

    Subsection “Display”

    Depth 24

    FbBpp 32

    Modes “720×480”

    EndSubsection

    EndSection

    Section “DRI”

    Group 0

    Mode 0666

    EndSection

     

     

    13. Run mythtv-setup

     

    Blank menu ..

    Check out http://www.gossamer-threads.com/lists/mythtv/users/286856

    I found this simple set of directions to add in a base set of true

    type fonts (which includes Ariel) to Fedora 7 and it solved the problem.

     

    1. Open a Terminal and cd to a directory you can work in

    2. Become root

    3. Download the MS Core Fonts Smart Package File

    wget http://corefonts.sourceforge.net/msttcorefonts-2.0-1.spec

    4. Make sure that the rpm-build and cabextract packages are installed

    yum install rpm-build cabextract

    5. Build the Core Fonts package:

    rpmbuild -ba msttcorefonts-2.0-1.spec

    6. Install the Core Fonts package

    rpm -Uvh /usr/src/redhat/RPMS/noarch/msttcorefonts-2.0-1.noarch.rpm

     

    The web site I found this one was: http://www.fedorafaq.org/#installfonts

     

    Sign in as mythtv

    $ mythtv-setup

     

    Setup Video Capture cards, listing source, channel setup

     

    $ mythfilldatabase

     

    Setup remote control

    Go get the LIRC packages from ATrpms

    # yum –enable=atrpms install lirc-0.8.3

     

    And get the kernel modules:

    # yum –enable=atrpms install lirc-kmdl-2.6.22.4-65.fc7-0.8.3-70_cvs20070827.fc7

    resulted in:

    Installed: lirc-kmdl-2.6.22.4-65.fc7.i686 0:0.8.3-70_cvs20070827.fc7

    Dependency Installed: lirc-devices.noarch 0:0.8-4.fc7

     

    Manual test:

    # modprobe lirc_mceusb2

     

    Update modprobe.conf to load LIRC

     

     

    /dev/lirc was symlinked to /dev/lirc/0 (which is the PVR-350 …) so re-linked to /dev/lirc/1 and restarted lircd

    # service lircd restart

    then irw showed button presses!

    Updated /etc/init.d/lircd to relink the /dev/lirc symlink to /dev/lirc/1

     

     

     

    Get MythTV to use remote:

    Copy lircrc file into ~mythtv/.mythtv/lircrc and ~mythtv/lircrc (no dots)

     

     

    Install lirc kernel modules

    NO yum –enable atrpms install lirc-kmdl-2.6.22.4-65.fc7

    Had to uninstall all the modules I installed then re-install lirc 0.8.3 from atrpms

     

     

    Page on the Microsoft MCE remote: http://www.mythtv.org/wiki/index.php/MCE_Remote

     

     

    Setup (tune) screen size

     

    Overscan (image off the screen)

    http://www.mythtv.org/wiki/index.php/Overscan

     

    on Toshiba tv (pixels):

    width 632

    height 436

    GUI x offset 36

    GUI y offset 16

     

     

    Setup auto-login, auto-start of mythfrontend

     

    URL http://www.mythtv.org/wiki/index.php/Frontend_Auto_Login

     

    Tried ratpoison, Gnome and KDE – ratpoison I couldn’t get working without troubleshooting and Gnome and KDE are too heavy weight. fvwm works well, although the font sizes are a bit small – haven’t found where to adjust them yet.

     

    Use fvwm window manager:

     

    Add to inittab:

    c7:12345:respawn:/sbin/mingetty --autologin=mythtv tty7

     

    ~mythtv/.bash_profile

    if [ -z "$DISPLAY" ] && [ $(tty) == /dev/tty7 ]; then
    while [ 1 == 1 ]
         do
              startx
              sleep 10
         done
    fi

     

     

     

    HOLD

     

    Use ratpoison window manager:

     

    Add to inittab:

    c7:12345:respawn:/sbin/mingetty --autologin=mythtv tty7

     

    ~mythtv/.bash_profile

    if [ -z "$DISPLAY" ] && [ $(tty) == /dev/tty7 ]; then
    while [ 1 == 1 ]
         do
              startx
              sleep 10
         done
    fi

     

    .xinitrc:

    xset -dpms s off
    xsetroot -solid black
    ratpoison &
    x11vnc -many -q -bg -rfbauth .vnc/passwd
    mythfrontend > /home/mythtv/mythfrontend.log 2>&1
    for i in 5 4 3 2 1 ; do
      if [ -f mythfrontend.log.$i ]; then
        mv -f mythfrontend.log.$i  mythfrontend.log.$(($i + 1))
      fi
    done
    mv mythfrontend.log  mythfrontend.log.1

     

    .ratpoisonrc:

    # This is a sample .ratpoisonrc file
    #
    # Set the prefix key to that of screen's default
    escape C-a
     
    # put something informative on the screen while we load stuff
    exec xloadimage -onroot -quiet -center /home/mythtv/.mythtv/mythtvstart.jpg
     
    # Gets rid of that ugly crosshairs default cursor and set the background to black
    exec xsetroot -cursor_name left_ptr
     
    # Use the name of the program rather than the title in the window list
    defwinname name
     
    ### fire up an xterm with ctrl-A x
    bind x exec xterm -j -fn '*-courier-*-r-*-14-*'
     
    # Since running a 720x576 definition the ratpoison screens are too big for the
    # display so we reduce the size of them with defpadding to make them fit
    #defpadding 25 25 25 25
     
    keystate_numlock = enable

     

     

     

     

     

    KDE application file into ~mythtv/.kde/Autostart

     

    Had to setup desktop for mythtv (all black, no screen saver)

     

     

    Migrate from 120GB disk back to 320GB SATA

    2007/10/20 Fedora 7

    Use LVM to move the data, including /root, swap and /storage

     

    In BIOS, enable SATA drive, position as HDD #2 (120GB IDE as HDD #1)

     

    Boot single user (interrupt grub, select first kernel <e>dit, select kernel spec line, <e>dit, add “single” on the end of the line, <b>oot the system (off the old IDE disk)

     

    Display partition table for both drives just to be sure that the 320GB (new) disk is /dev/sbd and the current ‘production’ IDE disk is /dev/sda

    # fdisk -l /dev/sda

    # fdisk -l /dev/sdb

     

    Zero out the partition table and MBR on the SATA disk as we had previously installed Fedora 7, and that data will confuse the migration process.

    # dd if=/dev/zero of=/dev/sdb bs=1024k count=100

     

    Partition new disk to add similar partition structure, including LVM partition

    sdb1 100MB ext3 /boot (0x83)

    sdb2 <the rest> LVM (0x8e)

     

    Set the disk bootable (option a)

     

    [root@mythtv ~]# fdisk -l /dev/sdb

     

    Disk /dev/sdb: 250.0 GB, 250059350016 bytes

    255 heads, 63 sectors/track, 30401 cylinders

    Units = cylinders of 16065 * 512 = 8225280 bytes

     

    Device Boot Start End Blocks Id System

    /dev/sdb1 * 1 13 104391 83 Linux

    /dev/sdb2 14 30401 244091610 8e Linux LVM

     

    Zero out the LVM partition as we had already setup a fresh Fedora 7 install, and the LVM information will still be there (and called VolGroup00, it will confuse LVM on the old IDE disk)

    # dd if=/dev/zero of=/dev/sdb2 bs=1024k count=100

     

    Copy the /boot contents across:

    # mkfs.ext3 /dev/sdb1

    # mkdir /tmp/new

    # mount /dev/sdb1 /tmp/new

    # cd /boot

    # find . –print | cpio –pmd /tmp/new

     

    Now update (install) boot loader on new 320GB disk:

    # mount /dev/sdb1 /tmp/new (if not still mounted)

    # mv /tmp/new/grub/device.map /tmp/new/grub/device.map.old

    # /sbin/grub-install /dev/sdb

    # umount /tmp/new

     

    Label this filesystem as /boot to match /etc/fstab:

    # e2label /dev/sdb1 /boot

     

    Now ‘create’ the new physical volume in LVM and display the pv’s to ensure all’s good:

    # pvcreate /dev/sdb2
    # pvdisplay

     

    Add the new physical volume into the VolGroup00 volume group

    # vgextend VolGroup00 /dev/sdb2

     

    Move all the physical extents from the old IDE disk to the new SATA disk (this will tell lvm to move the physical extents from PV /dev/sda2 to some other free physical volume – the only other volume is the SATA disk we just added). Note this will take a LONG time and will display it’s progress:

    # pvmove /dev/sda2

     

    Remove the old disk:

    # pvremove /dev/sda2

     

    Power off and disconnect power to the old IDE disk, boot to ensure all comes up ok

     

    Power off and remove IDE

     

    HOLD Get firmware for Hauppauge PVR-350

    http://ivtvdriver.org/index.php/Firmware

     

    Firmware files (Video 4 Linux):

    v4l-cx2341x-enc.fw

    v4l-cx2341x-enc.fw

    v4l-cx2341x-init.mpg

     

    Place in hot plug directory for ivtv to get and load into the PVR-350 on boot:

    /lib/firmware/v4l-cx2341x-dec.fw

    /lib/firmware/v4l-cx2341x-enc.fw

    /lib/firmware/v4l-cx2341x-init.mpg

     

    Example of missing fw in dmesg:

    Sep 5 16:02:29 mythtv kernel: ivtv: ==================== START INIT IVTV ====================

    Sep 5 16:02:29 mythtv kernel: ivtv: version 1.0.0 (2.6.22.4-65.fc7 SMP mod_unload 686 4KSTACKS ) loading

    Sep 5 16:02:29 mythtv kernel: eth0: forcedeth.c: subsystem: 01462:7252 bound to 0000:00:14.0

    Sep 5 16:02:29 mythtv kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 19

    Sep 5 16:02:29 mythtv kernel: ACPI: PCI Interrupt 0000:04:08.0[A] -> Link [LNKA] -> GSI 19 (level, low) -> IRQ 20

    Sep 5 16:02:29 mythtv kernel: firewire_ohci: Added fw-ohci device 0000:04:08.0, OHCI version 1.10

    Sep 5 16:02:29 mythtv kernel: ivtv0: Autodetected Hauppauge card (cx23415 based)

    Sep 5 16:02:29 mythtv kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 18

    Sep 5 16:02:29 mythtv kernel: ACPI: PCI Interrupt 0000:04:06.0[A] -> Link [LNKC] -> GSI 18 (level, low) -> IRQ 21

    Sep 5 16:02:29 mythtv kernel: firewire_core: created new fw device fw0 (0 config rom retries)

    Sep 5 16:02:29 mythtv kernel: ivtv0: unable to open firmware v4l-cx2341x-enc.fw (must be 376836 bytes)

    Sep 5 16:02:29 mythtv kernel: ivtv0: did you put the firmware in the hotplug firmware directory?

    Sep 5 16:02:29 mythtv kernel: ivtv0: Retry loading firmware

    Sep 5 16:02:29 mythtv kernel: ivtv0: unable to open firmware v4l-cx2341x-enc.fw (must be 376836 bytes)

    Sep 5 16:02:29 mythtv kernel: ivtv0: did you put the firmware in the hotplug firmware directory?

    Sep 5 16:02:29 mythtv kernel: ivtv0: Error initializing firmware

    Sep 5 16:02:29 mythtv kernel: ivtv0: Error -19 on initialization

    Sep 5 16:02:29 mythtv kernel: ivtv: ==================== END INIT IVTV ====================

     

    Example of initialization of fw in dmesg:

    Linux video capture interface: v2.00

    ivtv: ==================== START INIT IVTV ====================

    ivtv: version 1.0.0 (2.6.22.4-65.fc7 SMP mod_unload 686 4KSTACKS ) loading

    eth0: forcedeth.c: subsystem: 01462:7252 bound to 0000:00:14.0

    ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 19

    ACPI: PCI Interrupt 0000:04:08.0[A] -> Link [LNKA] -> GSI 19 (level, low) -> IRQ 20

    firewire_ohci: Added fw-ohci device 0000:04:08.0, OHCI version 1.10

    ivtv0: Autodetected Hauppauge card (cx23415 based)

    ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 18

    ACPI: PCI Interrupt 0000:04:06.0[A] -> Link [LNKC] -> GSI 18 (level, low) -> IRQ 21

    firewire_core: created new fw device fw0 (0 config rom retries)

    ivtv0: loaded v4l-cx2341x-enc.fw firmware (3730290280 bytes)

    ivtv0: loaded v4l-cx2341x-dec.fw firmware (3730290288 bytes)

    ivtv0: Encoder revision: 0x02060039

    ivtv0: Decoder revision: 0x02020023

    tveeprom 2-0050: Hauppauge model 48132, rev K268, serial# 9868627

    tveeprom 2-0050: tuner model is LG TAPE H001F MK3 (idx 68, type 47)

    tveeprom 2-0050: TV standards NTSC(M) (eeprom 0x08)

    tveeprom 2-0050: audio processor is MSP4448 (idx 27)

    tveeprom 2-0050: decoder processor is SAA7115 (idx 19)

    tveeprom 2-0050: has radio, has IR receiver, has no IR transmitter

    ivtv0: Autodetected Hauppauge WinTV PVR-350

    tuner 2-0043: chip found @ 0x86 (ivtv i2c driver #0)

    tda9887 2-0043: tda988[5/6/7] found @ 0x43 (tuner)

    tuner 2-0061: chip found @ 0xc2 (ivtv i2c driver #0)

    saa7115 2-0021: saa7115 found (1f7115d0e100000) @ 0x42 (ivtv i2c driver #0)

    saa7127 2-0044: saa7129 found @ 0x88 (ivtv i2c driver #0)

    msp3400 2-0040: MSP4448G-A2 found @ 0x80 (ivtv i2c driver #0)

    msp3400 2-0040: MSP4448G-A2 supports radio, mode is autodetect and autoselect

    tuner 2-0061: type set to 47 (LG NTSC (TAPE series))

    ivtv0: Registered device video0 for encoder MPEG (4 MB)

    ivtv0: Registered device video32 for encoder YUV (2 MB)

    ivtv0: Registered device vbi0 for encoder VBI (1 MB)

    ivtv0: Registered device video24 for encoder PCM audio (1 MB)

    ivtv0: Registered device radio0 for encoder radio

    ivtv0: Registered device video16 for decoder MPEG (1 MB)

    ivtv0: Registered device vbi8 for decoder VBI (1 MB)

    ivtv0: Registered device vbi16 for decoder VOUT

    ivtv0: Registered device video48 for decoder YUV (1 MB)

    ivtv0: loaded v4l-cx2341x-init.mpg firmware (3730291512 bytes)

    ivtv0: Initialized Hauppauge WinTV PVR-350, card #0

    ACPI: PCI Interrupt Link [LAZA] enabled at IRQ 22

    ACPI: PCI Interrupt 0000:00:10.1[B] -> Link [LAZA] -> GSI 22 (level, low) -> IRQ 18

    PCI: Setting latency timer of device 0000:00:10.1 to 64

    ivtv: ==================== END INIT IVTV ====================

     

    As per http://wilsonet.com/mythtv/fcmyth.php?SID&expandables=closed&ivtv=open&pvr350out=open#capture:

     

    Alternatively, use yum to install from the ATrpms repository

    # yum install ivtv-firmware

     

     

     

    Get DVD libraries

     

    Download from ATrpms:

    libdvdcss-1.2.9-3.fc7.i386.rpm

    # rpm –install libdvdcss-1.2.9-3.fc7.i386.rpm

     


    Vendors for PVR computing parts in Canada:
    As a convenience to the Canadian members of our community, I’d like to start a list of retailers that sell harder-to-find components.

    New Type: www.ntcw.com — Zalman, Thermalright, Swiftech, Vantec, Alpha, Seasonic, Nexus updated Mar 23 03
    RP Electronics: www.rpelectronics.com — DIY Electronic supplies
    Digikey: http://canada.digikey.com — DIY Electronics, Panaflo, etc
    E-Compuvision: www.e-compuvision.com — Vantec, Alpha, Swiftech, Zalman updated Dec 31 03
    Bigfoot: www.bigfootcomputers.com — Thermalright, Panaflo, Swiftech, Zalman, Alpha & more updated Dec 31 03
    Tweakbox: www.tweakbox.com — Panaflo, tails & more
    QuietPC: www.quietpc.ca — Fortron, Zalman, Nexus, I-Style, PowerSnooze, VIA, AcoustiPak, Molex, Papst, moreupdated Dec 31 03
    Maxibyte: www.maxibyte.biz/cat4_1.htm — Zalman, Q-Technology, Papst
    MutePC: www.mutepc.net — Koolance, Zalman, Q-Technology, Papst, Akasa, Molex, Noiseblocker updated Dec 31 03
    Genitech: www.genitechcomputers.com/parts-cpu.shtml — Zalman
    autodeletepro: www.adpmods.coml — Panaflo, Evercool, Thermalright updated Dec 31 03
    Techniche SilentPC: www.silentpc.net — Silent PC retrofitting: Seasonic, Nexus, Thermalright, Panaflo, Zalman, etc. added Mar 19 03
    NCIX: www.ncix.com — Zalman, Alpha, ThermalRight, Antec, Papst, Panaflo, Vantec, Ahanix updated Dec 30 03
    FrontierPC: www.fronet.com — Zalman, Thermalright, Evercase, Nexus, Seasonic, NoVibes, Arctic Cooler, Antec, Panaflo, Samsung w/8MB buffer updated Jan 1 04
    Vibe Computers: www.vibecomputers.com — Thermalright, Zalman, Swiftech, Panaflo, Antec, Papst updated Dec 31 03
    Memory Express: www.memoryexpress.com — Panaflo, Samsung, Thermalright, Zalman, Ahanix/Nikao, updated Dec 31 03
    Canada Computers: www.canadacomputers.com — Zalman, Antec, etc. updated Dec 30 03
    La centrale informatique: www.shoplci.com — Evercase, Antec, Vantec. added Dec 30 03
    CIPC: www.cipc-info.com — Antec, Asaka, Panaflo,Papst, Vantec, Zalman. added Dec 30 03
    Lux-Design: www.lux-design.com — Panaflo, Thermalright. added Dec 31 03
    ByteWize: www.bytewizecomputers.com — Antec, Sparkle, Zalman, Samsung added Dec 31 03
    shopRBC: www.shoprbc.com — Thermalright, Antec, Zalman, Swiftech, Alpha added Dec 31 03
    Atop Online: www.atoponline.com — Samsung, Zalman added Dec 31 03
    Ajump: www.ajump.ca — Evercase, Ahanix, Antec, Sparkle added Dec 31 03
    myCableShop: www.mycableshop.ca added Dec 31 03
    Canadian Tire: — TrimBrite Door Edge Molding (see this posting) added Dec 31 03

    If I missed anything post a reply to this thread & I’ll add your updates.

    RAM based filesystems in Linux

    When doing I/O intensive processing on Linux systems, I’ve found that creating a RAM based filesystem can substantially improve processing times. Of course nothing but the transitory processing data should be written to the fake filesystem to avoid data loss in the case of unintended dismount or system crash.

    mount -t ramfs ramfs /tmp/ramfs -o size=4m

    Soekris net5501 SBC Linux installation

    Soekris Engineering net5501 SBC setup with Linux

    2008/09/03

    net5501 is a x86 SBC that I ordered with 4 10/100 ethernet ports, 512MB memory, 500MHz Geode LX CPU

    Serial console is used for setup of net5501 – BIOS writes to serial port since there is no xVGA port. <ctrl-p> to enter BIOS setup. DB9 pinout:

    2 — 3

    3 — 2

    5 — 5

    Use 19,200 bps 8 data bits, no parity, 1 stop

    With the Macbook Pro, I use a Keyspan USA-19HS USB <–> DB9 RS232 serial converter (and DB9-RJ45 adapters to implement the null modem configuration and allow me to use an ethernet cable for the serial console <–> Keyspan device.

    On OS X (10.5) I use “screen” to provide the serial terminal interface:

    $ screen /dev/tty.USA19H1a2P1.1 19200,8

    <ctrl-a><ctrl-\> to exit

    On the net5501 BIOS, PXEBoot is disabled:

    set PXEBoot=Disabled

    I setup voyage-0.5.0 on a compact flash card then installed the card into the net5501 – works great the first boot

    Default root info: root / voyage

    OpenBSD setup info:

    http://techblagh.blogspot.com/2008/08/installing-openbsd-43-on-soekris-5501.html

    MythTV FC7 LVM on RAID1 Configuration

    MythTV PVR HDD Mirroring 2008/07/24
    Host: n43 (mythtv)
    – Two SATA 500GB drives sda sdb
    – current production drive is sdb

    Problem: I’ve done migrations of LVM2 volumes from 320GB SATA to 500GB SATA and added
    a redundant 500GB SATA. Now I want to get software RAID 1 setup to protect the
    root, swap and /storage filesystems from damage if/when one of the shiny new 500GB SATA
    disks bite the dust.

    Followed howtoforge.com linux_lvm_p1 (start of article) to free up sda from LVM
    volume group VolGroup00 .. http://www.howtoforge.com/linux_lvm_p7

    0. Did a file level backup to the fileserver:
    [root@n59 20080724]# sshroot@192.168.1.2This e-mail address is being protected from spambots, you need JavaScript enabled to view it“tar cf – /lib” | dd of=mythtv-lib.tar
    (repeat for /boot /storage /var /etc /home)

    1. Free up sda2 LVM volume. I know this volume is not used anymore,
    but it still has same-disk backup of /storage from when I was tweaking
    MythTV.

    [root@mythtv ~]# pvmove /dev/sda2
    [root@mythtv ~]# vgreduce /dev/VolGroup00 /dev/sda2
    [root@mythtv ~]# pvremove /dev/sda2

    – now running on sdb only –

    Setup RAID 1 mirroring (md)

    2. Partition sda for mirroring (Auto RAID label)
    [root@mythtv ~]# fdisk /dev/sda
    <delete partitions>
    <add primary 1 whole disk>
    <set flag to fd – Auto RAID>

    [root@mythtv ~]# fdisk -l

    Disk /dev/sda: 500.1 GB, 500107862016 bytes
    255 heads, 63 sectors/track, 60801 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 19 152586 83 Linux
    /dev/sda2 20 60801 488231415 fd Linux raid autodetect

    Disk /dev/sdb: 500.1 GB, 500107862016 bytes
    255 heads, 63 sectors/track, 60801 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 19 152586 83 Linux
    /dev/sdb2 20 60801 488231415 8e Linux LVM

    Notice that sdb is still using only LVM, not RAID.

    Continue reading

    Linux iptables notes

    Add local redirection of low port to unpriv high port

    Remove any existing entries:

    iptables -t nat -D PREROUTING –src 0/0 -p tcp –dport 25 -j REDIRECT –to-ports 11025 2> /dev/null
    iptables -t nat -D PREROUTING –src 0/0 -p tcp –dport 80 -j REDIRECT –to-ports 8080 2> /dev/null

    Add new redirects:
    iptables -t nat -I PREROUTING –src 0/0 -p tcp –dport 25 -j REDIRECT –to-ports 11025
    iptables -t nat -I PREROUTING –src 0/0 -p tcp –dport 80 -j REDIRECT –to-ports 8080