Random tech notes and tutorials

First off, this is related to the post in VMware Communities here: http://communities.vmware.com/message/1576370

I recently updated a VMware Infrastructure to vCenter 4.1. All ESX servers were still on ESXi 3.5. This shouldn’t be a problem, you just can’t use some of the new features available in 4.1. That’s fine, except that I couldn’t view the managed paths in the ESX servers. I would get the error below. I would get the same error when trying to manually migrate a virtual machine to a different ESX server.

You can see the error states Item has already been added. Key in dictionary: ‘Vmomi.Host.PlugStoreTopology+Path’ Key being added: ‘Vmomi.Host.PlugStoreTopology+Path’. And of course you can see that two entries are listed.

I tried multiple solutions, including completely reinstalling vCenter with a new database and reconfiguring. Restarting the ESX servers didn’t matter. The real solution was upgrading all of the ESXi servers to 4.1. This shouldn’t seem necessary, but the upgrade through Upgrade Manager (they got rid of the Host Update Utility after 4.0) was relatively easy and smooth. After having issues with vCenter 4.1 I was reluctant to move the ESX servers there, but so far everything is working great, including migrations and managed paths.

Debian has its own package for Subversion, but most of the time you want to use the latest Subversion package that’s out there. This explains how to build and use that over Apache and HTTPS.

• Installation
• Configuration
• Maintenance and Use

Installation

This particular install does not use the Berkeley DB method of code repository storage, but rather the flat file system storage method. Both have their advantages, but the file is believed to be faster. Read more here.

First setup Apache and get all the Subversion dependencies.

Then we get this warning, but FSFS is fine to use instead of Berkeley.

configure: WARNING: we have configured without BDB filesystem support

You don't seem to have Berkeley DB version 4.0.14 or newer
installed and linked to APR-UTIL.  We have created Makefiles which
will build without the Berkeley DB back-end; your repositories will
use FSFS as the default back-end.  You can find the latest version of
Berkeley DB here:

http://www.oracle.com/technology/software/products/berkeley-db/index.html

Continue with the build:

After install, this error comes up, but it can be ignored and the next few steps will fix.

apxs:Error: Activation failed for custom /etc/apache2/httpd.conf file..
apxs:Error: At least one `LoadModule' directive already has to exist..
make: *** [install-mods-shared] Error 1

Create the file /etc/apache2/mods-available/dav_svn.load with the following:

Copy over the modules from source and install them in the Apache directories; also enable SSL since we want to push this over a secure channel:

Configuration

Create the file /etc/apache2/sites-available/svn so that Apache knows about the SVN repository:

Now enable the site and start/restart Apache:

Setup the initial repository with the svncreate command and make the user running the web service the owner, since they will be the user actually modifying the repository files.

Now we can create the username/password files along with the access files.

Create the access file to your repositories.

And now edit the file. You can set users using r and rw access writes. First you list the repository, and then the folder location after that for more fine grained permissions.

Now reboot the server and test access; it should start up automatically.

Maintenance and Use

The best way to use SVN over HTTPS is with Tortoise for Windows or some other tool if using Linux, like RapidSVN.

Adding Additional Users

To add more users, just run the htpasswd command linked to your dav_svn.passwd file, same as the initial configuration for users.

And now edit the access file containing the other users and defined in the Apache configuration. You can set users using r and rw access writes. First you list the repository, and then the folder location after that for more fine grained permissions.

Backing Up the Repositories

To backup a repository, use the svnadmin dump command which will export the entire database and revisions. You can then tar up and gzip the dump file for compression, and back it up to tape or disk somewhere else. There are also incremental backups that can be done of disk/tape space is an issue.

Restoring the Repositories

Restoring the SVN database is simply rewriting all the revisions from the dump back into a database. The restore process also works well for moving an older repository over to a new one since restoring the dump into a new SVN database will update it to that version.

Duplicity is a backup tool that works off of rsync and rdiff libraries to copy only changes to a backup location. It can use compression and encryption tools on the data and also has the ability to save to Amazon’s S3 service. More details can be found here.

• Installation on OpenBSD 4.4
• Installation on OpenBSD 4.6
• Installation on Debian Lenny 5.0
• Sample Backup Scripts

Installation on OpenBSD 4.4

The 4.4 version was the most difficult to get working since the majority of the issues came from the given OpenBSD libraries. Even installing the Duplicity port from the packages didn’t function right.

First we need to add a few packages. You can use the pkg_add function with whatever mirror to obtain the following, some depend on others so there will be others in the file install list:

• python-2.5.2p4
• py-boto-1.3
• gpgme-1.1.5
• librsync-0.9.7
• ncftp-3.2.1

When the main Python package is installed, it will ask you to create a few symbolic links, so create those.

Version 4.4 needs a separate Python XML package to work properly. If it’s not installed, you’ll get a series of errors when trying to send data to S3; I believe the XML error is when it tries to read the response. Something like this will error out:

To avoid that, a separate Python XML package needs to be downloaded and installed:

Now we can install Duplicity.

If you run the Duplicity jobs as root in a cron job, there is something about OpenBSD (I’m sure a security issue) that causes it to fail. I would get the output below in my log only when it ran as a cron job:

Odd that it can’t read the temporary folder that it created. Changing the folder location also did not work. The solution is to create a separate user for only backups. The can be an issue if you have files that cannot be read by all users and need backup, but I found in my case this worked for the specific files that needed to be saved.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn’t any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can’t be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Installation on OpenBSD 4.4

The 4.4 version was the most difficult to get working since the majority of the issues came from the given OpenBSD libraries. Even installing the Duplicity port from the packages didn’t function right.

First we need to add a few packages. You can use the pkg_add function with whatever mirror to obtain the following, some depend on others so there will be others in the file install list:

• python-2.5.4p1
• py-xml-0.8.4p8
• py-boto-1.7a
• gpgme-1.1.5p0
• librsync-0.9.7p0
• ncftp-3.2.2

When the main Python package is installed, it will ask you to create a few symbolic links, so create those.

Now we can install Duplicity.

If you run the Duplicity jobs as root in a cron job, there is something about OpenBSD (I’m sure a security issue) that causes it to fail. I would get the output below in my log only when it ran as a cron job:

Odd that it can’t read the temporary folder that it created. Changing the folder location also did not work. The solution is to create a separate user for only backups. The can be an issue if you have files that cannot be read by all users and need backup, but I found in my case this worked for the specific files that needed to be saved.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn’t any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can’t be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Installation on Debian Lenny 5.0

The Debian install is a little bit simpler and can run the backup job as root inside cron. Get some install packages first:

Install Duplicity:

Creating a user is optional, but good security practice for it not to be root.

Make sure to add the new user to the deny list in SSH with DenyUsers dpbackup in the file /etc/ssh/sshd_config; there isn’t any reason for it to log in.

Now su as this new user. A GPG key needs to be created so that the compressed backups can be encrypted and signed. This way no one else that may have access to our S3 account (Amazon employees) can read the data.

There will be a series of questions, most of the defaults are fine.

• Choose option 1 for DSA and Elgamal (the default)
• Choose the default key size of 2048
• Leave the default that the key will not expire, option 0
• Enter a User ID, Email address, and comment for the key.
• Type O for OK to accept.
• Enter a long passphrase for the key and allow it to be generated. I usually do at least 20 characters since the password will just sit in a script anyway.

Move the keys to some other safe place so that they can’t be lost. No key means the backups are worthless. Typically a second backup source is a good idea.

See sample scripts below for backup jobs.

Sample Backup Scripts

The first portion of the script defines the variables we’ll need to use. The AWS keys are defined for you when you sign up for S3. Passphrase is the GPG passphrase set on the key generated from gpg –gen-key. The S3 bucket should be fairly unique, so I use the host name of the server. The others are pretty obvious but will be explained later.

Just some sample backup methods for MySQL or Subversion if needed.

This is only necessary on OpenBSD since it’s a security feature. We open it up now from 128 and close it back down later.

Most of these options can be read in the man page of Duplicity, and there are many more to choose from. Basically this backup is going to do a full backup ever 7 days (from the $FULL_IF_OLDER_THAN variable), and use encryption with the highest bzip compression, before sending it to S3. It will write a fresh backup log to the defined file, which we’ll email out later.

This line just gives us some space in the log file; really it’s just for email formatting.

This command will check how many full backup sets are already on S3, and remove any more than what is defined in KEEP_MAX_SETS.

Again, for formatting purposes.

This command lists out the current files in our backup set so they can be reviewed in the email, making sure everything is working out it should.

Now we can mail out the log file. The -s flag is for the subject line, and the TO_EMAIL is defined in our variables. We’re just writing the log file as the body of the email.

Since we exported the keys and passphrases, we want to make sure we don’t leave those around any longer than we have to; set them null.

Just a little clean up so we don’t waste space.

This is for OpenBSD only. Since we opened the open file limit up at the beginning of the script, close it back down.

End it.

Out of the box the trackpad didn’t scroll or have “click” capability. Just creating these two files fixed the issue.

/etc/hal/fdi/policy/x11-synaptics.fdi

/etc/hal/fdi/policy/mouse-wheel.fdi

Reboot and give it a try.

Operating System: Debian Lenny 5.0

This server needs an /opt directory for the package install, so the partitioning is a little bit different than a typical Linux setup. This is what mine ended up looking like:

Setup a few packages necessary for the server first.

Now users and groups need to be added for permissions and the Samba folder share access.

Create the folder where the QuickBooks data files will be stored and set the appropriate permissions.

Now configure Samba by moving the built in configuration and writing your own.

The configuration file should read:

Now restart Samba and test the permissions using a Windows client. You should be able to see the logs created by each client and who was accessing the share.

Using Alien, we’ll create a deb package from an rpm so it can be installed. Some other directories and files need to be created for logging purposes since Debian uses rsyslog and QuickBooks won’t create them on its own.

We need to add a line to the syslog configuration in /etc/rsyslog.conf, just put it at the end.

Setup the QuickBooks binaries to startup automatically.

Modify the file /opt/qb/util/qbmonitord.conf in include the directory where the QuickBooks data will live.

Restart the server and you should be able to run a ps -e and see the following processes running indicating the server is up. There also should be a /home/qbdata/qbdir.dat file created automatically.

I had some trouble getting the wireless to function properly on my T61 with Slackware 13. I tried combinations of wicd (the wireless network manager) and DHCP clients, different drivers, but nothing seemed to work. I could see the wireless points, but they always showed up as “hidden” and appear to connect, but would dever be able to get an IP address.

At this point I moved to Debian to see if that would connect using wicd. Sure enough, wicd connected and authenticated fine, but a kernel panic in Lenny using that wireless adapter would only leave it connected for about 5 minutes and then lock. Enough of that.

Back to Slackware. One thing I noticed was that Debian used the latest wicd, version 1.6.2.2 where the Slackware extras includes the 1.6.2.1 Slackware package. Even the wicd site recommends using the included package in the extras.

Slackware also came with the same firmware for the 4965 wireless as Debian, so I know if I used that, I should be good to go on that end. First, enable the firmware as root:

Restart your computer and make sure the wireless adapter is loading properly on boot. You should be able to do an lsmod | grep iwlagn and see a few lines with the module enabled. Now grab wicd 1.6.2.2 from source; you can view them here: http://sourceforge.net/projects/wicd/files/. Unpack it and install wicd.

You can check /etc/rc.d and find a rc.wicd executable. This means the daemon should start on it’s own when booting. Start the wicd daemon and then the curses version of the client.

The curses GUI is pretty easy to understand and you should be able to configure the network no problem. When you hit Shift+C to connect to an AP, you can see that it will authenticate and grab an IP this time…finally. I’ve been able to connect to WPAv2 and WPAv1. Previously I could connect to neither, although I never tried plain old WEP. Others clamined WEP would work and WPA would not, but not being able to connect to a WPA network was a big show stopper for me.

Handy little script for backing up Subversion to an FTP server.

Operating System: OpenBSD 4.4

Sendmail is configured and enabled by default in OpenBSD, but it only allows you to send mail out from the machine itself (on localhost, as it should). These steps will allow you to relay from the server and set relay restrictions.

As root, make a copy of the original localhost config file to one of your own.

Open the file you just created and comment out the line:

by adding dnl to the the front to read

Then modify this line so that Sendmail will listen on all interfaces rather than just local:

to read…

Now compile the configuration that you created and make it the default Sendmail config:

Open /etc/mail/relay-domains and add IP addresses/ranges that are allowed to relay through the server. The format used is: 192.168.1 which is equivalent to 192.168.1.0/24. This will allow other hosts on your network to relay mail through this server.

Modify /etc/rc.conf and replace:

with…

This will tell the flags to use our newly created .cf file we compiled earlier. I usually change the q30m (which means keep things in the queue for 30 minutes) to q2d, keeping the queue active for 2 days before ditching it.

Do a clean reboot and make sure the correct configuration comes up. You can test access by using a server with the same subnet as in your “relay-domains” file and telnet-ing to port 25.

You can restart Sendmail quickly by killing the process first…

…and then restarting:

Operating System: OpenBSD 4.4

Installation

First grab the necessary compiled packages from OpenBSD.

Then get the Apache source code for the HTTP server, configure and install. Use a 2.2.x version.

Next get the newest Subversion source code, configure and install.

Add the proper user to run the httpd daemon

Configuration

Setup the initial repository with the svncreate command and make the user running the web service the owner, since they will be the user actually modifying the repository files.

Now edit your main httpd.conf file in /usr/local/apache2/conf/ to read these changes. They’re not all in the same place, just scattered throughout the file. The first two changes should already be there after installing the Subversion source, just require slight modification. The last “location” change you’ll need to add manually. You’ll see the dav_svn* files in there, we’ll get to those next.

Now we can create the username/password files along with the access files.

Create the access file to your repositories.

And now edit the file. You can set users using r and rw access writes. First you list the repository, and then the folder location after that for more fine grained permissions.

Naturally you’ll want to lock this service down with SSL and possibly make it available outside the network. To simply create a self-signed certificate and add it to Apache, do the following.

Now add the lines in the httpd.conf file in /usr/local/apache2/conf/ just about the Location setting.

Edit the rc.conf.local file in /etc/ to turn on Apache.

And then edit the rc.local file to auto start Apache.

As well as the shutdown file rc.shutdown to kill the process.

Now reboot the server and test access; it should start up automatically.

Maintenance and Use

The best way to use SVN over HTTPS is with Tortoise for Windows or some other tool if using Linux, like RapidSVN.

Adding Additional Users

To add more users, just run the htpasswd command linked to your dav_svn.passwd file, same as the initial configuration for users.

And now edit the access file containing the other users and defined in the Apache configuration. You can set users using r and rw access writes. First you list the repository, and then the folder location after that for more fine grained permissions.

Backing Up the Repositories

To backup a repository, use the svnadmin dump command which will export the entire database and revisions. You can then tar up and gzip the dump file for compression, and back it up to tape or disk somewhere else. There are also incremental backups that can be done of disk/tape space is an issue.

Restoring the Repositories

Restoring the SVN database is simply rewriting all the revisions from the dump back into a database. The restore process also works well for moving an older repository over to a new one since restoring the dump into a new SVN database will update it to that version.

Operating System: Debian Etch 4.0

Install and Key Generation

First we just need to grab the primary packages from the repos and install. Make sure you’re root.

Next find the easy-rsa directory, and copy those files over to the OpenVPN configuration directory so we can setup a certificate.

Now in the /etc/openvpn directory open up the vars file and make some edits that suit you. I only made changes to the very end of the file.

Save this file. Then run:

Yeah, there’s a dot, a space, and then another dot in there. Then these commands:

You’ll be asked the cert questions, but most of the defaults should be filled in for you since you manually entered them in the vars file. Now build the server key:

You’ll be asked the same type of questions, but for common name you need to enter something. “Server” is the default. Run this next command, which will take awhile.

Then generate your TLS-AUTH keys:

Now create a key directory closer to the root folder to stay organized and copy the necessary keys there:

Server Config File

My server configuration is located in /etc/openvpn/server.conf. It’s what worked for me. The 172.21.0.0 subnet is the virtual one used by the VPN. The 10.10.0.0 subnet is the LAN I’m trying to connect to.

More info on configuration options is here: http://openvpn.net/howto.html. You’ll also have to enable packet forwarding so packets can flow from the VPN interface to the ethernet interface. Open the file /etc/sysctl.confand uncomment this line:

Restart the server.

Setup the Revocation List

Now setup a revocation list so you can block certificates and users that you create. Execute your variables again.

I had to modify my openssl configuration and repoint to my openvpn directory.

Edit the config file openssl.cnf at the end and comment out the pkcs11 section if you’re not using it, otherwise it will throw errors. Then create your CRL:

User Configuration

Now create your first user:

Answer the same prompts and give it a password. If you don’t want to use a password, just use build-key instead. Restart the OpenVPN server for it to read the config:

Now, on the client machine run the same install commands (assuming you’re using an Ubuntu or Debain box) and create a keys directory:

Copy the keys ca.crt, user1.crt, user1.key, and ta.key into the keys directory and then create a file called client.conf in the /etc/openvpn directory. Be sure you restrict access and lock down the keys directory, since compromise of these files will give someone else access.

Here’s my config:

You can get more info on the configuration here: http://openvpn.net/howto.html. Now start up the VPN:

You can check the logs for errors, but in a few seconds, if you run an ifconfig, you can see a tun0 device has been created and has one of the virtual IP addresses. You can then ping the remote VPN server’s inside address for testing.

Routing Issues

In my situation, my VPN server was not the default gateway on my LAN, so I had to add some permantent routes to my clients so they could find their way back through the tunnel and to my remote client. For Linux boxes use:

And on Windows use:

Adding and Removing Other Users

When you need to add new users or client certificates, simply run:

This will generate the keys for the new client to copy down to their machine, just the same as the initial client.

Removing users is easy as well.

You may see a bunch of error 23′s at the end, but that’s normal and just testing that the certificate does not have access anymore.