IDE RAID - OpenBSD 3.3 current

Table of Contents:

• Introduction
• Retrieving Current
  • Create Kernel with RAIDframe disk driver support
• Building the release/distribution Binaries
  • Scripted Process
• Create minimal raid install
  • Create Build Directories
  • Extract files from the binary distribution
  • Make additional Directories
  • Create Customisation Files
  • Create the minimal install tarball.
• Machine Install Phase 1 - Installing the minimal pre-raided configuration
  • Partitioning the Hard Drive
  • Files to Install
  • Image partition information onto 2nd Drive
  • Copy the root partition onto the 2nd Drive
  • Make the 2nd Drive bootable
  • Power Recycle Testing
• Machine Install Phase 2 - Creating the raided Array
  • Configuration
  • Partitioning RAID
• Installing OpenBSD current on the raided array
• mount raided partitions
• extract release files
• configure system
• configure a new root password
• finalise raided configuration
• test: eare we really raided?
• Testing RAIDFramed server
• Power Recycle Testing
• Forced Fail
• Forced Recovery
• Author and Copyright

Introduction

[ref: raid(4), raidctl(8)]
[ref: Andreas F. Bobak's "Installing OpenBSD 3.3 on a Soft-RAID Array"]

These are notes expanding on the configuration of the RAIDFrame disk driver [raid(8)] from Andreas F. Bobak's mini-howto. There is also another good howto by Sven Kirmess, discovered after the initial build of this howto.

The primary instigator of these notes is to clarify the process as I perform it on a machine for which I am putting the RAIDframe on an IDE RAID configuration. That is to say, I want to be able to rebuild the system if it dies horribly one day or I get paid to put one together.

These notes will take you through the major phases of preparing and creating a raid enabled IDE RAID system using standard IDE drives.

• Compiling RAID support into the GENERIC Kernel
• Building a minimal RAID install tarball
• Preparing a system for RAID installation
• Installing/Configuring RAID onto that system
• Verifying RAID is functioning correctly on the system

Testing Environment: OpenBSD3.3 current (2003-07-09), Pentium4 motherboard with onboard RAID IDE controller (ASUS P4B533)

Retrieving Current

The source is required for building RAID support into the kernel. We will be building the RAID configuration from source, and therefore the initial part of these instructions are on acquiring the source and building a release. Maybe one day this can be expanded and put into its own walk-through. The 'release' build is primarily to get it into a state that is similar to getting distribution binaries.

The primary reason for generating a binary distribution is to allow the process to use available binary distributions (aka buy the cd or download snapshots)

For those who will be using the release source code available on CD to create their custom kernel, then you can skip the source retrieval section.

For my configuration, it is faster for me to suck the source down on a non-local site, compress the files for transfer to the local site. For those with fast enough pipes or who can get the source pre-compressed, please skip this stage.

Using anoncvs get the current release of OpenBSD. This is not necessary, but a note for how I did it.

cd /some/location

export CVSROOT=:pserver:(anoncvs site of your choice)

cvs login

cvs get src

cvs get XF4

cvs get ports

Compressing current so I can get it faster down the pipe to my work machines.

gtar -cjf src.tbz2 src/

gtar -cjf XF4.tbz2 XF4/

gtar -cjf ports.tbz2 ports/

Bring it across the wire on ftp or sftp (which is apparently better for these things than scp)

Decompressing on local system

cd /usr

gtar -xjf /path-to-file/src.tbz2

gtar -xjf /path-to-file/ports.tbz2

gtar -xjf /path-to-file/XF4.tbz2

Create Kernel with RAIDframe support

[ref: FAQ et. al.]

There are a good amount of documentation already available for customising the kernel, and we are just quick-stepping through the bits you have to do. For explanation of what we are doing here please refer to the FAQ and other documentation.

modify the RAIDKERN file to include the following options

The primary modifications are inclusion of the pseudo-device and the RAID_AUTOCONFIG. For documentation completion, I have also included the below for my kernel.

continue customisation by doing the following:

On my system, the build machine is the same arch as the proposed RAID machine, so I can test the kernel, and use it for building binaries by doing the following.

Restart the machine to ensure the kernel build worked.

Building the release/distribution Binaries

[ref: (jross_at_openvistas.net post ), release(8), http://www.geocities.com/easybakeoven88/release.html]

To ensure that the binaries are in synch with the RAID kernel, we now need to build binaries 'current' with the 'current' kernel created above.

Create the Script mybuild.sh inspired by posting noted above.

#!/bin/sh
#
# OpenBSD - Release Building Shell Script v2.4
# Created by FenderQ - November 9 2002
# 
# SEE ALSO release(8)
# 

DEST="/usr/new-root"
RELEASE="/usr/new-release"

build_kernel() {
    echo "********** Build and install a new kernel **********"
    cd /usr/src/sys/arch/i386/conf
    config GENERIC
    cd ../compile/GENERIC
    make clean depend bsd
    cp /bsd /bsd.old && cp bsd / && chown root.wheel /bsd
}

build_system() {
    echo "********** Build a new system **********"
    rm -rf /usr/obj/*
    cd /usr/src && nice make obj
    nice make build
    cd /dev && ./MAKEDEV all
}

make_release() {
    echo "********** Make and validate the system release **********"
    cd /usr/src/distrib/crunch && make obj depend all install
    export DESTDIR=$DEST RELEASEDIR=$RELEASE
    rm -rf $DESTDIR
    mkdir -p $DESTDIR $RELEASEDIR
    cd /usr/src/etc && nice make release
    cd /usr/src/distrib/sets && sh checkflist
    unset DESTDIR RELEASEDIR
}

build_XF4() {
    echo "********** Build and install XF4 **********"
    rm -rf /usr/Xbuild
    mkdir -p /usr/Xbuild
    cd /usr/ports/lang/tcl/8.3 && make install
    cd /usr/ports/x11/tk/8.3 && make install
    cd /usr/Xbuild && lndir /usr/XF4 && nice make build
}

make_release_XF4() {
    echo "********** Make and validate the XF4 release **********"
    export DESTDIR=$DEST RELEASEDIR=$RELEASE
    rm -rf $DESTDIR
    mkdir -p $DESTDIR $RELEASEDIR
    nice make release
    unset DESTDIR RELEASEDIR
}

clean_everything() {
    echo "********** Clean everything **********"
    rm -rf /usr/obj/* $DEST $RELEASE
}

usage() {
    echo "Usage: $0 options" 
    echo 
    echo "Options:"
    echo 
    echo "  kernel             - Build and install GENERIC kernel"
    echo "  system             - Build a new system"
    echo "  release            - Make and validate the system release"
    echo "  xwindow            - Build and install XF4"
    echo "  xwindow-release    - Make and validate the XF4 release"
    echo "  clean              - Clean everything"
    echo
}

if [ `whoami` != "root" ]; then
    echo "You probably should be root instead of `whoami` to run this safely." 
    exit 1
fi

START=`date`
echo
echo "***** OpenBSD - Release Building *****"
echo
echo "Dest: $DEST"
echo "Release: $RELEASE"
echo "CVS Revision Tag: $CVSTAG"
echo "CVS Server: $CVSROOT"
echo

if [ $# = 0 ]; then usage; exit 1; fi

for i in $*
    do
    case $i in
        kernel)
            build_kernel
            ;;
        system)
            build_system
            ;;
        release)
            make_release
            ;;
        xwindow)
            build_XF4
            ;;
        xwindow-release)
            make_release_XF4
            ;;
        clean)
            clean_everything
            ;;
        *)
            echo "********** Abort! Abort! **********"
            echo "Invalid option encountered: $i"
            echo "Exiting......."
            echo
            exit 1
            ;; 
    esac
    done
    
echo
echo "Start Time  : $START"
echo "Finish Time : `date`"
echo
	

Notes:

/usr is an 80GB Hard-drive with 60GB free space on this test configuration (i.e. make sure you have enough space on the partition you intend to place the files onto)

Depending on how sluggish your system is, this can be speedy or its time to weed the garden. You could of course just manually key in the commands above without need to resort to the script.

After everything is completed, we should have a RELEASE SNAPSHOT in $MYRELEASEDIR with the GENERIC KERNEL.

Create minimal RAIDframe install set

We will create a new tarball "site33.tgz" to be our minimal install set as the completion of this section of the guide.

To save on some keyboard retyping, I'm setting up some environment variables that will be used later in the process (i.e. if you use different directories it'll be easier to make the change here.)

MYRELEASEDIR is where the snapshot, cd-release, or src-build release is located.

MYBUILDDIR is the empty location where we are going to build our minimal release.

MYRAIDRELEASE is where we are going to put the tgz/compressed release build for ftp or cd-writing.

MYRAIDKERNEL is the location and RAID enabled kernel

Now, on to the extraction process. We will:

• Create the directories where we will build our RAID binaries
• Extract the files for compilation
• Make some further directories
• Create some files to simplify our install process
• Create the install tarball

Create Build Directory

Extract Files for the Build

We're splitting up part of the extraction process, so we can make some notes on what is being sucked out here. (and my ssh sessions weren't handling unlimited command-line pastes duhhhh)

We are adding the following so we can have more complete ssh support in the minimal environment (i.e. support for scp and sftp) Very useful when you forget to put something, or you can just leave it out.

Tools I soon found out that were necessary, at least in my testing period.

Copy the standard 'GENERIC' kernel as bsd.old into the working directory and copy our raid enabled kernel into the current/build directory.

Make some directories reuired during boot process

For the minimal install we can get rid of some large directories in ./var such as www. We will also need to create some directories that are 'accessed' during full startup by the ./etc/rc script.

Create customisation files

Create a dummy ./etc/rc.conf.local

Just ensuring a few things are turned off in our environment.

Now we'll make a few basic login environments for root. I'm not too good at this stuff so have copied others work.

Create ./root/.profile with the following preliminary contents

Create ./root/.login with the following preliminary contents

Create ./root/raid0.conf.new with the following preliminary contents

[ref: raid(8)]

The raid configuration file is well documented in the raid(8) man page. Take special note of the device wd0d and wd1d. These, in my configuration, are the device/partition on which we will install our raid system. You can change this now, or later as fits your configuration.

The file will be used in two sections of the installation. 1st it will be required in creating the raid configuration, and 2nd will be used by the system to 'auto-detect' the raid enabled system.

If you hate superfluous error messages then you can also perform the following 'optional' instructions, to 'create' dummy files:

Create the minimal install tarball

We are now ready to ftp-install or create a cdr with our minimal RAID enabled kernel & short-list binaries.

To complete preparations for an ftp install, we'd probably need to copy the release files onto the MYRAIDRELEASE directory:

For a boot media you can either create a boot floppy, or burn the contents of $MYRAIDRELEASE onto a CDR. To create a boot floppy, use something like the below.

To burn a CDR, make sure you use the $MYRAIDRELEASE/cdrom33.fs as the boot-image and include your standard install binaries from $MYRELEASEDIR and the raid enabled tar ball fom $MYRAIDRELEASE.

Machine Install Phase 1 - Installing the minimal pre-raided configuration

There is a two phase process in installing RAID which first requires building a minimal system that can start the RAIDFrame disk drivers, and then building the RAID enabled system on top of that.

On the new machine, we are assuming you will have two or more identical drives for RAID connected on an iDE RAID controller. In our example we will be installing and configuring on the 1st (primary) drive of this configuration, the others in the set will be created from this install drive.

My two drives are wd0 and wd1.

Partitioning the Hard-Drive

When you reach the the disk creation process create a small (50M) 'a' and similarly smal (32M) 'swap' partition. Specify 'd' as type "RAID."

For example:

Remember that in our example we are not touching the other drive(s).

Files to install

When you get the list of files to install, specify

• bsd, and
• site33.tgz.

Do not select any other set. Remember that we are only using a 50M '/' partition in our example.

After the installation is complete, restart to ensure installation is a workable installed system.

If the installation is functional, you should be able to SSH into the new machine which will also let you upload any other (read: missing) binaries required for your installation.

Image partition information onto 2nd Drive

With a working, minimal system we now need to configure the 2nd drive.

We are going to make a 'copy' of the disklabel information from our primary drive and modify it for use on our 2nd drive.

Edit the file /root/disklabel.wd1 and change the entry that showed:

To now show as:

To copy the disklabel for the 2nd drive follow the below commands to initialise a new Master Boot Record (MBR) on the drive and then to write the new partitioning information.

Our 2nd drive is now 'partitioned' and ready for use.

Create the filesystem on the '/' partition by doing the following

Copy the root partition onto the 2nd drive:

Before we make the 2nd drive bootable, we'll need to ensure all files from the 1st drive are copied across. Let's make a mount point for the 2nd drive and mount it.

Now, we need to copy all our files across from wd0a to wd1a

Make the 2nd drive bootable

Now, we'll copy the boot files across and make the 2nd drive bootable

Depending on your system, you may or may not need to perform the below configuration. Fix the fstab on the 2nd drive by editing /mnt/etc/fstab to modify the line referring to

to now refer to:

On my test configuration, the above change works fine so long as there is a detected and active wd0. If wd0 is totally failed, even though wd1 is on a 2nd controller it will start as wd0 (i.e. fstab needs to remain as original). Oh well, easier to edit in single user mode.

Power Recycle Testing

Test #1:

We now test to see whether we have configured the 2nd drive correctly. Use "halt" to stop the machine and when you restart, specify the 2nd drive as the boot drive:

Watch the process, as the system should boot from the 2nd drive, and if not it may automatically boot from the 'primary' drive.

Once this is working flawlessly, we are now assured that the boot configuration is working and can continue with the 2nd test.

Test #2:

The 2nd test will be a physical powerdown/ drive disconnection.

Use "halt -p" to powerdown the system and after the system has halted, physically disconnect the primary hard-drive (wd0) and restart the machine to test whether the 2nd drive will start correctly.

Creating the RAID Array

[ref: raidctl(8)]

During our minimal configuration we had created a ./root/raid0.conf.new file that we are going to now use for configuring the raid array.

Initialize the component labels with an ID. We are arbitrarily picking an easy number '100'

Initialize the parity set

If you make a mistake above, you can restart the process by "undoing" raid with the command "raidctl -u raid0"

From the manpage:

Now, at this point you will be probably better off going and making dinner, possibly eating it as well. On this test system it took about 2 hours to get past this stage. Strangely enough, the 2nd time around (i.e. I screwed up a few things and restarted often) this process only took about 30 minutes (?)

Partitioning your RAID configuration

Once completed with the initial configuration of the raid0 environment, we create the partitions using disklabel.

Create your partition information as you would a normal drive (except mount points are not yet associated,) so keep a list of your partition 'letters'. As per standard configuration 'b' is the swap partition, 'c' is reserved.

For my configuration, and for this documentation, I have created the following partitions all FS-TYPE BSD4.3 (except swap of course.)

There's an interesting note about 'swap' that we'll discover later. But if you can, avoid 'swap' on the 'raid' system. [ooops, have to mention that I don't have much history on this so make your call]

Format the filesystem(s) you have created:

To continue configuration, we turn on auto-configuration on raid0, making it bootable/rootable will be done later.

Installing OpenBSD current on the raided array

We are now ready to install our OpenBSD binaries onto the raided environment by

• mounting the raid partitions, then
• installing the binaries.
• configure the system
• configure a new root password
• finalise the raid configuration
• test: are we really raided?

Mount raided partitions

To install the binaries we'll create mount points, and mount the raid partitions for installation of software. We'll put '/' on /mnt and subsequent subpartitions under that.

Within the new 'root' environment (/mnt) create the mount points for the other partitions in your configuration.

Now we can mount the partitions as per our desired configuration, noted above.

Extract Release Files

Now on this configuration, I've made ./tmp a 2GB partition so we can ftp the binaries onto tmp before installation (or you can of course just grab the files from a mounted CDR.) Install the binaries by going into the new 'root' and untar'ing the files

Of course you can install all your binaries

I haven't compiled the X environment yet, which is fine since my new raid box is going to be faster than my build machine.

Configure System

Now we have a binary complete RAID configuration, except it doesn't have any of the system configuration we got from doing the proper install onto the hard disk, so we need to copy those files across onto the raid0 configuration. In this case we are concerned with at least the /dev /boot directories and bsd*/files as well as a number of configuration files.

Edit the /mnt/etc/fstab file to refer to the 'correct' configuration, by changing the line that refered to:

To now refer to our new slicing:

Configure a new root password

After this, we now need to configure a password for the raid0 environment by first changing rooting into the new environment.

Finalise RAIDframe Configuration

Now we are ready to change raid0 so that it is recognised as a root partition.

This can be reversed by using

Tell the system that raid0 is available for auto-configuration by copying the config file into the /etc directory.

This tells the minimal configuration (via /etc/rc) to boot raid0 if the kernel (in raid0:/) is available.

Reboot to test whether our hard-work has paid of.

Test: are we really raided ?

So, how can you tell whether the machine has gone into raid0 ?

The most obvious is which password gets you into root. (That is, of course, if you've selected a different root password for the raid0 configuration from the wd0a configuration)

You can tell by seeing if your booted environment is the full install (i.e. just cat /etc/fstab would be a good start)

The fallback is to watch the startup process (check with dmesg) to see if something like the below has shown up.

When it works, remember that the system is not wd0a, but raid0a which means if you are using ssh to connect, the signature has changed. (this is good)

Testing RAID configuration

To verify the raid0 configurations are actually working, i.e. backing up files (mirror in this example) we need to do some power down, remove drive, testing.

Power Recycle Testing

1st. Power down the system and physically disconnect drive 1 (wd0)

Restart the machine to ensure the raid0 array is still the boot system, although performing from the 2nd drive.

At the root prompt verify the system is running by typing

you should get a message similar to the below.

Note that 'component1: failed' line above indicates a failure.

What I did next is probably not 'proper', I powered down the system again to reconnect the primary drive and disconnected the 2nd drive.

The system showed the same message above (note message made previously about how this system designates wd0 for the 1st 'active' drive it finds, irregardless of controller)

Anyhow, reconnect both drives and restart the system.

Because of my smarty activities above, the system automatically complains about 'parity' and starts checking the system. At this point I go for lunch and come back afterwards to find an AOK, working system.

Remember how we put 'swap' into the raided environment. Well, there you go, the swap space was different between the two drives and forced a dirty-parity check.

hmmm, walk away and because I went away for a while, we really should check the status of the system with: raidctl -s raid0

OK, seems the thing figured itself out (although probably not the optimal way to do this.) I guess, in a manual sense I should have chosen to:

Verify the parity is ok using : raidctl -P raid0

If a new drive has been put in, to rebuild that drive from the working drive.

For example, if drive 2 has failed we would recreate the partition information on drive 2 (as above) and then use the following command(?):

Forced Fail

A test mentioned in the man page, is to perform a forced 'Fail' (which just fails the raid0 component and does not reconstruct the drive as partitioned from the beginning.) The command is supposed to begin reconstruction immediately.

so, let's go for it. Better find out now, then later when it doesn't work on a critical system.

OK, if you try it on a two drive system, then you'll probably get a message like this:

Ooops, ok it seems we really needed a spare for that command to do both the fail and reconstruct.

Forced Reconstruction

So, I guess it's a good time to test whether the "-R" really does works:

OK, something seems to be happening. To make sure we know what stage our 'reconstruction' is at, keep a watch on 'raidctl -s raid0' and you will get a progress report such as the following:

I've highlighted as bold the areas that should be changing. raidctl will perform the work in the background and when it finally finishes it should give you a message similar to the below.

Coffee Please

And there we should have it, we've rebuilt our failed drive.

Now for that well deserved cup of coffee. (snore, snore, snore)

Afterthoughts - What have we learned

Word: quiescence

n 1: a state of quiet (but possibly temporary) inaction [syn: dormancy, quiescency] 2: quiet and inactive restfulness [syn: quiescency, dormancy, sleeping]

Source: WordNet ® 1.6, © 1997 Princeton University (via dictionary.com)

ldd - there's a lot of dependencies involved with binaries, so per file copying is more difficult and is a trial and error environment if you don't use ldd to find dependencies for some of the files you are bound to use and need.

raidctl(8) - is very well documented and with pleasant examples. After seing the potential from Andreas F. Bobak's mini-howto the man pages were very helpful in completing things I couldn't get from the howto. It was the source for a number of misunderstandings on my part.

Author and Copyright

Copyright (c) 2003 Samiuela LV Taufa. All Rights Reserved.

I reserve the right to be totally incorrect even at the best advice of betters. In other words, I'm probably wrong in enough places for you to call me an idiot, but don't 'cause you'll hurt my sensibilities, just tell me where I went wrong and I'll try again.

You are permitted and encouraged to use this guide for fun or for profit as you see fit. If you republish this work in what-ever form, it would be nice (though not enforceable) to be credited.

Copyright  © 2000/1/2 NoMoa Publishers All rights reserved. Caveat Emptor