!NO Moa 'O Sauce

Secret Sauce, OpenBSD, NGinx and PHP

Mon, 11 Mar 2013 08:54:00 GMT

Secret Sauce, OpenBSD, NGinx and PHP

11 March 2013

I've been struggling with getting nginx and php to be friendly to each other in OpenBSD.

Read all the wonderful allocates for nginx and thought it was time to test the waters when OpenBSD embraced the web server by incorporating it into the Base build.

Successfully deployed html serving nginx, reverse proxy nginx, and now I really really need to get PHP hosting, and SSL hosting to work. All the documentation out there says it is sooo simple, but why haven't I been able to do it for the 1st 3 tries?

pkg_add nginx
pkg_add php-fpm

Ingredient #1

As always, the key thing to (L)earning something new is, not to chew off more than you can.

So, if you're going to try something new. Take a list of the minimal ingredients, and only work with those ingredients.

Like the fool I am, I tried the new install using a mix of the base install of nginx with the package install.

nginx is in the base install of 5.2
unless you really know what you're doing, just use that.

Ingredient #2

Basics Works.

Keep your above, simplified environment, sane.

Don't be like me, where I'm mixing things randomly.

Example:

install nginx web server package
install php-fpm package for PHP FastCGI
/etc/rc.d/nginx start (base install) works (typo)
change configurations
/etc/rc.d/enginx restart (ports install) everything html works, php breaks.
confirm /etc/php-fpm.conf has chroot enabled
/etc/rc.d/enginx restart everything breaks
read documentation, ooops nginx package is not yet chroot
edit /etc/php-fpm.conf has chroot disabled
/etc/rc.d/enginx restart everything breaks

When I finally realised:

php-fpm configuration sample is chroot
nginx base install is chroot
nginx package install is not chroot

My random behaving system isn't so random after all. It just took a long time to realise this.

The php-fpm sample configuration file is chroot (just like the default Apache 1.3 chroot base install.)

Ingredient #3

Don't get ahead of myself. Why am I installing the nginx package?

There's mention that there are some features supported in ports that aren't available in the base install
There's a new version on the web, and I may need some of those features?
My install is going live and will need to add features along the way, do I need those non-base install features?

OpenBSD systems are relatively easy to rebuild, reinstall, just build the basic, feature complete system you require and nothing more.

Trawling the mail archives

Fri, 15 Feb 2013 09:30:00 GMT

Trawling the mail archives

15 February 2013

A cheap archive is only as good as getting back information from that archive.

We built a Mail Archiving solution using a spare VM box, disk space, OpenBSD and Postfix, and Procmail. but it isn't that useful if all you're going to do is put to tape and tell everyone you have the archive.

How do you actually make use, trawl, the archives and retrieve information from the archive when users have a bad mail day and need to retrieve mail that you have hidden on that tape?

The basics of our configuration is we have a separate machine (the archiving box.)

Postfix recieves mail

Postfix as the Mail Transport Agent (MTA) is configured to:

Forward all messages to their destination
Accept all mail from mail server(s)
BCC Deliver a copy of the message to a local account
Forward all mail to the next destination: oblivion

Procmail for archiving

For our local mail delivery, the local user account forwards processing to procmail.

Procmail stores the messages in a predefined folder/filename structure that meets our business archiving needs (e.g. year/month/day)

Trawling the archives

And we finally get to the strategies for making use of those archives.

I've been vacillating on a search engine installation, originally drooling over htdig and various failed install attempts, to Apache Solr gaining traction and recently documented for FreeBSD by BSDMag

I still need to use a search engine at some point, because the archives are growing to 30GB+ a month, but in the meantime I got a job to extract mail for userX over 5 days.

I've documented how I did that in trawling the archives and it boils down to using procmail and formail (part of the procmail package) to wade through the messages and suck out the messages that met my criteria (to userX).

We already had the messages separated by date, so it was just a matter of feeding those days of messages into my procmail recipe and getting the mail that our user wanted.

Once the recipes are built, the whole process is relatively fast and pain free. We even have a work-around for getting that archive mail to our Outlook friends.

Our recipe is rather simple, but it highlights the flexibility you have to trawl the arcives with your own dig(solr)ing into the procmail recipe book.

SSH: What it takes to get your work done

Thu, 17 May 2012 11:35:00 GMT

SSH: What it takes to get your work done

17 May 2012

Michael W Lucas' book: SSH Mastery: OpenSSH, Putty, Tunnels and Keys.

Good enough that I avoided buying the book, even when it was released with funding support my favourite Open Source project (OpenBSD with OpenSSH.) Good enough that after recieving a blogger review copy the first thing I did was to hit the corporate buy button to order a legitimate print/e-book copy for my cohort, fellow sysadmin, users. Why?

I was under some insane self delusion that I didn't want to be bound by the book's research, so that I can ethically 'document' my own stumbles into SSH to share freely with others. Fortunately, a short look at the books contents and the better solution for users and System Administrators, is to just buy this book.

What value is there in this book:

The Guru in the room
Saving Money
Augmented Reality (extending your infrastructure)
Saving Time

The Guru in the room

We don't know what we don't know.

The fastest path of learning I've enjoyed has been as the new kid amongst 'zen masters' who danced on their keyboards making magic happen across our network(s). Unfortunately the real masters moved on and we graduate a little higher up the ladder until we've reached the peak of our incompetence.

The book is a good reference source, with fine examples for many features, and like the zen masters, some of the answers is in the 'debug' sections, how to determine whether what you think you should get, is how SSH is seeing it.

Online articles are often short, make assumptions about how OpenSSH/Putty works, 'script' a lot of commands that require version X.Y of this and M.N of that. Rarely are there supportive notes on how to diagnose the instructions, or related system has response.

SSH Mastery explores, explains, provides samples, provides debugging techniques so we can explore, understand, type-in the SSH commands to see all those features at work. Not the guru in the room, but the next best thing, someone knowledgeable to go to.

Saving me money?

Chapter 3: The OpenSSH Server
- Testing and Debugging

A technical configuration to start in a book? After the general introduction to the topic, data encryption, it seemed odd to dive into configuring the server?

I was hoping for magical command-line tricks. But it is understated how critical it is to configure your server correctly, and how to validate the server is working correctly: debug

4 years ago I was locking down a machine in the USA (from Australia.) I'd spent a month configuring some complicated Mail Processing system on that box, and was almost ready for the 'live' output. The only thing left to do was formalise the lock down of the machine.

2 minutes later, I'd locked myself out with a typo in my ssh server configuration. After ripping my hair out, I found the answer (documented in Chapter 3) and published it online and @serverfault.com

That lost server, lost time, lost configuration was throwing money out the door.

Augmented Reality (a flexible and secure infrastructure)

SSH Tunnels have many uses, but I have always found it difficult to follow the manpage ssh. SSH Tunnels lets us augment, extend our existing network/infrastructure in ways the physical configuration would not allow.

Chapter 9: Port Forwarding
- Services on localhost
- The web from somewhere else

We tunnel extensively at work to let us run services on Unix hosts, but lock those services down for access only from localhost (i.e. a legitimate user account, using SSH Keys is required to tunnel onto the machine and using port forwarding download e-mail (which contains a lot of diagnostic information, system reports) et. al. onto our monitoring host.

Automation scripts/.fetchmailrc configuration files get forgotten, we're always falling back to documentation when its time for upgrades and changes on our network.

As mentioned, tunnels tend to be hard to understand (and the command-line ordering can still confuse those who've been using it.) SSH Mastery is a good introduction, with good examples, and a good connection

I was in Tonga over the Christmas break when I needed to do some funds transfers on some accounts in Australia, but the internet awareness/security doesn't allow any transfers from an IP Address from Tonga.

Thanks to OpenSSH, Putty and socket routing, Christmas ended well.

Saving time.

Is SSH Mastery comprehensive? Not nearly, which is good. There's still a lot out there) waiting for your articles. It does however cover a lot of things that I haven't been considering, and need to within my day job and home network.

The Guru in the book definitely covers a lot of things that I now use daily, because others better than myself were "doing it" and quickly led me in the right direction.

Chapter 4: Verifying Server Keys
Chapter 5: SSH Clients
Chapter 6: Copying Files over SSH
Chapter 7: SSH Keys
Chapter 8: X11 Forwarding
Chapter 9: Port Forwarding
Chapter 10: Keeping SSH connections Open
Chapter 11: Host Key Distribution
Chapter 12: Limiting SSH
Chapter 13: SSH Virtual Private Networks

Some of the information seem so basic now, after years of stumbling through them but the details and exploration helps to clarify my own understanding. Some areas I don't use, I should know, and now I have a reference that tells me some of what I need to attend.

Summary

Even if you have some one with patience and wizard knowledge to help you with this fundamental tool, I'm finding this book useful. It is a great investment for both end-users, system administrators and developers.

Refer to other reviews on the web for the utility of this title:

SSH Mastery: A Very Welcome Addition to Any Unix User's Bookshelf
@amazon
Title: SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys
Author: Michael W. Lucas
Publisher: Tilted Windmill Press (January 18, 2012)

A little network knowledge really does help

Mon, 14 May 2012 10:02:00 GMT

A little network knowledge really does help

14 May 2012

One of our clients was having serious problems with installing and getting Microsoft Lync to work. The previous Support organisation spent a couple of months on the problem and gave up, but the user never gave up.

When we took on the contract to provide support, our support technicians could get the accounts to work outside the customers environment, and intermittently at the customers site.

Web Proxying
Firewalls
Solution

After working with every possible iteration of installing, uninstalling, configuring the software, using separate desktops, different versions of Windows.

Web Proxying

Suggestions were made that the problem was with the web proxy deployed at the customer site, but we had success at other sites with proxies, and the software was failing login even when bypassing the proxy.

Again, the problem only occured at the client site.

Firewalls

Suggestions abound in a corporate environment, that whenever an Internet service fails it's something to do with the Firewalls.

To validate the user assumptions, special rules were inserted to allow the clients full, unfettered access to the Internet.

Still no success. Again, other sides have both proxies and firewalls restricting Internet access (except via the proxies) and they work fine.

Solution

Where to go after the basics clearly show something is broken, but not what?

A packet trace shows the connection query from the client going out and then no further connection attempts?

Turns out the DNS A record for the service hosting for Microsoft Lync (a Hosted Service for the client) didn't exist, so the client stopped processing at some point because it couldn't find a place to connect to (our guesstimate at this time.)

Manually insert a DNS record for the external site, on our DNS server, and magically the Lync client connects.

Wow, amazing how many things depend on DNS, and how even large companies with large budgets can screw it up. The sad irony is that the client is a reseller for the big ISP that hosts Microsoft Lync services, and it was that ISPs DNS server that was screwed. The ISPs DNS records for their resellers are apparently not responding with the same records as for external users.

Remember your network kung-fu.

Postfix smtp_tls_policy_maps fingerprinting

Wed, 02 May 2012 15:51:00 GMT

Postfix smtp_tls_policy_maps fingerprinting

02 May 2012

It bothered me enough that I need to record it, and hopefully the path to a solution that others will follow.

(delivery temporarily suspended: Server certificate not verified)

Lesson: Document things properly, especially if it's something interesting, more so if the technology/thing you're doing is normally not what you do, and it's already taken you a long while to get it working properly in the first place.

Mind you, the above may be a difficult task when rushed to get a system out and the only way to confirm the installation is to break it apart and start from scratch

Scenario:

We exchange e-mail with an external organisation (duh!!) with regulatory standards that requires us to ensure e-mail sent to them is encrypted. We achieve this through the following:

Certify that the server we're connecting to is theirs by using:
- using SSL certificates
- smtp_tls_policy_maps and
- fingerprinting
Encrypt the traffic between the two sites using TLS

So, we follow the online Postfix TLS Support and smtpd_tls_fingerprint documentation and have it up and running with the basic configuration:

File extract: /etc/postfix/main.cf

smtp_tls_policy_maps = hash:/etc/postfix/tls_policy

File extract: /etc/postfix/tls_policy

example.com    fingerprint
    fingerprint-digest-is-here

Problem:

External Organisation used a 1 year self-sign certificate, it expires (as most eventually do) and no messages go through them. We get the below "cryptic" message in our logs:

(delivery temporarily suspended: Server certificate not verified)

Answer:

Seems easy enough, we just need to re-do/fix our 1st step above for Certifying the connection.

Get updated certificate from remote site
Update the fingerprint

Load up the online documentation and follow it through.

Oooops, it doesn't work.

The logs laugh: /var/log/maillog

(delivery temporarily suspended: Server certificate not verified)

The message is not sent (deferred) with the error message "Server Certificate not verified".
The message is never sent, since the Server Certificate is never validated.
Bypass certification and send e-mail. The short-term configuration is to not require the fingerprint to be 'certified'.

I'm sure I followed the steps correctly ... (wrong)

Solution:

Walk away from the documentation for a while, walk through it again with the presumption that you've screwed everything up so you need to take all your knowledge and check the basics (verify assumptions) as you go along.

digest format
fingerprint
policy file

Digest Format

[smtp_tls_fingerprint_digest]

Verification of an SMTP server certificate fingerprints, uses a message digest.

Don't get trapped putting together fingerprints that are invalid, or unnecessary. Find out which fingerprint digest is supported by your configuration, and use that.

postconf | grep fingerprint

lmtp_tls_fingerprint_digest = md5
smtp_tls_fingerprint_digest = md5
smtpd_tls_fingerprint_digest = md5

The above configuration output shows we're using the MD5 digest format. It should be fine, but read the documentation about what it says may be the better choice digest for you.

Fingerprint

[Ref: openssl x509]

After acquiring getting your SSL Certificate through some 'trusted' method, generate the fingerprint for the 'trusted' certificate in the following method.

openssl x509 -noout -fingerprint -md5 -in /etc/ssl/certs/example.pem

MD5 Fingerprint=fingerprint-digest-is-here

After comparing the above fingerprint-digest-is-here with what I have in the tls_policy file, it is obvious they don't look anything similar.

Policy File

With the above fingerprint, and digest we can fix the TLS Policy table such as the below:

example.com    fingerprint
    fingerprint-digest-is-here

Remap the file to make sure the correct hashed version is active:

# postmap /etc/postfix/tls_policy

Restart the server and things are coool.

postfix reload

But isn't that what the Postfix documentation says you have to do?

I guess it does, but for some reason the steps I took those days weren't the correct steps. And now that I've rehashed the already hashed, I hopefully will not mis-read the documentation the next time through.

TLS and Postfix

Thu, 30 Jun 2011 16:15:00 GMT

TLS and Postfix

30 June 2011

Upgrading some of our Mail Servers to support for TLS (Transport Layer Security) in Postfix and apart from learning how to do it, also learned a key maxim of programmers (readily applicable to system administrators)

DO NOT PRE-OPTIMISE

Wasted two days of my life, with increased anxiety during the install, configuration process because I was trying to be too smart too early.

After a Duhhh moment, I went back to the very beginning of the install process, and did everything as per the known guides (without that little tweak I had preconceived, and the install worked in less than an 1 hour)

My failure? I got too far ahead of myself, with bright ideas, untested of how I wanted things to work, and started modifying my plans (and solidifying assumptions about how things will work) before collecting evidence for that the assumptions for each stage, were valid.

My idea was for the TLS roll-out on 5 different servers (all requiring SSL certificates) could all use one Certificate Authority. I'd made self-signed certificates before, so presumed/guessed at an approach for one centralised Certificate Authority. Unfortunately, instead of verifying my assumptions of how that can be done, I steam-rolled ahead ass-uming some minor modifications to the process would just work.

Create Certificate Authority (CA) key
Create Certificate Signing Request (CSR) for the host
Create a Certificate (CRT) from the CSR, signed by my new the CA key

The install failed, but gave error messages hinting at problems with the key created in my step #2, or the certificate created in step #3. After agonising through different diagnostic processes from the various error messages. It took 2 whole days to throw away the assumption that caused the error, my change in how I was generating (or using a Certificate Authority.) Arggghhhh!!!

I had been blindly looking at various avenues for why Step #2 or Step #3 were not working correctly, including trying stupid hints from random websites.

The error that Postfix was throwing up said that:

File extract: /var/log/maillog

warning: cannot get RSA private key from file /etc/ssl/private/server.key.pem:disabling TLS support
warning: TLS library problem: xxxxxx certificates routine xxxx key values mismatch xxxxx src/crypto/x509/x509_cmp.c:318:

Can't read the Key
There is no match between the key and certificate

OK, the key file is there, I can see it in the file system. I can open it up with openssl and verify that it is a valid key file by using:

sudo openssl rsa -noout -text -in /path-to/private/server.key.pem

I could even validate that the signed certificate is a valid certificate, likewise the Certificate Authority certificate (so far as our current understanding tells us.)

sudo openssl req -noout -text -in /path-to/server.crt.pem
sudo openssl req -noout -text -in /path-to/private/ca.crt.pem

I blissfully ignore the 2nd error message until I could resolve why my Postfix server was complaining about the Server Key. The assumption, it's probably an 'artifact,' an error caused by the previous error (can't open the key.) We find all sorts of "solutions" on the web, which may work on other OS's, but irrelevant for our OpenBSD install (most related to using 'openssl rsa -in server.key.pem -out server.key.rsa.pem to make sure that the key file is not password protected ?) Not relevant for our OpenBSD install.

It was well into the third day before I found references to verifying that a certificate is created from a key.

$ sudo openssl rsa -noout -text -in /path-to/private/server.key.pem -modulus \
    | grep ^Modulus | openssl md5
$ sudo openssl x509 -noout -text -in /path-to/server.crt.pem -modulus \
    | grep ^Modulus | openssl md5

The use of "| openssl md5" just simplifies the comparison of the Modulus values which are supposed to be the same if they are paired (i.e. certificate was generated from the key.) There's also the requirement that both "public exponent" are equal but the above Modulus comparison is a quick verification process.

OK, I'm running the above command line on my self-signed certificate, and server key. The Modulus DO NOT MATCH.

What?? That doesn't make sense?

I wander through comparisons of all the key & certificate pairs, to find out that the Modulus for my designated CA Key, matches with the Self-Signed Certificate.

What?? That doesn't make sense?

Obviously (duhh) there must be something wrong with my signing process. We trace back our implementation steps and re-do, re-test.

Step #3. No that didn't work. No, don't repeat it again. Go back to
Step #2 then #3. No that didn't work. No, don't repeat it again. Go back to
Step #1 then #2, then #3. No that didn't work.

OK, something is seriously wrong!!!

The 2nd error (and quick perusal into the source code) definitely indicates that the key file is not related to the certificate. Our Modulus investigations above shows that the key/certificate pairs are not created correctly. Could my CA ideas be the cause of my install failures?

Throw that assumption away and create certificates how you've always done it.

Step #2 Sign the CSR using the Server Key.

Normal self-signed instructions always use the same key for the CA as well as the Server.

5 minutes later, we have Postfix TLS working as expected, and our documentation is complete. Postfix TLS without dovecot, without cyrus-sasl, woohoo, too easy.

Now to verify that TLS actually encrypts ?

Watching over your wall

Fri, 10 Jun 2011 15:39:00 GMT

Watching over your wall

10 June 2011

As networks continue to grow, sometimes against our wishes, sometimes with our full support, it becomes more important to get some overview of how and what is moving across your network(s.)

In the beginning, in a land far away, we only had a few machines wired up and life was simple.

Now, most of us have too many machines with an unknown quantity of malware pounding on them (and subsequently on your network.) That's before we even get to our beloved users.

If you get blamed when things go bad on your network, it's time you started taking charge of knowing what's going across your network. Michael W. Lucas' published an insightful book to help us with that Network Flow Analysis. More importantly, for us, is that he chose to describe the solution using tools accessible to everyone (aka Open Source.) We've finally cleaned up some internal notes for getting the software to work well in our favourite os (tm) OpenBSD

These notes augment the installation instructions from that book. Where the human factor is important, in customisation/localisation, interpretation, we don't do any of that here.

Buy the book.

Now you're back, follow through to find out how we put it together for Netflow with flow-tools

It's saved our bacon a number of times, we know who's packets are causing congestion, what times congestions occur, why things occur. AND, we can print out those meaningless charts that senior dweebs nod their heads and just love.

Michael W. Lucas has some war stories where traffic flow monitoring has helped him out, and we can attest to it's daily, weekly value.

Our notes on Netflow with flow-tools

Disk Utilisation i386 4.9

Thu, 19 May 2011 09:27:00 GMT

Disk Utilisation i386 4.9

19 May 2011

Every now and then people ask how they should partition their hard disk, this doesn't answer that question, but gives some view on how much disk space is used up on a bare system built for compiling OpenBSD from source.

Reference OpenBSD 4.9 i386, FAQ 5

The following is a summary of disk space used on a bare install built for and after compiling OpenBSD 4.9 i386. No packages installed.

path	Used	More Info
/etc	60M	Bare install, no modifications
/usr	6.0G	STABLE source extracted to src, xenocara, and compiled using ./obj, ./xobj, as well as ./rel for release files, and ./dest for pre-release files
/usr/src	851M	includes compiled kernel GENERIC.MP
/usr/ports	332M	No compiled packages, no distfiles
/usr/obj	1G
/usr/xenocara	540M
/usr/xobj	420M
$DESTDIR	2G	/usr/dest Includes cvs export for src, xenocara, and ports
$RELEASEDIR	500M	/usr/rel Includes tgz source for src, xenocara, ports
$CVSROOT	4.4G	CVS Tree scp'd from another server/workstation.
$CDBUILD	1.1G	Contains pre-build CD directory and install.iso created with mkhybrid(8) (no packages) approximately 490MB each

Preparation Pays Off - Big time

Thu, 07 Apr 2011 17:39:00 GMT

Preparation Pays Off - Big time

07 April 2011

One of those days, when the disaster you didn't want, barges through the door, but forward planning, preparations, testing gets you through the day. Also known as, we and our gweeky friends say "Ku-oool," while the rest of the family say, "uhhh, ok, we're happy for you."

We could have had a major disaster (i.e. my day ruined, as opposed to things melting down) which were nicely averted because of (as said before.)

forward planning
preparations
tests to verify the preparation.
activate on live system
what have we learned

The Disaster

Our PRIMARY data link provider suddenly went off the air. More of our workers are at remote sites, than are at the central office (where I'm sitting.) The WAN going down means that a lot of people are not able to do their work (or are impaired from using IT services they are normally reliant on.)

The diagram indicates the level of dependence those satellite sites have on this primary data center. Site A has a completely independent data service, so loss of the link limits a few operational issues for IT, but no loss of service to the business.

Site's B, and C, are independent for the majority of their business needs, but in the current situation are dependent on our Primary Data Center for shared services such as e-mail. Other than that, they can operate without the WAN link.

Sites D, E, and F can't work while the Primary Data Center is OFFLINE.

We couldn't connect to the provider's next hop link, and we definitely couldn't get any traffic, let alone BGP routing information.

All those nice tricks for verifying that your BGPD server is up and running are nice, but they don't do you any good when your 5 other sites confirm that the primary vendor's BGP Server is definitely not online

Forward Planning ?

After years of cajouling, the powers above folded and added a SECONDARY WAN service instead of the previous dependence we had of tunneling VPN through an Internet ISP connection.

Unfortunately, since there were budget constraints and the original WAN Data Link service was commissioned without regard for a secondary, we had to come up with some mechanisms for getting the SECONDARY connected.

After balancing different options with what the business operations required and our limited resources, we decided to configure the two systems as ACTIVE-STANDBY. One Link was ACTIVE (the Primary link) and the other configured as a STANDBY service. We could automate the switch, but given the reality of the infrastructure, we would meet a requirement of X hours to switch the data between the services(i.e. go from ACTIVE-STANDBY to OFF-ACTIVE)

Preparations

We gradually rolled out the secondary, backup, data link using off-the-shelf desktops as the routing/gateways. The routing, access policies were updated to include the potential for routing through the secondary link.

For some sites, and services, we load balanced traffic along both data links.

TEST

All the preparations were nice and dandy, but what would we actually have to do to make sure things were flipped from one service to the other? We needed to do a partial test on the actual network instead of our test network.

After some time, we just pushed through that downtime was required and a full service test is required taking all OFFLINE while we routing changes, tests (of course we had to do it during organisation down-time, which inevitably means that IT are up at odd hours or working during everyone else's downtime/bedtime)

Going through the preparations and controlled tests forced us to look at ways to minimise operator error during the process (controlled automation in as many bits of the process as possible.)

We successfully completed the tests on a subset of the full WAN network (site B, and D with the Primary Data Center,) found some further points in the operation that we wanted to improve and went through evolving those bits of the operation.

Suffice it to say, after that test, we were confident that we could switch over from FAILED-STANDBY to FAILED-ACTIVE well within the 2 ~ 4 hour window that was part of our agreement with business.

Activating on LIVE System

Doing my bit sleeping during one of those interminable meetings where you watch paint drying on the wall, or the back of your eye-lids (depending on how lucky you are.) One of the IT team woke me up, seriously disturbing the meeting, to say that all hell has broken loose. All sites were down, the WAN Link has disappeared. People were running trying to figure what to do next.

What do I tell XYZ at Site-A?

What do I tell everyone here at main office ?

What, when, where, who ?

I walk calmly to my desk, to find that my offsider (partner in these things) wasn't at his desk.

That's odd ?

Sit myself down at the desk. OK, look at through some of the charts generated by Smokeping, yup the primary link looks like it disappears about *here (pointing at the screen.) The charts also show that the secondary link is humming along just fine, although latency to Site B is off the charts (200 ms, is that even possible?)

My boss sees me working and goes to get a cup of coffee.

Log onto our WAN Gateway box, and yup our BGP Server is humming along just fine, we're advertising our LAN routes through BGP but that's all I can see (as mentioned earlier, the Primary linkn next hop is not responding to pings so we can't get to it and there's no hope of trying to get BGP traffic from/through there.)

Switching from the Primary Link to the Backup Link

ACTIVE-STANDBY to FAILED-ACTIVE

Using the shortcuts I've got, log onto 3 of the 6 remote sites through the secondary data link. Site D, E, and F. Site B is not connecting on either of its redundant active-passive gateways. Yep, BGPD is running fine on those sites, and showing advertising but no other routing information on those servers.

Run a script on each active gateway and we are now flipped over to the secondary link.

Total time to flip the link between 4 sites ? About 3 ~ 4 minutes after sitting down at the desk.

What happened to the other 3 sites?

Site A, and C we haven't rolled out the secondary links (Site A is wired but we haven't had anyone available to go down and plug things in. It's also a low prioarity. Site C is only a month old and just hasn't had reason for the secondary link, if the link failure is prolonged then users can work through the User VPN or we can set up a slow tunnel through the Internet.

Site B had the 200ms latency problem. My admin-buddy had to walk across to that office.

Testing the Service

Spent another 30~40 minutes going through the routing validation process, and refining the routing et. al. (yeah, you've really got to get a document together of these things, largely so you've actually gone through the exercise and have a clearer experience with what needs to be done.)

Fortunately, because we have QOS Queues on our gateways, specific for each Data Link Service, it is easy to confirm whether data is still routed through the Failed Primary Service, or if they are all going through the Active Secondary/Backup Service.

systat queue

We make some corrections in our queueing that were showing some traffic still showing up on the FAILED link. Adjusted a few things here and there that would simplify the whole process in the future.

Switch from STANDBY-ACTIVE to ACTIVE-STANDBY

Another 30 minutes passes, and the Primary Service comes back online. Since the Primary Service provides a much much bigger Data Link than our Secondary link, we are definitely very keen to put everything back onto it.

In two minutes, we were able to re-route all remote WAN sites to talk to each other through the Primary Link (to ease some of the traffic from the Secondary link) especially since this is a very minimal part of the traffic, but let's us look at the routing issue as well as whether the service can at least stay up for more than a few seconds.

After another while, we re-route all traffic back to the Primary link. That took another two minutes (at most.)

The last switch, no-one knew about.

What have we learned

Even with the knowledge we gained from the controlled TEST, we gained a whole lot more knowledge when having to perform the same process on the WHOLE network.

We've identified a few more areas that we can better administer, automate, and are in the process of updating those.

Putting the effort down up front sure saved my bacon, more important for the business, it meant that after jumping up and down that their network connection was down, the users could sit down and get on with work (making money for the company, serving customers et. al.)

Active - Active ?

Why aren't the Data Link's on Active-Active ?

Not really worth the effort at this point (not our call)

The Data Links are not equivalent, they have their different benefits but are not equal to make it an easy load balancing equation
Doable, but with a lot of 'moving parts' that will be difficult to maintain within our current resource constraints.
- Remember that whatever knobs are tuned to get ACTIVE-ACTIVE has to be easy and quick to switch back when one of the services fail and we have ACTIVE-FAIL or FAIL-ACTIVE.

Where was my admin-buddy ?

Sometimes the call of nature is of even higher priority than your IT needs.

Summary

Smiling on the train home, 'cause I'm not working overtime tonight (you do get overtime don't you ? (smiling because we know we don't.))

Oh yeah, those six sites? They're connected using OpenBSD 4.8 redundant ACTIVE-PASSIVE gateways. Connecting to them, monitoring, managing during uptime and downtime are just a blast!!

Ethicacy in Telephone Interview Answers

Wed, 30 Mar 2011 22:07:00 GMT

Ethicacy in Telephone Interview Answers

30 March 2011

Aka: Googling during a phone interview

This is tangentally relevant to OpenBSD, you can safely ignore it and you're life will not have missed anything. Take the road less travelled.

Ethics and IT
An example Ethical Dilemma
How many bits in a mac address
In Linux, what is the default signal sent by kill
Of the ps output what is the label D for
Summary

Ethics and IT

We continue to have some interesting discussions at work about the ethicacy of a lot of things we get around to in IT. For example, we're the guys that are brought on by various departments and HR to assist them in forensic type stuff which sometimes goes into trolling through peoples archives on our backup tapes (email, documents, etc.)

The generalised 'ethos' statement in the workplace seems to be:

if it's legal, then you do it.

But we have an abundant list of recent and current Global Events of totally unethical behaviour dressed 'legal' as defined by the conqueror to not be so enthralled by such simplistic misdirections.

An example Ethical Dilemma

Our ethical dilemma, within IT, for today was a phone interview I went through where purposeful trip-up questions were raised. Given time, some of the questions could possibly have been deduced, but why bother when you can easily Google/Bing to get your answer ?

Note: The field with a huge library of answers freely published online is IT (and fields where the IT crowd are fixated with, such as music, science fiction, and fantasy.)

The questions seem to have been good questions, in some manner, and definitely tripped me up because I didn't know, but do the questions reveal comparability of skills, or abilities to search the web?

One of my univesity courses, an Accounting course, had an open book final course exam (the only one I've ever been in) and this was largely so students didn't have to memorise any of the material, but if you didn't understand the material, there wasn't enough time to find answers and have it relevant to the problems in the exam.

Was this one of those problems ? Was my error in not asking / clarifying whether I could use [choice of favourite search engine]?

Hopefully you find the material educational in what it may be asking and how easy it is for IT personnel to find answers on the internet without having to memorise things. You still have to know your stuff to make use of the answers, but it is soo easy to find answers to IT things on the Internet these days.

Were these questions good IT questions ?

How many bits in a mac address

"Urgghhh, I don't know. I recall when I read them in places, that they're separated with colons, and theres something like four or more of them."

What races through my mind: "How could I figure this out with-out Googling?"

I'm talking with the interviewer that I'm trying to figure out the answer

I knew the address numbers were in hex (0,1,2...,d,e,f) but for the life of me I couldn't remember how many pairs there were.
I flipped over my laptop to look for a mac address, (you know some of the devices these days have it on a sticker) Nope, one of those stupid devices that has that sticker buried inside on top of the physical device.
The laptop was on, so I got a command-prompt and tried to look for the mac address. No go, Windows doesn't show it if the device isn't active (Grrrr, should have used ipconfig /all and that may have had the answer, and I just knew there was a reason I should have installed a Unix thing on this device, oh wait, I did and it didn't work for what I was using this laptop for: manpage: ifconfig )
OK, I've got a phone and these things have MAC address for their wifi. Can't use this, 'cause I'm on the phone.

Wait, ..., What's the difference between using Google/Bing and dissecting the answer from getting an example MAC and manually calculating the # of bits ?

What does it reveal

Have you had enough exposure in networking, especially at the command-prompt or configuration files, where this knowledge has become ingrained.

Why don't I know this ?

What was it again that makes it useful to have this knowledge? I recall some of those digits represent a unique id for the device vendor, and then the rest is used by the vendor to 'create' a unique ID for each physical device.
Tech Trivia: Microsoft published their standard where it used the MAC address with other items to create a GUUID for each word document (wow, that's even more useful knowledge) so they can track the origin of any word document around the globe.
We're the l33t of computer nerds, we are a fount of knowledge of the most trivial and irrelevant knowledge. This is just one of those that I now know, but had not come across it in any meaningful way before hand.
Where have I had actual reason to record them? MAC Address ACLs for squid-cache and dhcp, but obviously wasn't taking enough interest to even remember how many digits were involved, let alone the number of bits.
MAC addresses show up on ARP, but I haven't bothered to worry about them unless there was some conflict requiring further investigation.

And that was only the first question!!! Things are definitely not looking up for my interview.

We're in trouble and we haven't even passed the first step.

In Linux, what is the default signal sent by kill

Urggggh, never thought of that before. I may have read it somewhere but definitely haven't used it 'without an explicit' signal to 'know' what to expect as a default behaviour.

This one is simple enough to find from the manpage: kill(1) Straight there in the 1st Paragraph of the Description.

What does it reveal

Have you had enough exposure in Unix administration where this knowledge is ingrained.
NFI

Why don't I know this ?

Have to say, I've never used the kill command without an explicit signal. Didn't think it was the kind of command that was sane to be launching without explicitly telling it how to behave.
I guess the default is portable enough, since Linux and OpenBSD both agree on the default behaviour (using a sample of '2' to base this simplification)
I guessed at SIGHUP (-1) but that's just bias on what I try to do first before I do the KILL(-9).
Now, here lies a powerful tool not meant for most mortals. Including me 8<

Of the ps output what is the label D for

Urgghhh, OK, this interview is seriously becoming a disaster. Haven't really bothered with looking at the 'labels' except to see whether the service/app was a zombie or didn't even execute.

This one took a little longer to find (had to page through two screens to get at the answer), but it's right there in the ole manpage: ps(1) but look for it under the column 'state'

What does it reveal

Have you had enough exposure in Unix Administration where this information is ingrained?
NFI

Why don't I know this ?

Truthfully? Don't ever recall seeing this 'state' 'D' before to have investigated it.
Obviously haven't worked on enough resource constrained systems where the state 'D' was common enough to be noticeable.
The last time I had to really worry about an under resourced machine was with RedHat 4.0 or 4.2 and the i386 was blazingly fast, and we had 4 x 9600Kbps zyxel fax/voice/modem hanging off the box doing wonders no-one had ever heard off.
Well, the hosts I monitor are more single purposed, over engineered for their purposes (because that's the only hardware you can get these days.)

After learning a little more about 'D' I'm a little more pleased with my work environment than I was previously. There are some poor bastards out there who either don't get enough resources, or a dealing with real cool problems that have these 'D' issues.

Summary

If anything, I'm glad I've added to my glossary of commands, and leaves us with this lesson:

If you get a phone interview on a topic that is thoroughly covered by the Internet, clarify with the interviewer whether you're allowed to use the Internet as a resource, and if not, are you allowed to use other resources at your finger tips (and voice search on your phone doesn't count!!! because my phone runs an OS no one talks about.)

There may be no ethical dilemma, just the need to clarify.

8-)