Tue 08 May 2012
Tags: centos, rhel, yum.
Ok, this has bitten me enough times now that I'm going to blog it so I
don't forget it again.
Symptom: you're doing a yum update on a centos5 or rhel5 box, using rpms
from a repository on a centos6 or rhel6 server (or anywhere else with
a more modern createrepo available), and you get errors like this:
http://example.com/repodata/filelists.sqlite.bz2: [Errno -3] Error performing checksum
http://example.com/repodata/primary.sqlite.bz2: [Errno -3] Error performing checksum
What this really means that yum is too stupid to calculate the sha256
checksum correctly (and also too stupid to give you a sensible error
message like "Sorry, primary.sqlite.bz2 is using a sha256 checksum,
but I don't know how to calculate that").
The fix is simple:
yum install python-hashlib
from either rpmforge or epel, which makes the necessary libraries
available for yum to calculate the new checksums correctly. Sorted.
Fri 13 Jan 2012
Tags: aoe, centos, coraid, rhel.
Updated 2012-07-24: packages updated to aoe-80 and aoetools-34,
respectively.
I'm a big fan of Coraid and their relatively
low-cost storage units.
I've been using them for 5+ years now, and they've always been pretty
well engineered, reliable, and performant.
They talk ATA-over-Ethernet (AoE),
which is a very simple non-routable protocol for transmitting ATA
commands directly via Ethernet frames, without the overhead of higher
level layers like IP and TCP. So they're a lighter protocol than
something like iSCSI, and so theoretically higher performance.
One issue with them on linux is that the in-kernel 'aoe' driver is
typically pretty old. Coraid's
latest aoe driver is version
78, for instance, while the RHEL6 kernel (2.6.32) comes with aoe v47,
and the RHEL5 kernel (2.6.18) comes with aoe v22. So updating to the
latest version is highly recommended, but also a bit of a pain, because
if you do it manually it has to be recompiled for each new kernel
update.
The modern way to handle this is to use a
kernel-ABI tracking kmod, which gives you
a driver that will work across multiple kernel updates for a given EL
generation, without having to recompile each time.
So I've created a kmod-aoe package that seems to work nicely here. It's
downloadable below, or you can install it from my
yum repository.
The kmod depends on the 'aoetools' package, which supplies the command
line utilities for managing your AoE devices.
kmod-aoe (v80):
aoetools (v34):
There's an init script in the aoetools package that loads the kernel module,
activates any configured LVM volume groups, and mounts any filesystems.
All configuration is done via /etc/sysconfig/aoe.
Fri 28 Oct 2011
Tags: centos, ldap, linux, openldap, rhel, sysadmin.
Having spent too much of this week debugging problems around migrating
ldap servers from RHEL5 to RHEL6, here are some miscellaneous notes
to self:
The service is named ldap on RHEL5, and slapd on RHEL6 e.g.
you do service ldap start on RHEL5, but service slapd start
on RHEL6
On RHEL6, you want all of the following packages installed on your clients:
yum install openldap-clients pam_ldap nss-pam-ldapd
This seems to be the magic incantation that works for me (with real SSL
certificates, though):
authconfig --enableldap --enableldapauth \
--ldapserver ldap.example.com \
--ldapbasedn="dc=example,dc=com" \
--update
Be aware that there are multiple ldap configuration files involved now.
All of the following end up with ldap config entries in them and need to
be checked:
- /etc/openldap/ldap.conf
- /etc/pam_ldap.conf
- /etc/nslcd.conf
- /etc/sssd/sssd.conf
Note too that /etc/openldap/ldap.conf uses uppercased directives (e.g. URI)
that get lowercased in the other files (URI -> uri). Additionally, some
directives are confusingly renamed as well - e.g. TLA_CACERT in
/etc/openldap/ldap.conf becomes tla_cacertfile in most of the others.
:-(
If you want to do SSL or TLS, you should know that the default behaviour
is for ldap clients to verify certificates, and give misleading bind errors
if they can't validate them. This means:
if you're using self-signed certificates, add TLS_REQCERT allow to
/etc/openldap/ldap.conf on your clients, which means allow certificates
the clients can't validate
if you're using CA-signed certificates, and want to verify them, add
your CA PEM certificate to a directory of your choice (e.g.
/etc/openldap/certs, or /etc/pki/tls/certs, for instance), and point
to it using TLA_CACERT in /etc/openldap/ldap.conf, and
tla_cacertfile in /etc/ldap.conf.
RHEL6 uses a new-fangled /etc/openldap/slapd.d directory for the old
/etc/openldap/slapd.conf config data, and the
RHEL6 Migration Guide
tells you to how to convert from one to the other. But if you simply
rename the default slapd.d directory, slapd will use the old-style
slapd.conf file quite happily, which is much easier to read/modify/debug,
at least while you're getting things working.
If you run into problems on the server, there are lots of helpful utilities
included with the openldap-servers package. Check out the manpages for
slaptest(8), slapcat(8), slapacl(8), slapadd(8), etc.
Further reading:
Tue 18 Oct 2011
Tags: centos, linux, rhel, sysadmin.
rpm-find-changes is a little script I wrote a while ago for rpm-based
systems (RedHat, CentOS, Mandriva, etc.). It finds files in a filesystem
tree that are not owned by any rpm package (orphans), or are modified
from the version distributed with their rpm. In other words, any file
that has been introduced or changed from it's distributed version.
It's intended to help identify candidates for backup, or just for
tracking interesting changes. I run it nightly on /etc on most of my
machines, producing a list of files that I copy off the machine (using
another tool, which I'll blog about later) and store in a git
repository.
I've also used it for tracking changes to critical configuration trees
across multiple machines, to make sure everything is kept in sync, and
to be able to track changes over time.
Available on github:
https://github.com/gavincarr/rpm-find-changes
Tue 16 Nov 2010
Tags: brackup, centos, rhel, riak, sysadmin.
Been playing with Riak recently, which is
one of the modern dynamo-derived nosql databases (the other main ones being
Cassandra and Voldemort). We're evaluating it for use as a really large
brackup datastore, the primary attraction
being the near linear scalability available by adding (relatively cheap) new
nodes to the cluster, and decent availability options in the face of node
failures.
I've built riak packages for RHEL/CentOS 5, available at my
repository,
and added support for a riak 'target' to the latest
version (1.10) of brackup (packages
also available at my repo).
The first thing to figure out is the maximum number of nodes you expect
your riak cluster to get to. This you use to size the ring_creation_size
setting, which is the number of partitions the hash space is divided into.
It must be a power of 2 (64, 128, 256, etc.), and the reason it's important
is that it cannot be easily changed after the cluster has been created.
The rule of thumb is that for performance you want at least 10 partitions
per node/machine, so the default ring_creation_size of 64 is really only
useful up to about 6 nodes. 128 scales to 10-12, 256 to 20-25, etc. For more
info see the Riak Wiki.
Here's the script I use for configuring a new node on CentOS. The main
things to tweak here are the ring_creation_size you want (here I'm using
512, for a biggish cluster), and the interface to use to get the default ip
address (here eth0, or you could just hardcode 0.0.0.0 instead of $ip).
#!/bin/sh
# Riak configuration script for CentOS/RHEL
# Install riak (and IO::Interface, for next)
yum -y install riak perl-IO-Interface
# To set app.config:web_ip to use primary ip, do:
perl -MIO::Interface::Simple -i \
-pe "BEGIN { \$ip = IO::Interface::Simple->new(q/eth0/)->address; }
s/127\.0\.0\.1/\$ip/" /etc/riak/app.config
# To add a ring_creation_size clause to app.config, do:
perl -i \
-pe 's/^((\s*)%% riak_web_ip)/$2%% ring_creation_size is the no. of partitions to divide the hash
$2%% space into (default: 64).
$2\{ring_creation_size, 512\},
$1/' /etc/riak/app.config
# To set riak vm_args:name to hostname do:
perl -MSys::Hostname -i -pe 's/127\.0\.0\.1/hostname/e' /etc/riak/vm.args
# display (bits of) config files for checking
echo
echo '********************'
echo /etc/riak/app.config
echo '********************'
head -n30 /etc/riak/app.config
echo
echo '********************'
echo /etc/riak/vm.args
echo '********************'
cat /etc/riak/vm.args
Save this to a file called e.g. riak_configure, and then to configure a couple
of nodes you do the following (note that NODE is any old internal hostname you use
to ssh to the host in question, but FIRST_NODE needs to use the actual -name
parameter defined in /etc/riak/vm.args on your first node):
# First node
NODE=node1
cat riak_configure | ssh $NODE sh
ssh $NODE 'chkconfig riak on; service riak start'
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready
# All nodes after the first
FIRST_NODE=riak@node1.example.com
NODE=node2
cat riak_configure | ssh $NODE sh
ssh $NODE "chkconfig riak on; service riak start && riak-admin join $FIRST_NODE"
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready
That's it. You should now have a working riak cluster accessible on port 8098 on your
cluster nodes.
Mon 22 Mar 2010
Tags: centos, dell, linux, omsa, rhel, sysadmin.
Following on from my IPMI explorations, here's the next
chapter in my getting-down-and-dirty-with-dell-hardware-on-linux adventures.
This time I'm setting up Dell's
OpenManage Server Administrator
software, primarily in order to explore being able to configure bios settings
from within the OS. As before, I'm running CentOS 5, but OMSA supports any of
RHEL4, RHEL5, SLES9, and SLES10, and various versions of Fedora Core and
OpenSUSE.
Here's what I did to get up and running:
# Configure the Dell OMSA repository
wget -O bootstrap.sh http://linux.dell.com/repo/hardware/latest/bootstrap.cgi
# Review the script to make sure you trust it, and then run it
sh bootstrap.sh
# OR, for CentOS5/RHEL5 x86_64 you can just install the following:
rpm -Uvh http://linux.dell.com/repo/hardware/latest/platform_independent/rh50_64/prereq/\
dell-omsa-repository-2-5.noarch.rpm
# Install base version of OMSA, without gui (install srvadmin-all for more)
yum install srvadmin-base
# One of daemons requires /usr/bin/lockfile, so make sure you've got procmail installed
yum install procmail
# If you're running an x86_64 OS, there are a couple of additional 32-bit
# libraries you need that aren't dependencies in the RPMs
yum install compat-libstdc++-33-3.2.3-61.i386 pam.i386
# Start OMSA daemons
for i in instsvcdrv dataeng dsm_om_shrsvc; do service $i start; done
# Finally, you can update your path by doing logout/login, or just run:
. /etc/profile.d/srvadmin-path.sh
Now to check whether you're actually functional you can try a few of the
following (as root):
omconfig about
omreport about
omreport system -?
omreport chassis -?
omreport is the OMSA CLI reporting/query tool, and omconfig is the
equivalent update tool. The main documentation for the current version of
OMSA is here.
I found the CLI User's Guide
the most useful.
Here's a sampler of interesting things to try:
# Report system overview
omreport chassis
# Report system summary info (OS, CPUs, memory, PCIe slots, DRAC cards, NICs)
omreport system summary
# Report bios settings
omreport chassis biossetup
# Fan info
omreport chassis fans
# Temperature info
omreport chassis temps
# CPU info
omreport chassis processors
# Memory and memory slot info
omreport chassis memory
# Power supply info
omreport chassis pwrsupplies
# Detailed PCIe slot info
omreport chassis slots
# DRAC card info
omreport chassis remoteaccess
omconfig allows setting object attributes using a key=value syntax, which
can get reasonably complex. See the CLI User's Guide above for details, but
here are some examples of messing with various bios settings:
# See available attributes and settings
omconfig chassis biossetup -?
# Turn the AC Power Recovery setting to On
omconfig chassis biossetup attribute=acpwrrecovery setting=on
# Change the serial communications setting (on with serial redirection via)
omconfig chassis biossetup attribute=serialcom setting=com1
omconfig chassis biossetup attribute=serialcom setting=com2
# Change the external serial connector
omconfig chassis biossetup attribute=extserial setting=com1
omconfig chassis biossetup attribute=extserial setting=rad
# Change the Console Redirect After Boot (crab) setting
omconfig chassis biossetup attribute=crab setting=enabled
omconfig chassis biossetup attribute=crab setting=disabled
# Change NIC settings (turn on PXE on NIC1)
omconfig chassis biossetup attribute=nic1 setting=enabledwithpxe
Finally, there are some interesting formatting options available to both
omreport, for use in scripting e.g.
# Custom delimiter format (default semicolon)
omreport chassis -fmt cdv
# XML format
omreport chassis -fmt xml
# To change the default cdv delimiter
omconfig preferences cdvformat -?
omconfig preferences cdvformat delimiter=pipe
Thu 11 Mar 2010
Tags: centos, dell, ipmi, linux, rhel.
Spent a few days deep in the bowels of a couple of datacentres last week,
and realised I didn't know enough about Dell's DRAC base management
controllers to use them properly. In particular, I didn't know how to
mess with the drac settings from within the OS. So spent some of today
researching that.
Turns out there are a couple of routes to do this. You can use the Dell
native tools (e.g. racadm) included in Dell's
OMSA product, or you can use
vendor-neutral IPMI,
which is well-supported by Dell DRACs. I went with the latter as it's
more cross-platform, and the tools come native with CentOS, instead of
having to setup Dell's OMSA repositories. The Dell-native tools may give
you more functionality, but for what I wanted to do IPMI seems to work
just fine.
So installation is just:
yum install OpenIPMI OpenIPMI-tools
chkconfig ipmi on
service ipmi start
and then from the local machine you can use ipmitool to access and
manipulate all kinds of useful stuff:
# IPMI commands
ipmitool help
man ipmitool
# To check firmware version
ipmitool mc info
# To reset the management controller
ipmitool mc reset [ warm | cold ]
# Show field-replaceable-unit details
ipmitool fru print
# Show sensor output
ipmitool sdr list
ipmitool sdr type list
ipmitool sdr type Temperature
ipmitool sdr type Fan
ipmitool sdr type 'Power Supply'
# Chassis commands
ipmitool chassis status
ipmitool chassis identify [<interval>] # turn on front panel identify light (default 15s)
ipmitool [chassis] power soft # initiate a soft-shutdown via acpi
ipmitool [chassis] power cycle # issue a hard power off, wait 1s, power on
ipmitool [chassis] power off # issue a hard power off
ipmitool [chassis] power on # issue a hard power on
ipmitool [chassis] power reset # issue a hard reset
# Modify boot device for next reboot
ipmitool chassis bootdev pxe
ipmitool chassis bootdev cdrom
ipmitool chassis bootdev bios
# Logging
ipmitool sel info
ipmitool sel list
ipmitool sel elist # extended list (see manpage)
ipmitool sel clear
For remote access, you need to setup user and network settings, either at boot time
on the DRAC card itself, or from the OS via ipmitool:
# Display/reset password for default root user (userid '2')
ipmitool user list 1
ipmitool user set password 2 <new_password>
# Display/configure lan settings
ipmitool lan print 1
ipmitool lan set 1 ipsrc [ static | dhcp ]
ipmitool lan set 1 ipaddr 192.168.1.101
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.254
Once this is configured you should be able to connect using the 'lan' interface
to ipmitool, like this:
ipmitool -I lan -U root -H 192.168.1.101 chassis status
which will prompt you for your ipmi root password, or you can do the following:
echo <new_password> > ~/.racpasswd
chmod 600 ~/.racpasswd
and then use that password file instead of manually entering it each time:
ipmitool -I lan -U root -f ~/.racpasswd -H 192.168.1.101 chassis status
I'm using an 'ipmi' alias that looks like this:
alias ipmi='ipmitool -I lan -U root -f ~/.racpasswd -H'
# which then allows you to do the much shorter:
ipmi 192.168.1.101 chassis status
# OR
ipmi <hostname> chassis status
Finally, if you configure serial console redirection in the bios as follows:
Serial Communication -> Serial Communication: On with Console Redirection via COM2
Serial Communication -> External Serial Connector: COM2
Serial Communication -> Redirection After Boot: Disabled
then you can setup standard serial access in grub.conf and inittab on com2/ttyS1
and get serial console access via IPMI serial-over-lan using the 'lanplus' interface:
ipmitool -I lanplus -U root -f ~/.racpasswd -H 192.168.1.101 sol activate
which I typically use via a shell function:
# ipmi serial-over-lan function
isol() {
if [ -n "$1" ]; then
ipmitool -I lanplus -U root -f ~/.racpasswd -H $1 sol activate
else
echo "usage: sol <sol_ip>"
fi
}
# used like:
isol 192.168.1.101
isol <hostname>
Further reading:
Sat 19 Sep 2009
Tags: centos, linux, rhel, yum.
Found a nice post today on
how to use yum to download source RPMs,
rather than having to do a manual search on the relevant mirror.
