7.1 :: DAS Disaster Discovery
|| Prev | Next | Index ||
What happens if your DAS servers are destroyed by fire or flood? What happens if they are stolen? How do you handle this? First, we will examine a few disaster scenarios. Then, we will look at the steps necessary to restore critical DAS files that include user information and Kerberos data.
Fortunately, if the DAS slave is destroyed, a new one can be rebuilt following the original installation instructions for DAS-S. While this is going on, the DAS master server will handle all authentication and name service lookups.
If DAS-M is destroyed, or the NIS/Kerberos files are corrupted, the system should be taken offline and DAS-S will handle all of the DAS functions. In a worst-case scenario, you will have to build DAS-M from scratch. Although you can recover the Kerberos database or the NIS maps from DAS-S, we will assume that you have been a good sysadmin and regularly saved your backups to another server or some type of removable media. As long as you have recent backups available, you will be able to restore your authentication and account information. The backup information also includes the backup DAS configuration files that you made during the initial setup. You can use the backup config files for a speedy rebuild.
Assuming that your DAS-M server has to be completely rebuilt, you will follow all of the installation instructions except for the creation of the Kerberos database. Here are the step-by-step instructions for restoring from the following backup files, which were located on floppy disk:
Remember that the original location for these files on DAS-M is the /home/Backups directory, where the files are automatically saved once an hour.
Step 1: Extract the backups to the new DAS-M
[root@das-m Backups]# mount /mnt/floppy [root@das-m Backups]# cd /mnt/floppy [root@das-m floppy]# ls -l total 4 -rwxr-xr-x 1 root root 2355 Nov 10 09:50 313-03.das-m.kerb5.gz -rwxr-xr-x 1 root root 1362 Nov 10 09:50 313-03.das-m.nis.tar.gz [root@das-m floppy]# cp -v 313*gz /root `313-03.das-m.kerb5.gz' -> `/root/313-03.das-m.kerb5.gz' `313-03.das-m.nis.tar.gz' -> `/root/313-03.das-m.nis.tar.gz' [root@das-m floppy]# cd /root [root@das-m root]# gunzip 313*gz [root@das-m root]# ls -al 313* -rwxr-xr-x 1 root root 6068 Nov 10 09:58 313-03.das-m.kerb5 -rwxr-xr-x 1 root root 10240 Nov 10 09:58 313-03.das-m.nis.tar [root@das-m root]# tar -xvf 313-03.das-m.nis.tar group.nis hosts.nis passwd.nis
Step 2: Restore the /etc/passwd, /var/yp/hosts, and /etc/group files
This assumes that you have already created the non-root local account with the same UID and given that account a password. Whenever we manipulate /etc/group or /etc/passwd, we need to be very careful. We will first backup those two files and we will make sure to keep a root terminal or window open while we are testing this.
[root@das-m root]# cp -v /etc/passwd /etc/passwd.org `/etc/passwd' -> `/etc/passwd.org' [root@das-m root]# cp -v /etc/group /etc/group.org `/etc/group' -> `/etc/group.org' [root@das-m root]# cp -v /root/hosts.nis /var/yp/hosts `/root/hosts.nis' -> `/var/yp/hosts' [root@das-m root]# pico passwd.nis [root@das-m root]# cat passwd.nis >> /etc/passwd [root@das-m root]# pico group.nis [root@das-m root]# cat group.nis >> /etc/group [root@das-m root]# ls -al /etc/passwd -rw-r--r-- 1 root root 2066 Nov 10 10:19 /etc/passwd [root@das-m root]# ls -al /etc/group -rw-r--r-- 1 root root 701 Nov 10 10:19 /etc/group
Once this is done, you can run the nisupdate script or the make -C /var/yp command to update the NIS maps. Make sure that you can still login as root and as the non-root local user before closing the root terminal.
[root@das-m root]# nisupdate make: Entering directory `/var/yp' gmake: Entering directory `/var/yp/kerb.org' gmake: `ypservers' is up to date. gmake: Leaving directory `/var/yp/kerb.org' gmake: Entering directory `/var/yp/kerb.org' Updating passwd.byname... Updating passwd.byuid... Updating group.byname... Updating group.bygid... Updating hosts.byname... Updating hosts.byaddr... gmake: Leaving directory `/var/yp/kerb.org' make: Leaving directory `/var/yp'
Step 3: Restore the Kerberos database
[root@das-m root]# /etc/init.d/krb5kdc stop Stopping Kerberos 5 KDC: [ OK ] [root@das-m root]# /etc/init.d/kadmin stop Stopping Kerberos 5 Admin Server: [ OK ]
[root@das-m root]# /usr/kerberos/sbin/kdb5_util load /root/313-03.das-m.kerb5
You can use the -verbose flag after kdb5_util to see all of the principals that are being restored. If you look at the /var/kerberos/krb5kdc directory, you will see the restored database files. They start with "principal":
[root@das-m root]# ls -l /var/kerberos/krb5kdc total 44 -rw------- 1 root root 51 Oct 20 09:38 kadm5.acl -rw------- 1 root root 22 Sep 9 14:34 kadm5.acl.org -rw------- 1 root root 244 Sep 22 10:06 kadm5.keytab -rw------- 1 root root 326 Sep 22 09:51 kdc.conf -rw------- 1 root root 694 Sep 9 14:35 kdc.conf.org -rw------- 1 root root 16384 Nov 10 10:41 principal -rw------- 1 root root 8192 Nov 10 10:41 principal.kadm5 -rw------- 1 root root 0 Nov 10 10:41 principal.kadm5.lock -rw------- 1 root root 0 Nov 10 10:41 principal.ok
Recreate the stash file, using the same password that you used when you created the original Kerberos principals' database. Without the password, the database is useless. Here is how you do it:
[root@das-m root]# /usr/kerberos/sbin/kdb5_util stash kdb5_util: Cannot find/read stored master key while reading master key kdb5_util: Warning: proceeding without master key Enter KDC database master key: [root@das-m krb5kdc]# ls -al /var/kerb*/k*/.k5* -rw------- 1 root root 30 Nov 10 10:53 /var/kerberos/krb5kdc/.k5.KERB.ORG
Since this is a new install, you will also need to extract the host key into the keytab again for DAS-M:
[root@das-m sbin]# kadmin.local Authenticating as principal root/admin@KERB.ORG with password. kadmin.local: ktadd host/das-m.kerb.org Entry for principal host/das-m.kerb.org with kvno 5, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5.keytab. Entry for principal host/das-m.kerb.org with kvno 5, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/etc/krb5.keytab.
Step 4: Bring DAS-M back online and test
Now that you have restored the database and recreated the stash file, you should be able to start up your Kerb5 services and test the system:
[root@das-m krb5kdc]# /etc/init.d/krb5kdc start Starting Kerberos 5 KDC: [ OK ] [root@das-m krb5kdc]# /etc/init.d/kadmin start Starting Kerberos 5 Admin Server: [ OK ] [root@das-m krb5kdc]# kadmin -p super/admin Authenticating as principal super/admin with password. Enter password: kadmin: listprincs K/M@KERB.ORG betty@KERB.ORG emery-k@KERB.ORG host/das-m.kerb.org@KERB.ORG host/das-s.kerb.org@KERB.ORG host/labdemo2.kerb.org@KERB.ORG justo-k@KERB.ORG kadmin/admin@KERB.ORG kadmin/changepw@KERB.ORG kadmin/history@KERB.ORG kingk-k@KERB.ORG maple-k@KERB.ORG rush-k@KERB.ORG oak-k@KERB.ORG hatchet-k@KERB.ORG kitty@KERB.ORG krbtgt/KERB.ORG@KERB.ORG super/admin@KERB.ORG user/admin@KERB.ORG wiang-k@KERB.ORG kadmin: quit [root@das-m krb5kdc]# kinit -p kitty Password for kitty@KERB.ORG: [root@das-m krb5kdc]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: kitty@KERB.ORG Valid starting Expires Service principal 11/10/03 10:58:17 11/10/03 20:58:17 krbtgt/KERB.ORG@KERB.ORG Kerberos 4 ticket cache: /tmp/tkt0 klist: You have no tickets cached [root@das-m krb5kdc]# kdestroy
You will now want to make sure that user "kitty" can login from a DAS client, and that replication is working between the DAS-M and DAS-S.
The first thing to determine is whether or not the failure is a network failure. If it is not a network failure, then you will need to notify your users and begin restoring/rebuilding your DAS servers. In the meantime, you may need to reconfigure the DAS clients through the /etc/nsswitch.conf file and the various /etc/pam.d config files to not attempt to use NIS or Kerberos. Other temporary changes may be necessary, as well.
Restoring critical authentication and user info is relatively simple. It all depends on how well you implemented your backup procedures. If you are worried about disaster recovery, you will want to have offsite backup storage and good documentation. It would be a shame to have a great backup system but have no idea what the database master key is!
7.1 :: DAS Disaster Discovery
|| Prev | Next | Index ||