"The Linux Gazette...making Linux just a little more fun!"

Locked Out of His Mailserver

From Henry A. Lee on Fri, 04 Dec 1998

I am having trouble logging into my Linux mailserver, as any of my users or as ROOT. All passwords are incorrect. I had to bring all my users up on WinNT / Exchange box yesterday to get the email rolling again. Do you know of ANY way to hack the box?

I have about 15 hours of mail that I need to get off the box, and without being able to login, I can't forward it to the new server.

I can't login at the server itself, can't telnet into it, but I can FTP SOME files from it and can maybe get some files back to it. Looking at the PASSWD and PASSWD- files in a text editor, seem fine. Any suggestions would be immensely appreciated.

Thanks for your time,
Henry

I don't know what's caused your inability to log in. It sounds like your /etc/passwd file might have been converted to shadow format ('pwconv' or similar utility) while your authenticating utilities and services aren't shadow capable. However that is only one of several possibilities (the passwd file could be corrupt, it's permissions could be wrong, you might have missing or corrupt PAM modules, etc).

[ I've seen corrupted shadow-passwd files prevent logins before; in both cases, there was the wrong number of colons (:) on a line, and everyone after that couldn't get in. If you managed to break the first line, that would prevent root getting in. -- Heather ]

As for fixing the problem or "hacking the box" as you put it. If you have physical access to the system it is trivial to "hack into" it. Normally this can be done by using the [Ctrl]+[Alt]+[Del] (PC "nerve pinch" or "three finger salute"), to reboot the system (most Linux systems have an entry in their /etc/inittab that looks something like:


# what to do when CTRL-ALT-DEL is pressed
ca::ctrlaltdel:/sbin/shutdown -r -t 4 now

... which allows the 'init' process (the grandfather of all processes) to respond to this console event.

Failing that you can wait for a bit while there is minimal disk activity and reset or power cycle the system.

As you reboot you wait until the LILO boot load prompt is display and type in a command like:

linux init=/bin/sh

... (assuming that you have a boot stanza named "linux" --- hit the [Tab] key at that prompt for a list of those).

This passes a parameter to the kernel which forces it to use an alternative to the 'init' program (a copy of the shell in this case). From there you might need to mount the /usr filesystem (assuming that the system follows professional conventions rather than common Linux installation defaults). Then you can issue the '/usr/bin/passwd' command to set a new root password.

If that doesn't solve the problem you can edit the passwd file. if necessary remove everything but the entry for root --- don't put any comments or blank lines in this file! (Obviously you should save a copy if you're going to try that).

If that still doesn't work, and if there are no clues in your logs (look at /var/log/messages for hints), then you have some other troubleshooting to do.

At that point it might be best to just call a consultant for some voice support. You don't provide enough information for me to explain the next troubleshooting without writing a whole book (and I'm already working on one).

I can do phone support or you can look for anyone in the Consultants HOWTO. (Considering that you have data on this system that you don't want to lose, and that it sounds like you don't have any backups, I wouldn't suggest too much experimentation and learning curve climbing while trying to recover from this situation).

If you have another Linux or Unix system anywhere else on your network --- one with 'sendmail' properly installed (assuming that the affected system was also running 'sendmail') it's possible to copy all of the files from /var/spool/mqueue to some arbitrary directory on the working system (from the ailing one, obviously). Then you can run a command like:

/usr/lib/sendmail -v -q -O QueueDirectory=/tmp/q

... to tell sendmail to verbosely (-v) make a processing pass through the queue (-q) with the option (-O) to over-ride the QueueDirectory set to some place like /tmp/q (or where ever you ftp'd those df and qf files to).

As for the user mail that's already been delivered to "mbox" files under /var/spool/mail, you can copy those to another system and append them to file under the /var/spool/mail on the new system. To avoid possible corruption you'd want to disable the sendmail and popd (etc) processing on the new system before trying this.

The easiest way to do that is to shut the system down to single user mode after you've copied (ftp'd) all of the mbox files (inbox folders) to the system.

Naturally you'll need to create user accounts that correspond to each of these users from the old system, and you'll need to ensure that the ownership and permissions of each mbox file are set properly.

There are other ways to do this. However they depend on the situation and/or involve some more complicated command lines then I'd want you to try without a thorough understanding of how they work.

In the 'procmail' man pages there is an example of a script to "postprocess" an mbox. It would be possible to use something like that to "break apart" each mbox file and resend it to the original recipient.

If your users were using MH, 'elm' or 'pine' (or most any Unix/Linux mail reading package) they could copy an mbox file to any convenient place and either treat it as a folder ('elm -f') or "incorporate" it into their MH folders using the 'inc' command. These users should either know how to do that, or read the man pages for their favorite mail user agent for details.

If you do hire a consultant, look for one that will provide you with some good tutorial/mentorship on Linux and consider having him or her help you prepare a comprehensive "Recovery Plan and Disaster Procedures" package. This will be vital to your company's IS/IT regardless of what OS or platform you choose for your future needs.

My phone number can be found on my web pages:

Starshine Technical Services

http://www.starshine.org

... I normally don't advertise my consulting services in this column, and I don't plan to do so often. However, there are situations where the most prudent advice I can give is: "Call someone to walk you through this."

As I say, you are encouraged to find a Linux consultant that is local to you. Look in the Consultant's HOWTO at:

http://metalab.unc.edu/LDP/HOWTO/Consultants-HOWTO.html

... You can also find a wealth of help at any Linux Users Group (LUG) and there are a couple of "Lists of LUG's" that I've listed in previous columns. There's even a Users Group HOWTO at:

http://metalab.unc.edu/LDP/HOWTO/User-Group-HOWTO.html

... which includes links to the three biggest lists of LUG's.

I wish I could say: "Look for the union label" when considering entrusting your system's integrity to a consultant or volunteer. However, there is no widely recognized certification for sysadmin's yet. There isn't even a "better business bureau" of sysadmins and/or consultants. As a member of SAGE (the SysAdmin's Guild) I'm involved in an ongoing effort to provide some such process. However it's a contentious issues, and Unix sysadmins are a contentious lot(*). I'll be continuing this work while I'm in Boston next week at the annual LISA conference.

• (Certainly your chances of getting a competent and experienced sysadmin are better if you find someone who went to the effort to join SAGE, or at least has cogent reasons for not doing so; and they are drastically diminished if you're talking about someone who's never heard of USENIX or SAGE).

Good luck.

Copyright © 1999, James T. Dennis
Published in The Linux Gazette Issue 36 January 1999

						a	b	c	1	2	3	4	5	6	7	9	10	11	12
						15	16	18	19	20	21	22	23	24	25	26	27	28
						29	31	32	33	34	35	36	37	38	39	40	41	42	44
45	46	47	48	49	50	51	52	53	54	55	56	57	60	61	62	63	64	65	66
67	69	72	76	77	78	79	80	81	82	84	85	86	87	91	94	95	96	97	98