LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 06-04-2001, 05:22 PM   #1
elucid
LQ Newbie
 
Registered: Apr 2001
Location: Georgia
Posts: 8

Rep: Reputation: 0
Exclamation


I had a problem today with my sendmail...none of my clients were able to check their mail, including us here in the office. Anyways, I figure restarting the service would correct this problem. No. Once I restarted the service it said that it started except it stated:
"warning: /var/lock/subsys/????: no space left on device!, So begining to panic i rebooted the machine and upon reboot, all of my services began getting the same error, httpd, smb, etc...what the hell is going on, i cleared my tmp directory and that seemed to fix it, however it quickly began again. I getting to my last nerve now.

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 1.9G 1.0G 839M 55% /
/dev/sda5 15G 12G 2.5G 83% /home
/dev/sda6 15G 13G 1.2G 92% /usr

somebody please help me out! thanks
 
Old 06-04-2001, 11:17 PM   #2
mcleodnine
Senior Member
 
Registered: May 2001
Location: Left Coast - Canada
Distribution: s l a c k w a r e
Posts: 2,731

Rep: Reputation: 45
Did you check to see that all of the mount points are there? ie: what is in your /etc/fstab file? If it's a mail server there's a good chance your /var or /var/spool is on its own partition. Did you set the box up yourself or was the admin job handed off to you?
 
Old 06-04-2001, 11:45 PM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
run a "df -i" on ure partition and itll tell u if ure out of inodes. unfortunately u cant just add inodes like water. u'll need to reformat that partition & specify another blocksize.
 
Old 06-05-2001, 01:47 AM   #4
DMA
LQ Newbie
 
Registered: Jun 2001
Posts: 3

Rep: Reputation: 0
why dont you call joey. he can probably help you.
 
Old 06-05-2001, 09:13 AM   #5
DrDebian
LQ Newbie
 
Registered: May 2001
Posts: 23

Rep: Reputation: 15
Quote:
Originally posted by unSpawn
run a "df -i" on ure partition and itll tell u if ure out of inodes. unfortunately u cant just add inodes like water. u'll need to reformat that partition & specify another blocksize.
That would indeed be a possible approach. I'd recommed to switch to ReiserFS while he's at it, saving tons of space with all those small files and speeding up things as well.
 
Old 06-05-2001, 10:46 PM   #6
elucid
LQ Newbie
 
Registered: Apr 2001
Location: Georgia
Posts: 8

Original Poster
Rep: Reputation: 0
Question crazy


BEFORE:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 256512 256512 0 100% /
/dev/sda5 4173824 51105 4122719 1% /home
/dev/sda6 4098048 136434 3961614 3% /usr

now this is crazy. I figure out that it is actually a program called interchange an e-commmerce server appending hundreds of logs in a crazy format, something like error~1.log.gz.1.gz.2.gz.3...., the directory itself is impossible to manipulate. you can't remove files or copy them, both commands with give you an error like "too many arguments..." it's kinda crazy, but I was able to successfully recover all the inodes by using midnight comm to just remove the entire directory but it went on forever and slowed the system down enourmously, so i stop it. the problem that i have now is i can't really do anything with the directory. Crazy stuff

AFTER:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 256512 95882 160630 37% /
/dev/sda5 4173824 51105 4122719 1% /home
/dev/sda6 4098048 136434 3961614 3% /usr


[Edited by elucid on 06-06-2001 at 12:14 AM]
 
Old 06-06-2001, 01:03 AM   #7
mcleodnine
Senior Member
 
Registered: May 2001
Location: Left Coast - Canada
Distribution: s l a c k w a r e
Posts: 2,731

Rep: Reputation: 45
Re: crazy

Quote:
Originally posted by elucid

BEFORE:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 256512 256512 0 100% /
/dev/sda5 4173824 51105 4122719 1% /home
/dev/sda6 4098048 136434 3961614 3% /usr

now this is crazy. I figure out that it is actually a program called interchange an e-commmerce server appending hundreds of logs in a crazy format, something like error~1.log.gz.1.gz.2.gz.3...., the directory itself is impossible to manipulate. you can't remove files or copy them, both commands with give you an error like "too many arguments..." it's kinda crazy, but I was able to successfully recover all the inodes by using midnight comm to just remove the entire directory but it went on forever and slowed the system down enourmously, so i stop it. the problem that i have now is i can't really do anything with the directory. Crazy stuff

AFTER:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 256512 95882 160630 37% /
/dev/sda5 4173824 51105 4122719 1% /home
/dev/sda6 4098048 136434 3961614 3% /usr


[Edited by elucid on 06-06-2001 at 12:14 AM]
Interchange has an option to cleanup cached pages - i think it's in the sessions directory. Where Interchange lives. I've been wrestiling with it for about 2 weeks and the friday deadline and the current workload could create enough input to fuel an improbability drive through at least a couple of galaxy tours.

Interchange has a nagging bit at the end of the 'makecat' script about cleaning this up. grep it and be wiser than me.

Cheers

(edit)

And MOST DEFINATELY make use of reiserfs. Highly recommended for your situation.
 
Old 06-06-2001, 08:50 AM   #8
elucid
LQ Newbie
 
Registered: Apr 2001
Location: Georgia
Posts: 8

Original Poster
Rep: Reputation: 0
Question i don't think it's cached pages

construct.error.log.1.gz.2.3.gz...

I don't know very much about interchange and it was being setup by another admin in our office. I've been reading the docs for it but it isn't getting me anywhere.
 
Old 06-06-2001, 09:18 AM   #9
mcleodnine
Senior Member
 
Registered: May 2001
Location: Left Coast - Canada
Distribution: s l a c k w a r e
Posts: 2,731

Rep: Reputation: 45
Re: i don't think it's cached pages

Quote:
Originally posted by elucid
construct.error.log.1.gz.2.3.gz...

I don't know very much about interchange and it was being setup by another admin in our office. I've been reading the docs for it but it isn't getting me anywhere.
Sorry - i meant the cached _user_ sessions in the the catalog directories. they can get nasty if the cgi goes thunk. However this is not the problem you are facing.


If the directories are totally bloated and contain a lot of files you can expect to wait a while for the cleanup. Kill interchange until you can fix your space problem and find out what's going on in the catalogs that are generating all those error logs. (reading one or two of them might help)
 
Old 06-07-2001, 09:36 AM   #10
elucid
LQ Newbie
 
Registered: Apr 2001
Location: Georgia
Posts: 8

Original Poster
Rep: Reputation: 0
Exclamation the files are completely empty

That the thing, all the log files are empty in the directory giving me the problem. You open it in vi and poof...nothing. The files exist, just nothing in them. I had already killed the interchange daemon so it would stop appending log files. And since they are empty, it's not giving me much to go by...
 
Old 06-07-2001, 03:05 PM   #11
mcleodnine
Senior Member
 
Registered: May 2001
Location: Left Coast - Canada
Distribution: s l a c k w a r e
Posts: 2,731

Rep: Reputation: 45
Re: the files are completely empty

Quote:
Originally posted by elucid
That the thing, all the log files are empty in the directory giving me the problem. You open it in vi and poof...nothing. The files exist, just nothing in them. I had already killed the interchange daemon so it would stop appending log files. And since they are empty, it's not giving me much to go by...
Right. But depending on how that partition was formatted you can chew up space with 'zero' size files as each file will take up at least [inode density] space. If it was formatted with the stadard setup it should be 4096 bytes, so even an empty file will still use 4K of space. I would 'rm -r <log_file_directory>' and wait until it completes. then re-create the directory with the proper permissions. That should at least clear up that problem.

Use reiserFS.

What tasks will the server be perorming? Looks like httpd and mail. It can bog down pretty quick with a few users. You might consider getting another couple (well 3 anyway) of SCSI drives so you can gain a little bit of performance by using different drives for some of the busier partiions. (Just my $.02)

At the very least I would do the following: (you should be able to do this while still keeping your root on the original drive.). Pleae keep in mind that this is the simplified version and that you should backup before you attmept anything like this.

1) get reiserfs working (at the kernel level)
2) Take some time to re-examine the tasks that server will be performing and decide where to put the 'busier' partitions. (/var/spool, /usr/local (or wherever your www pages are) and a partition for the interchange catalogs.) IMHO you should look at using a total of four drives. Partition and format the new partitions wih reiserfs.
3) Mount one of the partitions. Copy the desired directories to their new partition. I use this from within the folder I want to copy - 'find . -xdev | cpio -pm /<where_you_mounted_your_new_parttion>'. umount and mount the next new partition. Lather, rinse repeat.
4) change your fstab to mount the new partitions to the proper dirs.

It's a little more difficult to get your root fs to reiserfs as you'll have to do some extra shuffling. It's possible and not a bad idea.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Not enough free space on hard drive with 50g of free space??? auoq SUSE / openSUSE 5 10-13-2004 08:21 PM
Not enough space for root directory on a drive with 50g free space??? auoq Linux - Newbie 1 10-13-2004 12:44 PM
Samba: 70 Gb free space, samba reports no space left? HansLuijten Linux - Networking 1 08-18-2004 04:40 PM
"no space left on device" - But df shows free space! monita Linux - General 7 03-30-2004 01:14 PM
Formating free space: WinXP pro and RH9 dualboot with free space on 3rd drive Vermicious Linux - General 2 03-22-2004 05:10 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:17 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration