Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved]XFS filesystem stuck during libreoffice emerge
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Sat Aug 06, 2011 10:34 am    Post subject: [Solved]XFS filesystem stuck during libreoffice emerge Reply with quote

I use XFS for all my filesystems, including /tmp

In fact, my system uses a raid1 with LVM and the /tmp filesystem in LVM.

I have noticed, only once or twice, that under heavy load, my /tmp filesystem seems to 'wedge'.

I am using kernel 2.6.39-gentoo-r3

edit: It has wedged twice during the unpacking of impress in libroffice. It appears to happen during the recursive chmod.

XFS Status updates seems to indicate a change in XFS with regards to metadata processing, I suspect a bug.
I have created a gentoo bug, for want of a better place to put it.
_________________
...Lyall


Last edited by lyallp on Wed Aug 10, 2011 11:16 am; edited 1 time in total
Back to top
View user's profile Send private message
PanzerKanzler
Guru
Guru


Joined: 04 Dec 2005
Posts: 376
Location: Belgravia, London

PostPosted: Mon Aug 08, 2011 12:05 pm    Post subject: Reply with quote

What do you mean 'wedge'? Does it slow down, or does it stop completely?

XFS is known to choke when dealing with many small files, for instance untarring a kernel source. Try untarring a kernel source onto XFS, and then try untarring it onto EXT4, and you will know what I mean ;)
_________________
If you've got nothing nice to say, you're probably not alone...
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Mon Aug 08, 2011 12:34 pm    Post subject: Reply with quote

By wedge, I meant jammed, stuck, stopped. The entire filesystem in question just stops.
In this case, anything that hits /tmp just stops dead.

I have subsequently done the same build on another 64 bit Gentoo, which is not raid1 but does use LVM, etc. I admit, I have forgotten to check the kernel version on that system, I will check it out.

I have previously built libreoffice on this system, the only thing that has changed has been a kernel upgrade.

I may consider a kernel downgrade to pre-2.6.39, which is when the XFS updates that are of concern to me, happened.
_________________
...Lyall
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Mon Aug 08, 2011 3:15 pm    Post subject: Reply with quote

I haven't found any problems with small files, but I have found that XFS sometimes goes home for lunch when it gets near full, like 95%+. Never had it freeze though. Although, I use tmpfs for /tmp.

For tuning with recent kernels (including .39) I find mounting with a large log buffer helps (logbsize=256k) as do the usual nodiratime and noatime/relatime. If you have a UPS that will handle an orderly shutdown and your system is stable (probably not now if you're seeing freezes) nobarrier will give you a large performance boost.

Creating a filesystem with too many allocation groups is not a good thing. The defaults used to be way too big for large filesystems, at least for a desktop... I'm not sure where they're at now. But create as many allocation groups as you think you will have concurrent competing accesses to the filesystem; 4-6 seems to be a good rule of thumb. Each time you grow the filesystem using LVM and xfs_growfs, it will add another allocation group.

On the other side, too few allocation groups could be what's causing your /tmp to hang.

Lastly, the defaults for the log size used to be too small.

Here are the parameters for a couple of filesystems from my laptop. /var really has too many allocation groups, that's because I planned badly and grew it a few times.

Code:
$ xfs_info /
meta-data=/dev/root              isize=256    agcount=4, agsize=610468 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2441872, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ xfs_info /var
meta-data=/dev/mapper/vg-var     isize=256    agcount=13, agsize=393216 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=4980736, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Tue Aug 09, 2011 11:06 am    Post subject: Reply with quote

I use /tmp for my portage temp.
I don't use tmpfs because I do not have enough memory (8g) to be able to build libreoffice.
The filesystem is only a few weeks old and has never been extended.

Code:
root@lyalls-pc ~
# xfs_info /tmp
meta-data=/dev/mapper/vg-tmp     isize=256    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@lyalls-pc ~
#

_________________
...Lyall
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Tue Aug 09, 2011 1:58 pm    Post subject: Reply with quote

The log is only 2560 * 512 = 1280K, that's probably what's causing your problem. 1M isn't enough for compiling things... it would be fine for /tmp in normal usage, but not for /var/tmp.

Try building it with mkfs.xfs -l size=128m, and mount it with logbsize=256k

EDIT: Also, if your machine is has lots of cores, you should use one allocation group per core... for an i7 use mkfs.xfs -d agcount=8. Assuming you've configured your portage to use all the cores, the workload will be heavy parallel writes; having enough allocation groups will make sure that your processes never have to wait for each other when allocating disk space. This is one case where not enough allocation groups is a problem.
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Wed Aug 10, 2011 12:05 am    Post subject: Reply with quote

Awesome, I will give it a try - I was planning to re-format the partition anyway.
As a side note, I attempted again last night, it wedged during a 'mv'.
_________________
...Lyall
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Wed Aug 10, 2011 12:06 am    Post subject: Reply with quote

Just to clarify my last post (written too late, zzz sorry, not thinking clearly)

A larger log will dramatically increase the speed of removing lots of files (like a portage build tree). If you have lots of cores, the 4 allocation groups might be causing you a performance hit. But these are just performance tuning, you shouldn't be seeing hangs.

One case where you would definitely see hangs is if the filesystem fills up. I need 12G+ to build libreoffice on a 64 bit system.

The best resources for setting up and tuning XFS correctly (you need to watch out with RAID too) are:
the XFS FAQ and the SGI guide

If you're mounting dedicated /tmp partition, mount it with nobarrier... you'll get a performance boost, at the price of slightly more risk if you lose power/crash... but seeing as it's a temporary filesystem that shouldn't matter.

Note that the barrier/nobarrier thing is why people often get the wrong idea about XFS performance. XFS turns barriers on by default for safety. ext3 turns them off for speed, on the other hand ext uses ordered writes whereas XFS uses something more like writeback. More info on this is in the man page for mount, under the mount options for ext3.

Anyway that's my heads up on XFS, whether you need it or not :wink:. I've found XFS to be fast and rock-solid over a number of years and I've been using 2.6.39 myself without an issue, but these hangs of yours are worrying. Please post to let me know how you go!
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Wed Aug 10, 2011 12:18 am    Post subject: Reply with quote

There's really something wrong with my brain today, my fingers seem to be talking through my ass. You shouldn't be seeing hangs, even if the filesystem is full. Going back to re-read your original post...

Hmmmm. Well, a bigger log would help a recursive chmod. But still, wedging completely is not on.

I'm going to check if I've built libreoffice myself since 2.6.39, if not I'll give it a go.
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Wed Aug 10, 2011 11:15 am    Post subject: Reply with quote

Awesome, problem appears to be solved.
My /tmp filesystem is/was 20G, so there was plenty of space, I did not run out of room during my build.
I re-created my /tmp filesystem, as suggested, now my xfs_info is
Code:
lyall@lyalls-pc:~
$ xfs_info /tmp
meta-data=/dev/mapper/vg-tmp     isize=256    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Note the increase in blocks= in the log section.

I have also updated my /etc/fstab entry
Code:
# Expendable, if power fail, possibility of loss of data, but no barriers improves throughput.
LABEL=tmp      /tmp      xfs      defaults,noatime,logbsize=256k,barrier=0   1 2
although, at the time I did the build, the barrier=0 option had not been applied.

So, it appears my xfs log had filled.

Whilst I can understand it 'wedging', it's a pity it did not fail for some file operation, thus allowing the whole system to continue, rather than hang. I will update my bug report.



Thanks heaps for this.
_________________
...Lyall
Back to top
View user's profile Send private message
kimmie
Guru
Guru


Joined: 08 Sep 2004
Posts: 531
Location: Australia

PostPosted: Wed Aug 10, 2011 4:01 pm    Post subject: Reply with quote

np - built libreoffice today, works here too :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum