Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Some ext3 Filesystem Tips
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 13, 14, 15  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
RuiP
l33t
l33t


Joined: 15 Jan 2005
Posts: 643

PostPosted: Thu Jan 12, 2006 10:51 pm    Post subject: Reply with quote

XenoTerraCide wrote:
no putting portage in it's own partition is supposed to increase performance. the 1024 was to save space but I wasn't sure how much.

Yes, but what i was point is that you seems to have a portage tree with 1.2G. That's too big for just the portage. You probabilly have your /distfile inside /usr/portage.
The increase in performance will come from not fragmented and well used space optimization on partition, wich is hard to obtain from mixed very small files with large tar.gz and tar.bz2... Move distfile will give you much space then block size change, avoid fragmentation and possibility to choose a diferente filesystem for that partition if you want to try that.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Fri Jan 13, 2006 12:15 am    Post subject: Reply with quote

codergeek42,
After you convinced me to go ext3 on all my Gentoo boxes / partitions, I first tried full journalling for a week or two, then tried data=writeback. I honestly can't discern any difference- both are really good. I guess the key is if one's partitions have a lot of simultaneous reads/writes going on. I might switch back.

BTW, I've been fooling around with the new multiple block allocation ext3 patches. I compiled a new mm3 kernel (which already has them), and also applied them to 2.6.15-ck1 with no problem, and just finished applying them to the new 2.6.15-nitro1 kernel (haven't booted into it yet- had to fix one reject file, and 2 other rejects appeared correct when I looked at them- anyway, nitro1 compiled fine after I added the new ext3 stuff).

Have you ever tried XFS tuned for normal size files? I was reading on that, and it seemed interesting- might test it when I have time.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
codergeek42
Bodhisattva
Bodhisattva


Joined: 05 Apr 2004
Posts: 5142
Location: Anaheim, CA (USA)

PostPosted: Fri Jan 13, 2006 12:47 am    Post subject: Reply with quote

wrc1944 wrote:
Have you ever tried XFS tuned for normal size files? I was reading on that, and it seemed interesting- might test it when I have time.
To be honest, i've never tried XFS for any of my partitions. However, I may give it a try at some point in time on my music/videos storage partition to see how it fares against my current dir_index+journal_data ext3 :)
_________________
~~ Peter: Programmer, Mathematician, STEM & Free Software Advocate, Enlightened Agent, Transhumanist, Fedora contributor
Who am I? :: EFF & FSF
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Fri Jan 13, 2006 1:09 am    Post subject: Reply with quote

codergeek42,
I'll probably go back to dir_index+journal_data ext3 again as opposed to dir_index+ writeback, but my main uncertainty still remains- do I have enough simultaneous reads/writes going on to make it better than writeback?

I guess I'll really need to do my own set of benchmarks to really know anything other than subjective impressions- and I guess the benchmarks are also suspect, depending on which ones you use, and what your real-world computer usage is.

Also- just booted into the new nitro with the added ext3 multiple block allocation ext3 patches included. Works fine- as good as any other 2.6.15-xxx I've played with. Guess I should set up a better testing environment where I can really screw around (not on my main Gentoo box) to make any valid comparisons.

I'm still patiently waiting for a new 2.6.15-archck to test these ext3 patches on- been running archck for a while as my favorite kernel.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Fri Jan 13, 2006 2:21 am    Post subject: Reply with quote

yeah distfiles is in there and I get what ur saying. unfortunately repartitioning is somewhat of a pain. is it not? to resize and add a partition. especially because my extended partition is hda2. and I'm not gonna repartition at this time.
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Fri Jan 13, 2006 3:49 am    Post subject: Reply with quote

Yeah- just how far can we Gentooists go? Maybe it's time for a dedicated tuned XFS partition for distfiles, and another reiserfs partition for the rest of the portage tree. At some point, it surely becomes completely preposterous (unless serious brain exercise is NOT the primary objective). At 61, in my case, it is. :D

At this point, repartioning a bunch of Gentoo boxes is a recreational exercise! :D
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Fri Jan 13, 2006 4:01 am    Post subject: Reply with quote

I have a question at what point did the ext3 thread become the filesystem of choice threadhttps://forums.gentoo.org/viewtopic.php?p=2965333#2965333?
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Fri Jan 13, 2006 4:49 am    Post subject: Reply with quote

XenoTerraCide,
I'm sure you know all this stuff, and they are talking about the same thing- but for others I would think it was because 3-5 years ago, when people first got into Linux, they discovered the default ext2/3 performance was preposterously slow, and they subsequentcely tried reiserfs and saw a serious desktop improvement (as I did temporarily), they switched.
In my case, I was really ignorant, but later on as FS evolved, and thanks to my education in the Gentoo forums and googling, I became wiser about the supposedly non-existent linux fragmentation stuff, especially with reiserfs. :wink: The fact is, every file system fragments- it's just a matter of degree.

As hopefully your own research will teach you, there is no ONE best FS for all circumstances, or distros, or a file system "of choice" to cover all things, and make it simple.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Fri Jan 13, 2006 4:56 am    Post subject: Reply with quote

I agree and although these fs fragment less than say fat32 this is the ext3 tips discussion thread it's really not the place to compare and contrast reiserfs, xfs, jfs, blah blah blah... but then again this isn't my thread. maybe I should let codegeeker decide if he thinks talking about reiser and xfs and what have you is off topic. so I'm gonna but out till we get back to ext3.
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
codergeek42
Bodhisattva
Bodhisattva


Joined: 05 Apr 2004
Posts: 5142
Location: Anaheim, CA (USA)

PostPosted: Fri Jan 13, 2006 5:50 am    Post subject: Reply with quote

XenoTerraCide wrote:
I agree and although these fs fragment less than say fat32 this is the ext3 tips discussion thread it's really not the place to compare and contrast reiserfs, xfs, jfs, blah blah blah... but then again this isn't my thread. maybe I should let codegeeker decide if he thinks talking about reiser and xfs and what have you is off topic. so I'm gonna but out till we get back to ext3.
Thanks, Xeno.

If you want to discuss this stuff, please do so in the filesystem comparison thread linked to earlier. Thanks. :)
_________________
~~ Peter: Programmer, Mathematician, STEM & Free Software Advocate, Enlightened Agent, Transhumanist, Fedora contributor
Who am I? :: EFF & FSF
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Fri Jan 13, 2006 7:26 am    Post subject: Reply with quote

Sorry I went off on a tangent- I agree, and I'll try and keep on a specific topic for the thread I'm posting to. Hope I didn't cause any confusion :oops:
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Fri Jan 13, 2006 3:09 pm    Post subject: Reply with quote

NP codegeeker.
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
Drysh
Apprentice
Apprentice


Joined: 06 Apr 2005
Posts: 203
Location: São Paulo, Brazil

PostPosted: Mon Jan 16, 2006 10:21 pm    Post subject: Reply with quote

codergeek42,
Something to add to the original your tips. I just discovered this, and it's a very god tool.

Making your critical files safe (and setting other special cases)

Don't you hate when you acidently delete a critical file? The ext3 filesystem has a tool to prevent that: attributes. To see the current attribute of a file use "lsattr", to change it use "chattr". To prevent a file from being deleted, you need to set the "i" attribute. Notice that only the superuser may set (and unset) this attribute. Take a look at what the attributes may do:
Code:
man chattr
man lsattr

Notice that:

- To prevent a directory from being deleted, you shouldn't set the "+i" attribute of the directory (that will set all files as undeletable). Create a file in the directory and set it as undeletable, that will prevent the directory itself from being deleted.
Code:
touch .keep
chattr +i .keep


- If you mark a directory for append only (+a), you will be able to create new files, but won't be able to delete them. All files created will also be marked append only (+a).

[end of my tip]

Since you understand much better than me the journal options, please explain why would someone set only part of the filesystem as full journal (+j attribute). You could also take a look at the "+T", it will set the file to the top of the Orlov block allocator. I don't have the knowledge to comment on those.

Another thing I see as a nice application of the attributes is the attributes is to use "+D" (dirsync) and "+S" (sync) combined with a modified sync rate for the whole fs (I was thinking about comit=30 (up to 600 may be good: 10 minutes, more is insane)). I'll have to do some tests: ideas for where to sync are welcome. :)

BTW: The only other useful attribute is "+d" that excludes a file from "dump". The rest is either not implemented or is for internal use.
Back to top
View user's profile Send private message
codergeek42
Bodhisattva
Bodhisattva


Joined: 05 Apr 2004
Posts: 5142
Location: Anaheim, CA (USA)

PostPosted: Mon Jan 16, 2006 10:31 pm    Post subject: Reply with quote

Drysh:

I've not played with the extended attribute and ACL features of Ext2/Ext3, so I don't think it would be wise of me to give such advice on that of which I have very little knowledge. I just resumed classes, so hopefully I'll get a chance soon to learn more these features and see if I can add some of it to my tips.

Thanks for the suggestion, though! :D
_________________
~~ Peter: Programmer, Mathematician, STEM & Free Software Advocate, Enlightened Agent, Transhumanist, Fedora contributor
Who am I? :: EFF & FSF
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Tue Jan 17, 2006 12:22 pm    Post subject: Reply with quote

Hello, what would be best settings for /var/tmp with ccache in /var/tmp/.ccache or would it be better to use reiserfs?

This will be on lvm2.

cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
codergeek42
Bodhisattva
Bodhisattva


Joined: 05 Apr 2004
Posts: 5142
Location: Anaheim, CA (USA)

PostPosted: Tue Jan 17, 2006 4:32 pm    Post subject: Reply with quote

carpman wrote:
Hello, what would be best settings for /var/tmp with ccache in /var/tmp/.ccache or would it be better to use reiserfs?

This will be on lvm2.

cheers
Considering that such a partition is read from and written to a lot, generally, I think it would be wise to put Ext3 with full journalling and dir_index as my tips show. :)
_________________
~~ Peter: Programmer, Mathematician, STEM & Free Software Advocate, Enlightened Agent, Transhumanist, Fedora contributor
Who am I? :: EFF & FSF
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Tue Jan 17, 2006 5:26 pm    Post subject: Reply with quote

codergeek42 wrote:
carpman wrote:
Hello, what would be best settings for /var/tmp with ccache in /var/tmp/.ccache or would it be better to use reiserfs?

This will be on lvm2.

cheers
Considering that such a partition is read from and written to a lot, generally, I think it would be wise to put Ext3 with full journalling and dir_index as my tips show. :)


cheers, that is what i went with in the end.
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Tue Jan 17, 2006 5:31 pm    Post subject: Reply with quote

codergeek42 wrote
Quote:
Considering that such a partition is read from and written to a lot, generally, I think it would be wise to put Ext3 with full journalling and dir_index as my tips show.
I may be misunderstanding this, so my conclusion might be wrong, but I thought the Dan Robbins article about data_journal being the performance king was only where much reading and writing is to be done simultaneously.

If that's true, wouldn't we need to know to which partitions this actually applies, and not just which partitions are simply "read from and written to a lot" ? This seems critical to me.

When I first switched to ext3 and used all the tuning tweaks, I did indeed use data_jounal, but then after a week or two since I didn't really know which partions were actually engaged in a lot of simultaneous read/write activity, I decided data=writeback would probably be the better performance choice. Accordingly, I changed them all to data=writeback, and can't really say I've noticed any difference.

In other words, I guess we (at least I do) need to know which partitions and in what cases there is much reading and writing being done simultaneously. In my case, I generally do most of my compiling at night, with no other activity whatsoever- would this make a difference?

Another point- how do we actually define and/or know what activities are indeed consisting of so-called "simultaneous read/write" activity? For example, would sequentially extracting lots of tar.bz2 files qualify? Would a "data only" storage partition where data is only being saved to (or read from) at any given time NOT qualify?

I just don't really know, but it seems to me this knowledge would be an important factor to completely understand before we automatically set all our partitions to data_journal, especially considering most other info sources suggest data=writeback is best for performance.

Any thoughts or corrections on these matters is greatly appreciated.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental


Last edited by wrc1944 on Tue Jan 17, 2006 5:43 pm; edited 2 times in total
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Tue Jan 17, 2006 5:39 pm    Post subject: Reply with quote

data=writeback I think is very similar to data=journal. accept that it commits the information to the drive before the journal. where as data=journal commit's it to the journal first. once again I may be wrong... using data writeback could 'cause problems if you had a power loss or had to do a hard reset.
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
carpman
Advocate
Advocate


Joined: 20 Jun 2002
Posts: 2202
Location: London - UK

PostPosted: Tue Jan 17, 2006 6:01 pm    Post subject: Reply with quote

XenoTerraCide wrote:
data=writeback I think is very similar to data=journal. accept that it commits the information to the drive before the journal. where as data=journal commit's it to the journal first. once again I may be wrong... using data writeback could 'cause problems if you had a power loss or had to do a hard reset.



I alway hear this phrase: "could cause problems if you had a power loss or had to do a hard reset".

To be honest i don't care if /usr/portage /var/tmp and other non data criticle partition have power outage, what i would like is performance so if we can decide which are best suited to data=journal and data=writeback this would be helpful in context of this thread.

cheers
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Tue Jan 17, 2006 6:07 pm    Post subject: Reply with quote

you mean this part of the thread... honestly I couldn't tell you... I doubt there really is much of a difference... if you read back on this thread... the difference was discussed earlier... perhaps you should do that. or you could test them both for us and bring us benchmarks?
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3271
Location: Gainesville, Florida

PostPosted: Tue Jan 17, 2006 7:23 pm    Post subject: Reply with quote

Two days ago I switched from ext3 with data=writeback to ext2 with dir_indexing for my /mnt/portage partition, as suggested by 6thpink on another related thread. I'm also considering it for my /var and /tmp partitions. The reasoning is that even having a journalling file system with it's inherent overhead might be overkill and really unnecessary for such specialized partitions, and it also avoids the "data=journal or data=writeback" choice dilemma. Given that ext2/3 benchmarks seem to indicate essentially equal performance, this seems reasonable. I guess I'll know more after observing portage activity for a few more emerges.

I have also placed the distfiles directory with larger files on another ext3 dir_index data=writeback partition, as it seems that in that directory files are either being written to, read from, or deleted, but not simultaneously. Of course we are talking about extremely fine tuning of the ext2/3 file system as related to Gentoo usage here, and shouldn't expect drastic performance differences, one way or the other, especially on a modern fast system. That being said, I like to think every little % of performance increase helps.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.30-r2, gcc-9.2.0 kernel-5.4.1-gentoo USE=experimental
Back to top
View user's profile Send private message
XenoTerraCide
Veteran
Veteran


Joined: 18 Jan 2004
Posts: 1418
Location: MI, USA

PostPosted: Wed Jan 18, 2006 12:12 am    Post subject: Reply with quote

I'm trying to put /usr/portage and /usr/portage/distfiles in their own partitions... however I keep getting out of space errors df show's otherwise...
Code:
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              21G  8.1G   12G  42% /
udev                  379M  236K  378M   1% /dev
/dev/sda6             942M  176M  719M  20% /usr/portage
/dev/sda7             1.9G   33M  1.8G   2% /usr/portage/distfiles
/dev/sda4              78G   52G   27G  66% /mnt/winntfs
/dev/hda1              38G  9.9G   28G  27% /mnt/windows
shm                   379M     0  379M   0% /dev/shm
I made a seperate thread in portage and programming... I don't get it... https://forums.gentoo.org/viewtopic-t-424151.html I'm hoping to find out why this is happening. I haven't deleted old portage yet. :D. I just mounted the the new partition's over it. oh /usr portage is ext2 blocksize 1024 and distfiles is ext3 blocksize 4096.
_________________
I don't hang out here anymore, try asking on http://unix.stackexchange.com/ if you want my help.
Back to top
View user's profile Send private message
RuiP
l33t
l33t


Joined: 15 Jan 2005
Posts: 643

PostPosted: Wed Jan 18, 2006 12:25 am    Post subject: Reply with quote

see suggestion by ruben and my posts on previous page.
Back to top
View user's profile Send private message
Drysh
Apprentice
Apprentice


Joined: 06 Apr 2005
Posts: 203
Location: São Paulo, Brazil

PostPosted: Wed Jan 18, 2006 12:37 am    Post subject: Reply with quote

wrc1944 wrote:
I like to think every little % of performance increase helps.

One word: tmpfs
... for /tmp (and if you are feeling lucky /var/tmp). I never managed to make it work in /var/tmp when emerging huge packages, but for everyday use it works well. For /tmp it's perfect.

It will use you memory to mount the partition (RAM and swap). It's very fast, and it manages very well the memory, so you won't run out of memory (make sure you have a lot of swap). My system here has 1GB RAM (+ 4GB swap), and I never use more than half of the RAM, even if I copy 3GB to /tmp. I limited /tmp to 3GB: using 4GB for swap, I garantee to have at least 2GB free (counting swap and RAM).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 13, 14, 15  Next
Page 8 of 15

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum