Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Convert your filesystem [hopefully] without data loss..
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
gfunkmonk
Tux's lil' helper
Tux's lil' helper


Joined: 10 Jul 2002
Posts: 129
Location: Freelandville, IN

PostPosted: Sun Sep 01, 2002 9:07 pm    Post subject: Convert your filesystem [hopefully] without data loss.. Reply with quote

This will work for these filesystems:
minix xfs jfs reiserfs ext2 ext3

Warning: This has caused at least 2 corrupt filesystmes (read posts below). Always backup your data. --pjp

OK, browse over to http://www.tux.org/pub/people/kent-robotti/looplinux/rip/
and download the bootable cd ISO, and burn it to a cd.

Reboot with the cd in the drive, and type:

Code:

convertfs /dev/hdxx current_fs new_fs

After the conversion is done its probally a good idea to run fsck

Code:

The 'reiserfsck' program is used to check and repair
a linux reiserfs filesystem.

The 'xfs_repair' program is used to repair a linux
xfs filesystem.

The 'fsck.jfs' program is used to check and repair
a linux jfs filesystem.

The 'e2fsck' program is used to check and repair
a linux ext2 or ext3 filesystem.


So something like:

Code:

fsck.jfs /dev/hdxx


Next, mount the filesystem,
Code:

mount /dev/hdxx /mnt/linux


and chroot into the filesystem
Code:

chroot /dev/hdxx /mnt/linux


and then proceede to edit /etc/fstab to use the correct filesystem, reboot and your done..

EDIT: Added warning, and modified title to include "hopefully." --pjp
Back to top
View user's profile Send private message
MoonWalker
Guru
Guru


Joined: 04 Jul 2002
Posts: 449

PostPosted: Tue Oct 01, 2002 2:41 am    Post subject: Reply with quote

Just a warning... This totally corrupted my 1.4 system when I tried to change fs from xfs to reiserfs, so a total reinstall remained the only choice.

HW facts was ABIT BG7 Mobo, P4 1.8, 512MB RAM, IBM 120GB HD devided in 3 primary and 1 extended and 9 partitions all together.

So it didn't work for me, but it may do for you... your choise.
_________________
/Joakim

Living on earth is expensive, but it includes a free trip around the sun
every year.
Back to top
View user's profile Send private message
nempo
Guru
Guru


Joined: 16 Apr 2002
Posts: 360
Location: Linkoping, Sweden

PostPosted: Tue Oct 01, 2002 2:36 pm    Post subject: Reply with quote

Dosn't seem very reliable since it corrupted my ext3 root partition.
Back to top
View user's profile Send private message
delta407
Bodhisattva
Bodhisattva


Joined: 23 Apr 2002
Posts: 2876
Location: Chicago, IL

PostPosted: Tue Oct 01, 2002 5:11 pm    Post subject: Reply with quote

I think the moral of the story here is to have a backup before you attempt to do an in-place conversion between two very dissimilar filesystems.
_________________
I don't believe in witty sigs.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 18165

PostPosted: Tue Oct 01, 2002 5:25 pm    Post subject: Reply with quote

Edited gfunkmonk's post to include a warning. Sent a PM and mentioned he should feel free to change the warning, but that I thought one should be there.
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
aardvark
Guru
Guru


Joined: 30 Jun 2002
Posts: 576

PostPosted: Thu Oct 03, 2002 6:01 pm    Post subject: Reply with quote

delta407 wrote:
I think the moral of the story here is to have a backup before you attempt to do an in-place conversion between two very dissimilar filesystems.


And when you're making a backup anyway, you might as well reformat the file system after backing up and placing it back on the new/different filesytem. (of course edit the fstab accordingly) You wont need to dl. that looplinux ISO.
This could be how:

1. boot from install CD. an press enter till you have a prompt.
2. mount the fs you want to backup: (choose '??' and "gentoobakfs" to your own needs)
Code:
 mkdir /mnt/gentoobakfs
mount /dev/hd??  /mnt/gentoobakfs

3. mount the place where the backup is to be written. (a partition with enough free space and a boot-cd-kernel-supported fs in the same manner as point 2. above)
4. cd to the mountpoint of point (2. )
5a. Backing up:
Code:
 tar -cvzpf /mnt/place_to_put/your_backup.tar.gz *

(This will not pack any dot-files (hidden) in the 'root', but you normally don't have those.)

5b. Wait for it to complete (gzip is MUCH faster that bzip2 and only little larger)
6. unmount the partition you backed up:
Code:
 umount /mnt/gentoobakfs

7. format the /dev/hd?? of point (2. ):
Code:
mk(your_fs_of_choice) /dev/hd??

8. repeat point 2 and 4
9. Putting the backup back in place
Code:
 tar -xvzpf /mnt/place_to_put/your_backup.tar.gz

10. If necessary,
Code:
 nano -w /mnt/gentoobakfs/etc/fstab

11. Unmount everything and you should be set after rebooting without CD.
Back to top
View user's profile Send private message
guero61
l33t
l33t


Joined: 14 Oct 2002
Posts: 811
Location: Behind you

PostPosted: Sat Nov 02, 2002 1:35 am    Post subject: Reply with quote

Just thought I'd pass along that I just did the tarball conversion and it worked great, but some pointers I came up with before rebooting to the CD:

1. Make sure you have a kernel in place that will handle both filesystems
2. Don't forget to GRUB your /boot again; otherwise you'll get error 17.

Other than that, it was a flawless, lossless strategy; way to go!
Back to top
View user's profile Send private message
ejwahl
Tux's lil' helper
Tux's lil' helper


Joined: 27 May 2002
Posts: 107
Location: Rhode Island

PostPosted: Sat Nov 02, 2002 6:13 pm    Post subject: Reply with quote

Here's code that worked for me when I backed up root:

Code:
find / -path /mnt -prune -o -path /dev -prune -o -path /proc -prune -o -print | cpio -pvdm /home/user/backup


This copies everything in / except for /mnt, /dev and /proc to /home/user/backup.

HTH
_________________
Erich J. Wahl
RLU 199500
ICQ 48100000
Back to top
View user's profile Send private message
co-D
n00b
n00b


Joined: 19 Mar 2003
Posts: 16

PostPosted: Mon Oct 27, 2003 2:05 am    Post subject: Reply with quote

This is what i made to make my backups easier, i'm a novice at scripting so revise it if you wish to use it, but i've backed up my system from it before when i've felt like reformatting root as a different filesystem.Just boot from the gentoo livecd (or any other bootable disk for that matter) and extract the tar.gz's to the new root and boot partitions.

You might have to reinstall your boot loader, but for some reason last time i did this it just worked.

Code:

#!/bin/bash
archive='/mnt/xfs/backup'
directories='bin boot dev etc home lib mnt opt root sbin sys tmp usr var'
Date=`date +%F`
command='tar -cvzpf'
exclude='/usr/portage/distfiles/* /var/tmp/portage/* /mnt/*/*'
excludefile=$archive/$Date/excluded
if [ ! -d $archive/$Date ]
  then
     mkdir $archive/$Date
  fi

if [ -e $archive/$Date/excluded ]
  then
     rm -f $archive/$Date/excluded
  fi
 
for f in ${exclude}
  do
   echo ${f} >> $archive/$Date/excluded
  done
 
echo -en '\E[40;37m'"\033[1m"Excluding Files:"

`cat $excludefile`\033[0m"   # White
   tput sgr0
echo ""
for d in ${directories}
do
  if [ -e $archive/$Date/${d}.tar.gz ]
   then
   
   echo ""
   echo -en '\E[40;36m'"\033[1m"Archive exists, skipping "$archive/$Date/${d}.tar.gz"..."\033[0m"   # Cyan
   tput sgr0
   echo ""
else
   echo ""
   echo -en '\E[40;32m'"\033[1m"Creating "$archive/$Date/${d}.tar.gz"..."\033[0m"   # Green
   tput sgr0
   echo ""

if [ $d == boot ]
   then
   mount /boot
   $command  $archive/$Date/${d}.tar.gz /${d} --exclude-from=$excludefile
   umount /boot
   else
   $command  $archive/$Date/${d}.tar.gz /${d} --exclude-from=$excludefile
   fi
fi

done

echo -en '\E[40;31m'"\033[1m
"%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%  NOTE  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
"\ \ \ \ \ \ \ \ create /proc and /sys directories in new system \ \ "
"%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%  NOTE  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"\033[0m"   # Red
echo ""
   tput sgr0
Back to top
View user's profile Send private message
Trellph
n00b
n00b


Joined: 26 Mar 2003
Posts: 19
Location: Houston

PostPosted: Wed Apr 21, 2004 3:37 am    Post subject: Reply with quote

I had me some fun with this. I thought like most applications that did this it would load itself into memory. I found out I was wrong when /bin was erased and it errored out
not to worry, /bin is easy to replace.
so I boot to a cd and do it, thinking that it loaded itself into memory but not mv rm cp and whatnot.

It made the image just fine, but then it stopped because it didn't load it's 2 other counterparts into memory. oops.

To resolv this I mounted the image to another dir, mv'd all the files back, and use a tmpfs in ram to copy over the convertfs files to from /sbin and it worked finally ;p

sure takes a long time for it to mv over to itself 12 gigs of data tho.
Back to top
View user's profile Send private message
Deathwing00
Bodhisattva
Bodhisattva


Joined: 13 Jun 2003
Posts: 4087
Location: Dresden, Germany

PostPosted: Mon Jun 14, 2004 10:51 pm    Post subject: Reply with quote

The command covertfs does not exist any more in the RIP CD... so this Doc is OBSOLETE/NEEDS RENEWAL.

OK, I have tweaked around and finally got how to do it:

--- Root Partition to be converted ---

1. If the partition you want to convert is the root one, then boot into your system.
2. # emerge convertfs
3. reboot the system with the gentoo LiveCD
4. mount your root partition (the one to be converted)
5. copy the following files from /mnt/gentoo/sbin/ to /sbin/:
- convertfs
- convertfs_dumb
- devclone
- devremap
- prepindex
6. unmount the partition
7. (this one is important!) # cd /sbin/
8. # convertfs /dev/hdxY current_fs target_fs
9. wait until it ends :)

--- Non-root partition to be converted ---

1. If the partition you want to conver is not the root one, then it's easier; boot into your system.
2. # emerge convertfs
3. make sure that the partition is not mounted
4. # convertfs /dev/hdxY current_fs target_fs
5. wait until it ends :)


Have fun!
Back to top
View user's profile Send private message
thrasher6670
Apprentice
Apprentice


Joined: 04 Aug 2003
Posts: 269
Location: London, Ontario

PostPosted: Wed Jun 23, 2004 5:11 pm    Post subject: Reply with quote

I have copied this article to

http://gentoo-wiki.com/HOWTO_Convert_Filesystems

I included Deathwing00's changes ... sorta :)

If anyone things anything needs changing, go for it
_________________
Gentoo-Portage.com - Where the portage meets the web
Gentoo Wiki
Back to top
View user's profile Send private message
Deathwing00
Bodhisattva
Bodhisattva


Joined: 13 Jun 2003
Posts: 4087
Location: Dresden, Germany

PostPosted: Wed Jun 23, 2004 5:20 pm    Post subject: Reply with quote

thrasher6670 wrote:
I have copied this article to

http://gentoo-wiki.com/HOWTO_Convert_Filesystems

I included Deathwing00's changes ... sorta :)

If anyone things anything needs changing, go for it


Nice one indeed :)
Back to top
View user's profile Send private message
ZeroDivide
Tux's lil' helper
Tux's lil' helper


Joined: 15 Mar 2003
Posts: 86

PostPosted: Fri Jul 09, 2004 11:02 pm    Post subject: Reply with quote

I'm not sure if it's just my setup, but convertfs is EXTREMELY slow.
It's taking about 3 1/2 hours to convert 1 gig of data from reiserfs to xfs.
I have DMA enabled and 1/3 of the partition is free space so I dont think its a problem with my drive.
Anyway, just thought I would post this to let people know how slow it can be before they embark on their long conversion journey :)
Back to top
View user's profile Send private message
Deathwing00
Bodhisattva
Bodhisattva


Joined: 13 Jun 2003
Posts: 4087
Location: Dresden, Germany

PostPosted: Sat Jul 10, 2004 7:54 am    Post subject: Reply with quote

This is very strange... it didn't take so much time for me.
Back to top
View user's profile Send private message
ZeroDivide
Tux's lil' helper
Tux's lil' helper


Joined: 15 Mar 2003
Posts: 86

PostPosted: Sat Jul 10, 2004 3:31 pm    Post subject: Reply with quote

It looks like i jumped the gun a little bit, converting 15GB only took 6 hours. :oops:
Back to top
View user's profile Send private message
asph
l33t
l33t


Joined: 25 Aug 2003
Posts: 738
Location: Barcelona, Spain

PostPosted: Mon Aug 23, 2004 8:09 am    Post subject: Reply with quote

i think that "How to convert your filesystem and lose all your data" would be more accurate.. i run it to convert from reiserfs to xfs, and after 2 secons it output "mv: comand not found" and stopped working.. after rebooting i had lost config files, binaries and a bunch of files so it won't boot!

of course i did a backup before so i have my system back again, but i don't understand how this tool is in portage, it totally sucks. You can't release a tool this dangerous if you are not 100% sure it works! I know they say that "warning, this can delete your data blah blah", but then just don't put it on portage.
_________________
gentoo sex is updatedb; locate; talk; date; cd; strip; look; touch; finger; unzip; uptime; gawk; head; emerge --oneshot condom; mount; fsck; gasp; more; yes; yes; yes; more; umount; emerge -C condom; make clean; sleep
Back to top
View user's profile Send private message
Moloch
Apprentice
Apprentice


Joined: 17 Mar 2003
Posts: 293
Location: Albuquerque, NM, US

PostPosted: Wed Aug 25, 2004 6:46 pm    Post subject: Reply with quote

Sounds like you ran it from your current live system. Idealy you want to run it from a boot CD so you aren't attempting to convert files/partitions that are in use.
Back to top
View user's profile Send private message
Conan
Guru
Guru


Joined: 02 Nov 2004
Posts: 360

PostPosted: Tue Nov 02, 2004 1:28 am    Post subject: Reply with quote

I just tried doing this... and when i try to copy convertfs from /mnt/gentoo/sbin/ to /sbin, it tells me that /sbin/ is a read only file system, is there some way to fix this?
Back to top
View user's profile Send private message
bigredgiant1
n00b
n00b


Joined: 23 Dec 2003
Posts: 62

PostPosted: Tue Jan 04, 2005 10:55 pm    Post subject: Reply with quote

I think a better way to do this if you had the space would be to copy everything over to a second drive, make the new filesystems on your partitions, and copy everything back over, like so:

Code:

<boot a live cd and mount all your partitions/drives>
# chroot /mnt/gentoo
# mount /dev/hdb1 /mnt/tmp_drive
# mkdir /mnt/tmp_drive/gentoo
# cd /
# find . -print | cpio -pamd /mnt/tmp_drive/gentoo
# umount /mnt/tmp_drive/gentoo
# exit
<this will probably take at least 10 or 15 minutes>
# umount /dev/hda3 /dev/hda5 ...
<I use reiserfs here but use whatever filesystem you desire>
# mkreiserfs /dev/hda3
# mkreiserfs /dev/hda5
<continue for all of your partitions...>
# mount /dev/hdb1 /mnt/tmp_drive
# mount /dev/hda3 /mnt/gentoo
# mount /dev/hda5 /mnt/gentoo/var
<continue as needed>
# cd /mnt/tmp_drive/gentoo
# find . -print | cpio -pamd /mnt/gentoo
<Reboot and you're done>

_________________
Vic @ Shellsage
Dual Opteron 242 -- AMD64 2.6.16 Development Sources
CFLAGS="-march=k8 -O3 -pipe" MAKEOPTS="-j3"
Back to top
View user's profile Send private message
rvkasper
n00b
n00b


Joined: 17 Mar 2005
Posts: 1
Location: Denver, CO USA

PostPosted: Thu Mar 17, 2005 4:47 pm    Post subject: I'm an idiot Reply with quote

Hi all,
I decided to convert a 200 gig partition from reiserfs to ext2 so that it will work when I move my Fedora Core 2 OS over to FreeBSD - to facilitate this I compiled convertfs from source and am running it from my current OS installation. 4 days later it is still going, running devremap - it is currently here:
Relocating block group at 2899145...

Any thoughts on how far it has left to go - is there some output from a command (like HDParm) that would help me figure that out?

Also, would booting from this gentoo livecd speed things up at all? This is not my root partition I am converting - simplay a data partition - my root partition is ext3.

- Ryan
_________________
Taco taco, burrrito burrito!
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2971
Location: Germany

PostPosted: Thu Sep 29, 2005 12:58 pm    Post subject: Reply with quote

Two issues with convertfs:

1. You need a lot of free space on the partition you want to convert.

convertfs uses 'mv' to move all directories (including subdirs and files) from the root tree to the image. When moving directories, 'mv' will delete the source files only after the whole thing was copied, so you need at least as much free space as the space consumed by the biggest directory (including subdirs and files) on your partition. You can find out that size using 'du'.

2. Relocation of blocks is damn slow.

After the image was completely created, you have your old filesystem with a single file in it (image of the new filesystem), and in that image there are all your files. Now, for the image to replace your original filesystem, convertfs has to move the blocks of the image file onto the appropriate positions on your hard disk.

Think of your partition now as a really fragmented system and you're trying to use Windows defrag to solve the system - every block of every single file has to be moved to a different location (and that location might already be occupied by other blocks).

So in short, there is a lot of shuffling around involved, and in the current convertfs, this is not solved very efficiently, so depending on how much block relocations have to be done, this will take AGES. And I really mean AGES - I'm trying to convertfs a 80GB partition from ext3 to xfs right now and after one day, the progress is at 1GB out of 80GB. So with any luck I only have to wait 79 more days for it to get finished.

The problem lies within devremap.c, in the function find_cross_block, which wastes a lot of time trying to find the correct physp location. The CPU has to do an awful lot of work, disk activity is extremely low (looks like one block write every 20 seconds).

I don't fully understand what devremap.c is trying to do here (there does not seem to be any documentation, grrr), but this can be done faster in any case.
Back to top
View user's profile Send private message
frostschutz
Advocate
Advocate


Joined: 22 Feb 2005
Posts: 2971
Location: Germany

PostPosted: Thu Sep 29, 2005 7:33 pm    Post subject: Reply with quote

filed a bug report for this: sys-fs/convertfs eats 100% CPU for days on end

comments are welcome...

EDIT:

attached a patch to the bug report, which fixed the 100% CPU / very slow progress problem for me. conversion of 80GB partiton ext3 -> xfs succeeded :-). Still, this is highly dangerous software and it has other bugs as well. But if you're already stuck in the 'relocating blocks' part of convertfs, the patch might help you. It's reasonably safe to ctrl+c out of the relocating (while there is nothing being written on the hard disk) and resume with the patched program.

Oh, and make sure you compile it with -march=your_cpu_type and -O2. It'll be really slow otherwise. And don't use -O3, for some reason that slows down the process as well.
Back to top
View user's profile Send private message
dj015
n00b
n00b


Joined: 05 Jul 2004
Posts: 11

PostPosted: Tue Jan 17, 2006 6:44 am    Post subject: Reply with quote

For those people that, like me, found this forum too late:

1. In the "Relocating block group at X" messages, X is the filesystem block number, not a kilobyte number. So if you have a filesystem with 4kB blocks, multiply X by 4 to get the number of kilobytes.

2. The reason it is slow is the poor block relocation algorithm (you can't code a triply nested loop, and then use "gcc -O3" to make it fast: compilers never reduce the algorithm's order of complexity...).

3. convertfs is very robust - it survived about 5 Ctrl+C's and a power failure.

4. The "very ugly patch" works like a charm. It stripped down the conversion time for my 120 GB filesystem from 11 days to 6 hours. I did e2fsck afterwards and it found no errors, my files were all there, and I opened a couple of them and they were fine. Thank you Andreas!

5. I am never using it again.
Back to top
View user's profile Send private message
dajja
n00b
n00b


Joined: 11 May 2006
Posts: 15
Location: Sweden

PostPosted: Sat May 20, 2006 12:32 pm    Post subject: Reply with quote

I have recently successfully converted an ext3 partition to reiserfs (no corruption reported by fsck) :D

But I am (still) somewhat suspicious if it will work in the long run (I will probably notice if it doesn't ;))---is fsck completely reliable in reporting a possible corruption/error in a filesystem? Are there other tools to consider?
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum