Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP] Cloning Disks/Partitions/Systems over Network
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
ocbMaurice
Tux's lil' helper
Tux's lil' helper


Joined: 14 Feb 2003
Posts: 82
Location: Switzerland

PostPosted: Fri Dec 03, 2010 10:06 pm    Post subject: [TIP] Cloning Disks/Partitions/Systems over Network Reply with quote

Hi,

I'd like to share a small script with I used dozens of times to clone/backup basically anything to anywhere. Gentoo installations from old servers to vmware images, vmware to virtualbox/virtualpc/younameit, etc. It's using dd to read on the source host and ssh to transfer the data via dd to the remote host. I always had problems remembering the correct commands, so I wrote this script.

Code:
#!/bin/bash

LOCAL_IN="/dev/sdx"
REMOTE_OUT="/dev/sdx"
REMOTE_HOST="192.168.0.100"

# use dd to read local and send through ssh/dd to remote
dd if="$LOCAL_IN" | ssh -e none \
  root@"$REMOTE_HOST" "dd of=\"$REMOTE_OUT\"" & pid_dst=$!
# get the pid of the pipe writer (source reader)
pid_src=$(/bin/ps -eo pid,args | grep "dd if=$LOCAL_IN" | grep -v grep | awk '{ print $1 }')
# give a message to the user and a copy/paste one-liner to monitor transfer
echo "started transfer in background, you may monitor the transfer (pid: $pid_src)"
echo "while ps -p $pid_src > /dev/null; do kill -USR1 $pid_src; sleep 10; done"
# while ps -p $pid_src > /dev/null; do kill -USR1 $pid_src; sleep 10; done

You'll need ssh keys to authenticate on the remote host as ssh doesn't support a password command line switch.
Therefore I also have a script which requires some interaction with the user.
Press CTRL+Z to put the dd process into the background after authentication.
Then you can use the commands echoed for you to monitor the transfer.

Code:
#!/bin/bash

LOCAL_IN="/dev/sdx"
REMOTE_OUT="/dev/sdx"
REMOTE_HOST="192.168.0.100"

# give the user some instructions and copy/paste commands
echo "started dd; use ctrl+z to get it to the background. Then copy/paste these commands"
echo "bg; pid_src=\$(/bin/ps -eo pid,args | grep \"dd if=$LOCAL_IN\" | grep -v grep | awk '{ print \$1 }')"
echo "while ps -p \$pid_src > /dev/null; do kill -USR1 \$pid_src; sleep 10; done"

# use dd to read local and send through ssh/dd to remote
dd if="$LOCAL_IN" | ssh -e none \
  root@"$REMOTE_HOST" "dd of=\"$REMOTE_OUT\""

# REMOTE_HOST/REMOTE_IN - LOCAL_OUT:
# ssh -e none root@"$REMOTE_HOST" "dd if=\"$REMOTE_IN\"" \
#  | dd of="$REMOTE_OUT"

# REMOTE_HOST_IN/REMOTE_IN - REMOTE_HOST_OUT/REMOTE_OUT:
# ssh -e none root@"$REMOTE_HOST_IN" "dd if=\"$REMOTE_IN\"" \
#  | ssh -e none root@"$REMOTE_HOST_OUT" "dd of=\"$REMOTE_OUT\""

# notify user that we have finished
echo "finished transfer"

I guess it should be possible to even clone a remote host to another remote host 8)
It's would also be possible to implement gzip compression, just pipe it for both dd calls.
Code:
# untested # dd if=src | gzip -c --fast | ssh root@host "gunzip | dd of=dst"

If you think your network link will be the bottleneck you may even want to use better compression (bzip, lzma).

I hope the variables are self-explaining. You can use files and/or devices for input and output. The remote host is where the data will be written to (to the file/device named in remote out). So you can also use this script to backup your system disk/partitions to a remote file.

The safe way to copy a system is to shut it down and reboot with a boot cd or usb stick. It is possible to clone a running system and I've done that very often, although I wouldn't recommend it. If possible make sure you shutdown all services (or at least the ones that write often to the disk). Then use the sync command to flush all pending writes. After cloning you should do a fsck before the first mount.

It's also possible to clone a system even if the hard disks are of different sizes. As long as all partitions you need to clone will fit on the new disk, it should work. Just copy till dd fails on the remote host because of not enough space. Then you should be able to use fdisk/cfdisk to correct the MBR on the new host. As far as I know you can also increase the last partition to the end of the disk and dynamically expand the fs (IMO at least for ext3).

If you i.e. want to clone an existing server to a virtual machine you would do the fallowing.
  • Create the new virtual machine (attach a disk, memory, cpus, controllers)
  • Boot any linux with network and ssh support (like gentoo minimal install cd, works also via usb)
  • Start the network and sshd. Then set a password for root (passwd)
  • Run the second script on the SOURCE host, enter the password, hit ctrl+z, copy/paste the commands and wait.
  • Maybe fix the Partition Table/Master Boot Record (if hd sizes don't match).
  • Compile a Kernel for your new (virtual) Hardware.
  • Maybe adjust /etc/fstab and /boot/grub/grub.conf if device node has changed.
  • It may not boot clean the first time, use the grub bootscreen to edit and find the correct parameters.
  • If you need to re-compile your kernel (maybe forgot ext3), just boot the cd again and chroot into the system.

I tested the scripts and hope they are bugfree. But still use it at your own risk.

Hope it's usefull for some people!
Maurice
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum