Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO fix&back "input/output error", on a mounted filesystem
View unanswered posts
View posts from last 24 hours

Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message

Joined: 13 Nov 2005
Posts: 388
Location: Socialist land of North America

PostPosted: Mon Aug 08, 2011 10:31 pm    Post subject: HOWTO fix&back "input/output error", on a moun Reply with quote

First, if it happen to you, your disk is dieing, change it. But if you need to fix the problem to do a backup, or simply want to fix the problem for now and tag the bad block so the disk is "safe" for a few more months, it worth trying.

I just had the problem, /usr/portage was broken and I could not use portage. It's quite bad, I will replace the disk soon, but for now, I just wanted it to work again, I needed portage. So, what could I do?

└─: rm /usr/portage/x11-themes/gtk-engines-nodoka -rf                                                      ──(Mon 17:49)─┘
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/Manifest': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild': Input/output error
rm: cannot remove `/usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog': Input/output error

Nope, too easy

fsck? no, it never found the problem.

After a little googling, I heard about debugfs for some other unrelated purpose, but I tried it. Bingo! This tool come with Linux and offer a wide range of tools to fix file system. See it as a manual fsck.


Available debugfs requests:

show_debugfs_params, params
                         Show debugfs parameters
open_filesys, open       Open a filesystem
close_filesys, close     Close the filesystem
feature, features        Set/print superblock features
dirty_filesys, dirty     Mark the filesystem as dirty
init_filesys             Initialize a filesystem (DESTROYS DATA)
show_super_stats, stats  Show superblock statistics
ncheck                   Do inode->name translation
icheck                   Do block->inode translation
change_root_directory, chroot
                         Change root directory
change_working_directory, cd
                         Change working directory
list_directory, ls       List directory
show_inode_info, stat    Show inode information
dump_extents, extents, ex
                         Dump extents information
link, ln                 Create directory link
unlink                   Delete a directory link
mkdir                    Create a directory
rmdir                    Remove a directory
rm                       Remove a file (unlink and kill_file, if appropriate)
kill_file                Deallocate an inode and its blocks
clri                     Clear an inode's contents
freei                    Clear an inode's in-use flag
seti                     Set an inode's in-use flag
testi                    Test an inode's in-use flag
freeb                    Clear a block's in-use flag
setb                     Set a block's in-use flag
testb                    Test a block's in-use flag
modify_inode, mi         Modify an inode by structure
find_free_block, ffb     Find free block(s)
find_free_inode, ffi     Find free inode(s)
print_working_directory, pwd
                         Print current working directory
expand_dir, expand       Expand directory
mknod                    Create a special file
list_deleted_inodes, lsdel
                         List deleted inodes
undelete, undel          Undelete file
write                    Copy a file from your native filesystem
dump_inode, dump         Dump an inode out to a file
cat                      Dump an inode out to stdout
lcd                      Change the current directory on your native filesystem
rdump                    Recursively dump a directory to the native filesystem
set_super_value, ssv     Set superblock value
set_inode_field, sif     Set inode field
set_block_group, set_bg  Set block group descriptor field
logdump                  Dump the contents of the journal
htree_dump, htree        Dump a hash-indexed directory
dx_hash, hash            Calculate the directory hash of a filename
dirsearch                Search a directory for a particular filename
bmap                     Calculate the logical->physical block mapping for an inode
imap                     Calculate the location of an inode
dump_unused              Dump unused blocks
set_current_time         Set current time to use when setting filesystme fields
supported_features       Print features supported by this version of e2fsprogs
help                     Display info on command or topic.
list_requests, lr, ?     List available commands.
quit, q                  Leave the subsystem.

So, now I had everything I needed not only to fix the file system, but the information about the physical sector to ban too (bmap).

debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/ChangeLog
debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/Manifest
debugfs:  rm /usr/portage/x11-themes/gtk-engines-nodoka/gtk-engines-nodoka-0.7.5.ebuild


rmdir /usr/portage/x11-themes/gtk-engines-nodoka/

and then

rm -rf /usr/portage/x11-themes/gtk-engines-nodoka/ #In bash (not debugfs)

everything went back to a working state. A few S.M.A.R.T command later, the disk is now safe to use again. It's quite cool to know when you need to fix thing now. A nice thing to know for every plumber/sysadmin out there.

-THIS IS A TEMPORARY SOLUTION- but it still cool to have

I hope it can save your day or help you laugh at in the face of the Murphy law in a critical moment of your career 8)

Update: As "planned" the disk became unusable a few days later. To backup your failed disk offline, use
cp -a /mnt/old/* /mnt/new/

This will properly copy everything and keep the symlink. Dont use DD as you will probably produce a corrupted file system.

Or, if you really want no downtime, online
ls / | grep -vE "(dev|proc|sys)" | xargs -i cp -a /{} /mnt/new/ ; mkdir /mnt/new/{dev,proc,sys}

To see what you lost, run this command:
for FILE in `find /mnt/old/ | sed "s/\/mnt\/old//"`; do find /mnt/new/  | sed "s/\/mnt\/new//" | grep $FILE > /dev/null || echo $FILE | tee /mnt/new/lost_file_list.txt ;done

As root, it will take a while (hours)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum