Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Problem with filesystem after power outage
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
cgits
n00b
n00b


Joined: 15 Jul 2005
Posts: 70
Location: Europe

PostPosted: Mon Oct 17, 2016 12:07 pm    Post subject: Problem with filesystem after power outage Reply with quote

Hello, I have a HDD named /dev/sdc which hosts a filesystem other than my root. I boot after a power outage, and I cannot mount it. I wonder if it is possible for the filesystem to be corrupted beyond repair. I don't think it was even used at the moment of the failure. What can I try to make it mount again?

Code:
user@gandalf ~ $ sudo fdisk -l /dev/sdc
...
Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 5860532223 5860530176  2.7T Linux filesystem


From /etc/fstab:
/dev/sdc1 /mnt/data_c ext4 defaults 0 0

Code:
user@gandalf ~ $ sudo mount /dev/sdc1
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
user@gandalf ~ $ dmesg|tail
...
[154150.717274]  sdc: sdc1
[154564.927084] EXT4-fs (sdc1): VFS: Can't find ext4 filesystem
user@gandalf ~ $ sudo fsck -t ext4 /dev/sdc1
fsck from util-linux 2.26.2
e2fsck 1.42.13 (17-May-2015)
ext2fs_open2: Bad magic number in super-block
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/sdc1

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>



I tried again twice (with the -b parameters suggested by fsck above) with no result.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43207
Location: 56N 3W

PostPosted: Mon Oct 17, 2016 1:06 pm    Post subject: Reply with quote

cgits,

Try mount with alternate superblock locations.
Don't use fsck. It can make a bad situation worse. If it runs at all, it will make assumptions in the face of missing or conflicting data about your filesystem.
It will make the filesystem metadata self consistant but that says nothing about the user data that was on the filesystem.
At first, do no harm, so don't do anything that will change the filesystem. Ideally, make an image of it, so you have an undo.

Try mount with alternate superblock locations.
mount -t ext4 -o sb=131072,ro /dev/sdc1 /mnt/someplace

That's a read only mount, using the first alternate superblock.

See man mount, the bit about ext2. That's common to ext3 and ext4 too.

Code:
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000

These numbers are in filesystem blocks, which will be 4k for you.
Mount wants its sb= in terms of 1k blocks, so you need to multiply all the fs block numbers by 4.
Hence fs block 32768*4 is sb=131072 to mount.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
cgits
n00b
n00b


Joined: 15 Jul 2005
Posts: 70
Location: Europe

PostPosted: Tue Oct 18, 2016 6:25 am    Post subject: Reply with quote

wow, thank you for the very detailed analysis of the situation. Unfortunately none of the alternate superblock locations worked. I always got the same error and (dmesg)
Code:
EXT4-fs (sdc1): VFS: Can't find ext4 filesystem
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43207
Location: 56N 3W

PostPosted: Tue Oct 18, 2016 8:30 am    Post subject: Reply with quote

cgits,

Now it gets more difficult.
Its time to ask about backups, if you have space to make an image of the drive and how much time do you want to spend recovering the data.
At this stage, it looks like that in place data recovery isn't going to happen.
Knowing what the data is (photos, videos, flac etc and how it was written (write mostly, lots of additions and deletions) may be useful too.
There are tools like photorec that find files by the file headers, then make assumptions about the disk layout to recover the files.

Some background questions are also overdue.
Are you attempting the data recovery on the same machine using the same install that the drive was normally used on?
We don't want to have GPT support normally and be missing it for data recovery.

Posting the output of
Code:
dumpe2fs -h /dev/sdc1
may be useful.
Read
Code:
man dumpe2fs
and play with the -o superblock=superblock -o blocksize=blocksize and even the -f options.
This is all harmless, as its only reading things.

Healthy output (from my /boot) looks like:-

Code:
$ sudo dumpe2fs -h /dev/sde1
Password:
dumpe2fs 1.43.3 (04-Sep-2016)
Filesystem volume name:   <none>
Last mounted on:          /boot
Filesystem UUID:          c400b18c-0210-4338-a0fd-f437ecbaaf99
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              30976
Block count:              123904
Reserved block count:     6195
Free blocks:              61564
Free inodes:              30931
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         1936
Inode blocks per group:   242
RAID stride:              4
RAID stripe width:        4
Flex block group size:    16
Filesystem created:       Sun Dec 28 12:54:24 2014
Last mount time:          Fri Oct 14 16:09:46 2016
Last write time:          Fri Oct 14 16:27:02 2016
Mount count:              30
Maximum mount count:      -1
Last checked:             Sun Dec 28 12:54:24 2014
Check interval:           0 (<none>)
Lifetime writes:          113 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:             128
Default directory hash:   half_md4
Directory Hash Seed:      02028996-f637-4b92-91e3-0ae757d3347d

_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
cgits
n00b
n00b


Joined: 15 Jul 2005
Posts: 70
Location: Europe

PostPosted: Tue Oct 18, 2016 3:42 pm    Post subject: Reply with quote

Thank you again NeddySeagoon,

I have backups of most files, but not all of them. I must be missing more than 200 files. Yes, I am using the same machine to recover the data, I did not touch the physical disk drive

Code:
user@gandalf $ sudo dumpe2fs -h /dev/sdc1
dumpe2fs 1.42.13 (17-May-2015)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdc1
Couldn't find valid filesystem superblock.

Also the same with -o superblock=... and -f options (but I did not try changing blocksize)

BUT some good news. I emerged photorec and it did recover some hundreds of files. But it does not recover filenames and also the playback of the videos is not always smooth. Minor glitches like pauses, freezes, jumps, fast-forwards, that I don't think were present in the original videos. This needs a huge amount of time and effort, and I also don't have enough free space on another drive to copy the files at the moment.

BUT then I tried testdisk, which comes together with photorec. It searches for lost partitions, it finds my partition, it lists all (I think all) my files on that disk. I must have done something not entirely correctly though, because I told it to write the changes to the partition table, it told me like "done, but you will have to reboot" and after reboot I still cannot mount /dev/sdc1. (But testdisk still finds the partition and sees my files)

Anyway, maybe I should check Testdisk's documentation deeper.

You have been unbelievable help

EDIT:
I guess the partition table was fine to start with, I just got excited to see my files and stopped thinking. Anyway, I went to "[Advanced] Filesystem Utils", then "Locate ext2/ext3/ext4 backup superblock" and I got
Code:
Disk /dev/sdc - 3000 GB / 2794 GiB - CHS 364801 255 63

     Partition                  Start        End    Size in sectors

  MS Data                     2048 5860532223 5860530176 [data3]
superblock 0, blocksize=4096 [data3]
superblock 32768, blocksize=4096 [data3]
superblock 98304, blocksize=4096 [data3]
superblock 163840, blocksize=4096 [data3]
superblock 229376, blocksize=4096 [data3]
superblock 294912, blocksize=4096 [data3]
superblock 819200, blocksize=4096 [data3]
superblock 884736, blocksize=4096 [data3]
superblock 1605632, blocksize=4096 [data3]
superblock 2654208, blocksize=4096 [data3]

To repair the filesystem using alternate superblock, run
fsck.ext4 -p -b superblock -B blocksize device


EDIT2:
I found this quote somewhere on the internets
Quote:
I can't make myself believe that you were unlucky enough to have all copies of the superblock wiped. So there must be something wrong with the partition table, which in turn is throwing off the logical block offsets in the filesystem causing fsck to not be able to find the alternate superblocks
I wonder if it applies in my case.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43207
Location: 56N 3W

PostPosted: Tue Oct 18, 2016 5:10 pm    Post subject: Reply with quote

cgits,

All the right answers so far.

I agree, I think your partition table was fine to start with, its GPT so there are two copies. That both agree is checked.
You were also getting the first/only entry listed. In this case, the overwrite is harmless.
Even if it wasn't, you don't actually need a partition table to mount filesystems within partitions.

If testdisk found some more superblock locations you can try feeding them to the mount command I outlined earlier.
Don't forget the *4 for the 4k block size.

Testdisk isn't reading your filesystem to show you your files. Its taking hints but if it could read the filesystem, so could mount.
I'll guess the the jumps and other replay issues in your recovered videos are where testdisk guessed wrong and blocks of data are missing or are entirely unrelated to the file that they were recovered as a part of.

If you really want to get your hands dirty
Code:
emerge  app-forensics/sleuthkit
Description: A collection of file system and media management forensic analysis tools. How much time you want to spend on this depends on how valuable your missing data is.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
cgits
n00b
n00b


Joined: 15 Jul 2005
Posts: 70
Location: Europe

PostPosted: Mon Oct 24, 2016 12:16 pm    Post subject: Reply with quote

Thank you again,

And now for the next episode in this series: So over the last few days, I used testdisk to compare which files I already had in my backups and which needed to be recovered. I recovered several hundred files until I was satisfied I was not missing anything important. I went ahead making a new filesystem on the device /dev/sdc1, but this time I chose XFS, because I was angry at ext4 for this trouble. After all, in the past I was using all kinds of filesystems: reiserfs, JFS, XFS and never had such problems.

After making the xfs filesystem, I mount it, populate it with data, I unmount, I remount, the data is there. But next day I reboot and I can no longer mount it (!)

Code:
XFS (sdc1): Invalid superblock magic number
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43207
Location: 56N 3W

PostPosted: Mon Oct 24, 2016 4:01 pm    Post subject: Reply with quote

cgits,

Start a new topic for XFS. I've never used it.

I understand that its fast and scales well but it keeps a lot of data in RAM cache (to get the speed) and doesn't take kindly to unclean shutdowns.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum