Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
large file to usb -> kernel panic [solved]
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Sun Jan 08, 2017 1:02 pm    Post subject: large file to usb -> kernel panic [solved] Reply with quote

I tried to create a bootable usb stick from a 3,5 Gb iso image.
So at first I tried
Code:
 dd if=image.iso of=/dev/sdc

..and after a while there was a kernel panic. Flashing keyboard leds and nothing I could do except pressing the reset switch.
Then I tried another usb stick with the same result.
That was using gentoo-sources-4.9.0

After that I tried it at another computer, also with kernel 4.9.0 with the same result.
I also tried unetbootin .. got also a kernel panic.

Then I booted with kernel 4.8.13 but it did not matter, again a kernel panic.

Is there a way to solve this problem?
It is almost a shame but now I started windows at my laptop and will try if rufus can make this stick.
That worked :-)

I did make a backup recently at a external harddrive connected to usb without any problems.
So it seems that writing 1 large file goes wrong but multiple smaller files gives no trouble?
For example my /usr is 13 Gb.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees


Last edited by houtworm on Tue Jan 10, 2017 5:10 pm; edited 1 time in total
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7128
Location: almost Mile High in the USA

PostPosted: Sun Jan 08, 2017 9:02 pm    Post subject: Reply with quote

That's not good...

How about a stable version of the kernel (4.4.39)?
Are you using vanilla or gentoo-sources all around?

Do you have a dump of the panic? Is it the same error each time?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Sun Jan 08, 2017 9:33 pm    Post subject: Reply with quote

I use only gentoo-sources

I will install gentoo-sources and try it again.
With dd on the commandline I can see the kernel dump but I have to write it down because the computer will not respond anymore.

I don't know if the error is the same every time. Perhaps I will try it twice to see it :-)

But first I must reinstall grub because that other OS has ruined my bootsector.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Mon Jan 09, 2017 1:01 am    Post subject: Reply with quote

Ok, I installed kernel gentoo-sources-3.4.113-r1 and tried again.
And.. got another kernel panic:
Code:

Kernel panic - not syncing: hungtask: blocked tasks
Pid: 48, comm: khungtaskd Not tainted 3.4.113-gentoo-r1 #1
Call trace:
[<ffffffff8163706e>] ? panic+0xbf/0x1b8
[<ffffffff8108bc68>] ? watchdog+0x250/0x260
[<ffffffff8108ba10>] ? hung_task_panic+0x10/0x10
[<ffffffff81645454>] ? kernel_thread_helper+0x4/0x10
[<ffffffff8104e230>] ? kthread_flush_work_fn+0x10/0x10
[<ffffffff81645450>] ? gs_change+0xb/0xb


It looks just like the other panic this morning.
Is it usefull to try it again?

dd wrote about 20 minutes to the stick and then the writing stopped. Then 2 minutes later the kernel panic showed up.
It is an 8 Gb usb2 stick and the iso file is 3,2 Gb
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3076
Location: Illinois, USA

PostPosted: Mon Jan 09, 2017 1:36 am    Post subject: Reply with quote

Pardon me for what may be a silly question, but was /dev/sdc mounted during the dd operation? Are you sure the device is /dev/sdc? Did you try bs=8192k as the wiki said? I can see writing on a mounted filesystem causing a panic.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13831

PostPosted: Mon Jan 09, 2017 1:45 am    Post subject: Reply with quote

That panic looks to be more about the lack of progress writing to the device than due to any confusion associated with changing the filesystem while it was mounted.

OP: what manufacturer/model USB stick is this?
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Mon Jan 09, 2017 2:07 am    Post subject: Reply with quote

It is a kingston usb stick.
I tried bs=8192 -> kernel panic
then bs=8192k -> kernel panic, did not take long
no bs -> kernel panic, after 20 minutes writing

of course /dev/sdc was not mounted.
Yes I am sure that /dev/sdc was my usb stick.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7128
Location: almost Mile High in the USA

PostPosted: Mon Jan 09, 2017 3:07 am    Post subject: Reply with quote

Do you have any other sticks to test with, is it just this one stick?

My guess is that this stick gave up on a write for whatever reason but reported that it's still trying - and the kernel got confused when it couldn't make forward process writing. Especially if there was a barrier... This is bad behavior in any case though I'm not exactly sure what the best way to handle this is when there's data to be written in flight.

It could also be a USB hardware issue...
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
yilmi
n00b
n00b


Joined: 03 Jan 2017
Posts: 3

PostPosted: Mon Jan 09, 2017 5:52 pm    Post subject: Reply with quote

Panic on a hung task seems to be enabled on your system (see https://www.kernel.org/doc/Documentation/sysctl/kernel.txt)
Try : sysctl -w kernel.hung_task_panic=0

However, you device shouldn't be unresponsive for a 120s (default).... it can also be caused by a hardware error or some cheap USB sticks/SD cards that have "smart" controllers that might produce some unexpected behavior (https://www.bunniestudios.com/blog/?p=3554)
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Mon Jan 09, 2017 9:03 pm    Post subject: Reply with quote

Ok what I have tried..
downgrade core-utils to 8.23 to see if dd has perhaps changed but I still got a kernel panic.
I noticed in gkrellm at the disk activity that the writing suddenly stopt but the reading continues.
At that moment dd can't be killed and I can't reboot either so... reset switch.

I will try another usb stick and see if that works.

I will also try the hint from yilmi.

I will know the result within 10 minutes.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Mon Jan 09, 2017 9:32 pm    Post subject: Reply with quote

Ok another stick, from lexar this time.
and.. another panic and now I could see it in the logfile, thanks to yilmi:

Code:
Jan  9 23:21:14 bergwerf kernel: udevd           D    0   675      1 0x00000000
Jan  9 23:21:14 bergwerf kernel:  ffff88042df453d8 ffff88042d37ab00 ffff88042df44f40 ffff88043fd56440
Jan  9 23:21:14 bergwerf kernel:  ffff88042f352200 ffffc90002e27bb8 ffffffff8175b0a4 ffffc90002e27b98
Jan  9 23:21:14 bergwerf kernel:  0000000000000018 ffff88042df44f40 ffff88042df44f40 ffff8803cd256b5c
Jan  9 23:21:14 bergwerf kernel: Call Trace:
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b0a4>] ? __schedule+0x184/0x580
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b4cd>] ? schedule+0x2d/0x80
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b705>] ? schedule_preempt_disabled+0x5/0x10
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175c7ab>] ? __mutex_lock_slowpath+0x8b/0x100
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175c836>] ? mutex_lock+0x16/0x30
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8117f60b>] ? __blkdev_get+0x3b/0x390
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff811802e9>] ? blkdev_get+0x109/0x2f0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81180510>] ? blkdev_get_by_dev+0x40/0x40
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81149789>] ? do_dentry_open.isra.1+0x149/0x2d0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81159166>] ? path_openat+0x526/0x11d0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8115ae39>] ? do_filp_open+0x79/0xd0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8116128a>] ? dput+0x2a/0x230
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8114a943>] ? do_sys_open+0x123/0x1f0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175ebe0>] ? entry_SYSCALL_64_fastpath+0x13/0x94


These panics kept coming..

I was perhaps wrong in my previous message. The reading stopped but the writing continued.
Now the usb stick has a led that kept flashing..

The kingston stick was new.
It has no priority anymore but it would be nice if anybody can help to find where it goes wrong.

--Kees
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7128
Location: almost Mile High in the USA

PostPosted: Mon Jan 09, 2017 10:00 pm    Post subject: Reply with quote

Different coreutils shouldn't affect this issue, this is a kernel or more likely hardware problem.

It should not panic or oops or WARN due to a unresponsive disk.

What USB hardware are you using?

This is weird, blkdev_get is used during opening a device - but the bulk write has already happened... or at least part of it... Did it write anything to the flash disk?
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Mon Jan 09, 2017 10:17 pm    Post subject: Reply with quote

It happens on both computers.
The other pc has a gygabyte x790ud3 mobo
Ths usb2 port works ok. I have copied my gentoo system to it without problems, from an externel harddrive connected to the usb2 port.
There is written on the stick. Perhaps if I can mount an iso.. I can see more.

it looks like everything is on the stick.
I will try if I can boot the pc from it.
..no does not work.
but it looks like everything is written to the stick
du -c gives the same result for the iso and for the stick

weird things happen..
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13831

PostPosted: Tue Jan 10, 2017 2:25 am    Post subject: Reply with quote

houtworm wrote:
and.. another panic and now I could see it in the logfile, thanks to yilmi:
Code:
Jan  9 23:21:14 bergwerf kernel: udevd           D    0   675      1 0x00000000
Jan  9 23:21:14 bergwerf kernel:  ffff88042df453d8 ffff88042d37ab00 ffff88042df44f40 ffff88043fd56440
Jan  9 23:21:14 bergwerf kernel:  ffff88042f352200 ffffc90002e27bb8 ffffffff8175b0a4 ffffc90002e27b98
Jan  9 23:21:14 bergwerf kernel:  0000000000000018 ffff88042df44f40 ffff88042df44f40 ffff8803cd256b5c
Jan  9 23:21:14 bergwerf kernel: Call Trace:
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b0a4>] ? __schedule+0x184/0x580
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b4cd>] ? schedule+0x2d/0x80
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175b705>] ? schedule_preempt_disabled+0x5/0x10
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175c7ab>] ? __mutex_lock_slowpath+0x8b/0x100
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175c836>] ? mutex_lock+0x16/0x30
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8117f60b>] ? __blkdev_get+0x3b/0x390
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff811802e9>] ? blkdev_get+0x109/0x2f0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81180510>] ? blkdev_get_by_dev+0x40/0x40
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81149789>] ? do_dentry_open.isra.1+0x149/0x2d0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff81159166>] ? path_openat+0x526/0x11d0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8115ae39>] ? do_filp_open+0x79/0xd0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8116128a>] ? dput+0x2a/0x230
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8114a943>] ? do_sys_open+0x123/0x1f0
Jan  9 23:21:14 bergwerf kernel:  [<ffffffff8175ebe0>] ? entry_SYSCALL_64_fastpath+0x13/0x94
I think that is not a panic, but a WARN. Panic, by definition, halts the machine so abruptly that it cannot write the message to any local log file because it never runs the filesystem code after the panic starts. (You can still save panic output by writing it to a serial console and using a separate machine to record the text shown on the serial console.)
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Tue Jan 10, 2017 2:41 am    Post subject: Reply with quote

I think it is a kernel panic because my machine blocked every time with flashing keyboard leds. Until I did what yilmi wrote:

Code:
 sysctl -w kernel.hung_task_panic=0


After that I could see the kernel message in the logfile.
It is however a different message from the message I could see with a kernel panic before.


It is strange that dd did not stop with an error message.
'i/o error' or whatever message.

Tomorrow I will try to write a smaller iso to the stick.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Tue Jan 10, 2017 9:08 am    Post subject: Reply with quote

Ok, I wrote another iso to the stick and that worked. No bs=xx used with dd
It was 368 Mb instead of 3,5 gb

So it goes wrong if the filesize is too large.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Tue Jan 10, 2017 5:09 pm    Post subject: Reply with quote

ok problem solved!

When using direct i/o it works:

Code:
dd if=image.iso of=/dev/sdc bs=4M iflag=direct oflag=direct


..and I can't boot from the stick hahaha
But the problem is solved.
Thanks for thinking with me!

--Kees
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3076
Location: Illinois, USA

PostPosted: Tue Jan 10, 2017 5:28 pm    Post subject: Reply with quote

Never heard of the direct i/o flag before. It would seem that writing to a device IS direct.
But I found this: http://www.alexonlinux.com/what-is-direct-io-anyway

Maybe you don't have enough swap to cache 3.5G?

Thanks for the thread. One learns more every day.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Tue Jan 10, 2017 5:47 pm    Post subject: Reply with quote

Tony0945,

swap is only used for dynamically allocated RAM. RAM content that does not have a permanent home on disk.
Thus dirty buffers - waiting to be written, will never be written to swap. They will just wait their turn to be written.

Meanwhile, as RAM fills up with dirty buffers, everything grinds to a halt.
Dynamically allocated RAM is pushed to swap, read only data/code is flushed, as it can be reread later and RAM is filled up by the read task of the copy operation.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3076
Location: Illinois, USA

PostPosted: Tue Jan 10, 2017 6:20 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Tony0945,

swap is only used for dynamically allocated RAM. RAM content that does not have a permanent home on disk.
Thus dirty buffers - waiting to be written, will never be written to swap. They will just wait their turn to be written.

Meanwhile, as RAM fills up with dirty buffers, everything grinds to a halt.
Dynamically allocated RAM is pushed to swap, read only data/code is flushed, as it can be reread later and RAM is filled up by the read task of the copy operation.


I see. I had thought the kernel swapped those write buffers. Then those usb writes failed because of insufficient RAM for those buffers? If the OP had another 4G everything would not grind to a halt?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Tue Jan 10, 2017 6:29 pm    Post subject: Reply with quote

Tony0945,

He would still have got the panic due to the hung task but up to that point, everything would look normal.
It appears that 120 sec was not enough to write enough to keep the kernel from detecting a hung task.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3076
Location: Illinois, USA

PostPosted: Tue Jan 10, 2017 7:07 pm    Post subject: Reply with quote

Why did --directio solve that problem? What mechanism was bypassed?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Tue Jan 10, 2017 7:30 pm    Post subject: Reply with quote

Tony0945,

The kernels cache was bypassed.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7128
Location: almost Mile High in the USA

PostPosted: Tue Jan 10, 2017 8:30 pm    Post subject: Reply with quote

EIther way, there is still a problem with the kernel or hardware. This is merely a workaround.

Being able to panic or hang the kernel in userland should not be possible (other than by deliberate corruption, but that's a different issue.)
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
houtworm
Guru
Guru


Joined: 08 Mar 2003
Posts: 352
Location: Den Haag, Netherlands

PostPosted: Tue Jan 10, 2017 11:42 pm    Post subject: Reply with quote

Anybody can try it.
I just tried a copy:
Code:
mount /dev/sdc /mnt/stick
cp image.iso /mnt/stick


And I returned at the prompt.
But when unmounting the stick and de cache was written, it took a while and... again a kernel panic.
So anybody can test this without destroying a stick with dd.
The iso is about 3,2 Mb so any large file will do.

If you also get a kernel panic, it is most likely not a hardware failure.
If you get no errors the I am interested in your gentoo-kernel config.
Perhaps I can figure out the difference.
_________________
niemand is onbekwamer, dan een timmerman zonder hamer

Kees
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum