Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] How to make PCIe devices to sleep when not in use?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
GreshniK
n00b
n00b


Joined: 22 Nov 2013
Posts: 3
Location: Ukraine

PostPosted: Wed Nov 14, 2018 10:35 pm    Post subject: [SOLVED] How to make PCIe devices to sleep when not in use? Reply with quote

Hello everyone,

Hope someone could help me find a proper way around my problem.
I have a machine with two GPUs: one built-in into Intel CPU, and discrete Nvidia.

I run my Gentoo host system using Intel GPU.
Nvidia is only used for QEMU/KVM VFIO pci-passthrough, and both GPU and its Audio interface is using vfio-pci driver.

This is relatively fresh build and I have recently noticed that Nvidia GPU stays in D0 power state, i.e. fully ON and is heating up quite a bit when not in use.
It gets actually cooler when I have a guest OS running, because it takes over power-management and with proper driver handles power-saving appropriately.

I'm looking for a way to switch Nvidia preferably to D3 (cold off), but if not possible, at least to D1 or D2 to save some power and prevent unnecessary heating.
So far, what I was able to find around the Internet usually relates to PCI devices, not PCIe and is not applicable.
There are other people on the network who have same issues, e.g. this question on StackExchange, that has 0 answers to it: https://unix.stackexchange.com/questions/460928/how-can-i-put-a-pcie-device-into-the-d3-cold-power-state

Can anyone, pretty please, point me out into the right direction?

P.S. I don't believe specs are at all important here, but in case they are:

CPU: i7-8700k
Motherboard: AsRock Z370 Pro4
GPU: Nvidia GeForce 1080Ti

Thanks in advance.
Regards,
Garry


Last edited by GreshniK on Thu Nov 15, 2018 9:50 am; edited 1 time in total
Back to top
View user's profile Send private message
GreshniK
n00b
n00b


Joined: 22 Nov 2013
Posts: 3
Location: Ukraine

PostPosted: Thu Nov 15, 2018 9:47 am    Post subject: [SOLVED] How to make PCIe devices to sleep when not in use? Reply with quote

After reading more information, I discovered that vfio-pci is automatically putting PCIe device into D3 once it is bound to it.
I have confirmed that behavior after initial host boot and after booting and shutting-down a VM.

One missing case is sleep/hibernation. After resuming, card stays in D0 and heats up a lot, which is dangerous, because cooling fan does not start spinning.
So I have solved this issue by just unloading the vfio-pci module before sleep/hibernation and reloading after resume.

I'm using systemd so solved by creating

Code:

# cat /usr/lib/systemd/system-sleep/vfio.sh

#!/bin/bash
set -e

if [ "$1" == "pre" ]; then
  /sbin/modprobe -r vfio-pci
fi

if [ "$1" == "post" ]; then
  /sbin/modprobe vfio-pci
fi

Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum