FPGA DFL Changes for 5.4

This pull-request contains the FPGA DFL changes for 5.4
 
 - The first three patches are cleanup patches making use of dev_groups and
   making the init callback optional.
 - One patch adds userclock sysfs entries that are DFL specific
 - One patch exposes AFU port disable/enable functions
 - One patch adds error reporting
 - One patch adds AFU SignalTap support
 - One patch adds FME global error reporting
 - The final patch is a documentation patch that decribes the
   virtualization interfaces
 
 This patchset requires the 'dev_groups_all_drivers' tag from drivers
 core for the dev_groups refactoring as well as the DFL changes already
 in char-misc-next.
 
 Signed-off-by: Moritz Fischer <mdf@kernel.org>
 -----BEGIN PGP SIGNATURE-----
 
 iHUEABYIAB0WIQS/ea/a56fFi9QbQeJ+E8eWOj6VqQUCXW8mjQAKCRB+E8eWOj6V
 qVbHAQDnhqNNyMPY93VseWIyJ/9OpkSzunc820XfmVehPZkw+gD+KLByUWy6vZN0
 lh/LgdvIKYeUSm1ZZXj+DszemmaPiwk=
 =xtf/
 -----END PGP SIGNATURE-----

Merge tag 'fpga-dfl-for-5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/mdf/linux-fpga into char-misc-next

Moritz writes:

FPGA DFL Changes for 5.4

This pull-request contains the FPGA DFL changes for 5.4

- The first three patches are cleanup patches making use of dev_groups and
  making the init callback optional.
- One patch adds userclock sysfs entries that are DFL specific
- One patch exposes AFU port disable/enable functions
- One patch adds error reporting
- One patch adds AFU SignalTap support
- One patch adds FME global error reporting
- The final patch is a documentation patch that decribes the
  virtualization interfaces

This patchset requires the 'dev_groups_all_drivers' tag from drivers
core for the dev_groups refactoring as well as the DFL changes already
in char-misc-next.

Signed-off-by: Moritz Fischer <mdf@kernel.org>

* tag 'fpga-dfl-for-5.4' of git://git.kernel.org/pub/scm/linux/kernel/git/mdf/linux-fpga:
  Documentation: fpga: dfl: add descriptions for virtualization and new interfaces.
  fpga: dfl: fme: add global error reporting support
  fpga: dfl: afu: add STP (SignalTap) support
  fpga: dfl: afu: add error reporting support.
  fpga: dfl: afu: expose __afu_port_enable/disable function.
  fpga: dfl: afu: add userclock sysfs interfaces.
  fpga: dfl: afu: convert platform_driver to use dev_groups
  fpga: dfl: fme: convert platform_driver to use dev_groups
  fpga: dfl: make init callback optional
  driver core: add dev_groups to all drivers
This commit is contained in:
Greg Kroah-Hartman 2019-09-04 07:43:42 +02:00
commit b8bf2681dc
14 changed files with 1062 additions and 78 deletions

View File

@ -44,3 +44,65 @@ Description: Read-only. It returns socket_id to indicate which socket
this FPGA belongs to, only valid for integrated solution.
User only needs this information, in case standard numa node
can't provide correct information.
What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie0_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-Write. Read this file for errors detected on pcie0 link.
Write this file to clear errors logged in pcie0_errors. Write
fails with -EINVAL if input parsing fails or input error code
doesn't match.
What: /sys/bus/platform/devices/dfl-fme.0/errors/pcie1_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-Write. Read this file for errors detected on pcie1 link.
Write this file to clear errors logged in pcie1_errors. Write
fails with -EINVAL if input parsing fails or input error code
doesn't match.
What: /sys/bus/platform/devices/dfl-fme.0/errors/nonfatal_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. It returns non-fatal errors detected.
What: /sys/bus/platform/devices/dfl-fme.0/errors/catfatal_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. It returns catastrophic and fatal errors detected.
What: /sys/bus/platform/devices/dfl-fme.0/errors/inject_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-Write. Read this file to check errors injected. Write this
file to inject errors for testing purpose. Write fails with
-EINVAL if input parsing fails or input inject error code isn't
supported.
What: /sys/bus/platform/devices/dfl-fme.0/errors/fme_errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-Write. Read this file to get errors detected on FME.
Write this file to clear errors logged in fme_errors. Write
fials with -EINVAL if input parsing fails or input error code
doesn't match.
What: /sys/bus/platform/devices/dfl-fme.0/errors/first_error
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the first error detected by
hardware.
What: /sys/bus/platform/devices/dfl-fme.0/errors/next_error
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the second error detected by
hardware.

View File

@ -46,3 +46,56 @@ Contact: Wu Hao <hao.wu@intel.com>
Description: Read-write. Read or set AFU latency tolerance reporting value.
Set ltr to 1 if the AFU can tolerate latency >= 40us or set it
to 0 if it is latency sensitive.
What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcmd
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Write-only. User writes command to this interface to set
userclock to AFU.
What: /sys/bus/platform/devices/dfl-port.0/userclk_freqsts
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the status of issued command
to userclck_freqcmd.
What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrcmd
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Write-only. User writes command to this interface to set
userclock counter.
What: /sys/bus/platform/devices/dfl-port.0/userclk_freqcntrsts
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the status of issued command
to userclck_freqcntrcmd.
What: /sys/bus/platform/devices/dfl-port.0/errors/errors
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-Write. Read this file to get errors detected on port and
Accelerated Function Unit (AFU). Write error code to this file
to clear errors. Write fails with -EINVAL if input parsing
fails or input error code doesn't match. Write fails with
-EBUSY or -ETIMEDOUT if error can't be cleared as hardware
in low power state (-EBUSY) or not respoding (-ETIMEDOUT).
What: /sys/bus/platform/devices/dfl-port.0/errors/first_error
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the first error detected by
hardware.
What: /sys/bus/platform/devices/dfl-port.0/errors/first_malformed_req
Date: August 2019
KernelVersion: 5.4
Contact: Wu Hao <hao.wu@intel.com>
Description: Read-only. Read this file to get the first malformed request
captured by hardware.

View File

@ -87,6 +87,8 @@ The following functions are exposed through ioctls:
- Get driver API version (DFL_FPGA_GET_API_VERSION)
- Check for extensions (DFL_FPGA_CHECK_EXTENSION)
- Program bitstream (DFL_FPGA_FME_PORT_PR)
- Assign port to PF (DFL_FPGA_FME_PORT_ASSIGN)
- Release port from PF (DFL_FPGA_FME_PORT_RELEASE)
More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):
@ -102,6 +104,10 @@ More functions are exposed through sysfs
one FPGA device may have more than one port, this sysfs interface indicates
how many ports the FPGA device has.
Global error reporting management (errors/)
error reporting sysfs interfaces allow user to read errors detected by the
hardware, and clear the logged errors.
FIU - PORT
==========
@ -143,6 +149,10 @@ More functions are exposed through sysfs:
Read Accelerator GUID (afu_id)
afu_id indicates which PR bitstream is programmed to this AFU.
Error reporting (errors/)
error reporting sysfs interfaces allow user to read port/afu errors
detected by the hardware, and clear the logged errors.
DFL Framework Overview
======================
@ -218,6 +228,101 @@ the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.
FPGA virtualization - PCIe SRIOV
================================
This section describes the virtualization support on DFL based FPGA device to
enable accessing an accelerator from applications running in a virtual machine
(VM). This section only describes the PCIe based FPGA device with SRIOV support.
Features supported by the particular FPGA device are exposed through Device
Feature Lists, as illustrated below:
::
+-------------------------------+ +-------------+
| PF | | VF |
+-------------------------------+ +-------------+
^ ^ ^ ^
| | | |
+-----|------------|---------|--------------|-------+
| | | | | |
| +-----+ +-------+ +-------+ +-------+ |
| | FME | | Port0 | | Port1 | | Port2 | |
| +-----+ +-------+ +-------+ +-------+ |
| ^ ^ ^ |
| | | | |
| +-------+ +------+ +-------+ |
| | AFU | | AFU | | AFU | |
| +-------+ +------+ +-------+ |
| |
| DFL based FPGA PCIe Device |
+---------------------------------------------------+
FME is always accessed through the physical function (PF).
Ports (and related AFUs) are accessed via PF by default, but could be exposed
through virtual function (VF) devices via PCIe SRIOV. Each VF only contains
1 Port and 1 AFU for isolation. Users could assign individual VFs (accelerators)
created via PCIe SRIOV interface, to virtual machines.
The driver organization in virtualization case is illustrated below:
::
+-------++------++------+ |
| FME || FME || FME | |
| FPGA || FPGA || FPGA | |
|Manager||Bridge||Region| |
+-------++------++------+ |
+-----------------------+ +--------+ | +--------+
| FME | | AFU | | | AFU |
| Module | | Module | | | Module |
+-----------------------+ +--------+ | +--------+
+-----------------------+ | +-----------------------+
| FPGA Container Device | | | FPGA Container Device |
| (FPGA Base Region) | | | (FPGA Base Region) |
+-----------------------+ | +-----------------------+
+------------------+ | +------------------+
| FPGA PCIE Module | | Virtual | FPGA PCIE Module |
+------------------+ Host | Machine +------------------+
-------------------------------------- | ------------------------------
+---------------+ | +---------------+
| PCI PF Device | | | PCI VF Device |
+---------------+ | +---------------+
FPGA PCIe device driver is always loaded first once a FPGA PCIe PF or VF device
is detected. It:
* Finishes enumeration on both FPGA PCIe PF and VF device using common
interfaces from DFL framework.
* Supports SRIOV.
The FME device driver plays a management role in this driver architecture, it
provides ioctls to release Port from PF and assign Port to PF. After release
a port from PF, then it's safe to expose this port through a VF via PCIe SRIOV
sysfs interface.
To enable accessing an accelerator from applications running in a VM, the
respective AFU's port needs to be assigned to a VF using the following steps:
#. The PF owns all AFU ports by default. Any port that needs to be
reassigned to a VF must first be released through the
DFL_FPGA_FME_PORT_RELEASE ioctl on the FME device.
#. Once N ports are released from PF, then user can use command below
to enable SRIOV and VFs. Each VF owns only one Port with AFU.
::
echo N > $PCI_DEVICE_PATH/sriov_numvfs
#. Pass through the VFs to VMs
#. The AFU under VF is accessible from applications in VM (using the
same driver inside the VF).
Note that an FME can't be assigned to a VF, thus PR and other management
functions are only available via the PF.
Device enumeration
==================
This section introduces how applications enumerate the fpga device from

View File

@ -554,9 +554,16 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto probe_failed;
}
if (device_add_groups(dev, drv->dev_groups)) {
dev_err(dev, "device_add_groups() failed\n");
goto dev_groups_failed;
}
if (test_remove) {
test_remove = false;
device_remove_groups(dev, drv->dev_groups);
if (dev->bus->remove)
dev->bus->remove(dev);
else if (drv->remove)
@ -584,6 +591,11 @@ static int really_probe(struct device *dev, struct device_driver *drv)
drv->bus->name, __func__, dev_name(dev), drv->name);
goto done;
dev_groups_failed:
if (dev->bus->remove)
dev->bus->remove(dev);
else if (drv->remove)
drv->remove(dev);
probe_failed:
if (dev->bus)
blocking_notifier_call_chain(&dev->bus->p->bus_notifier,
@ -1114,6 +1126,8 @@ static void __device_release_driver(struct device *dev, struct device *parent)
pm_runtime_put_sync(dev);
device_remove_groups(dev, drv->dev_groups);
if (dev->bus && dev->bus->remove)
dev->bus->remove(dev);
else if (drv->remove)

View File

@ -39,8 +39,9 @@ obj-$(CONFIG_FPGA_DFL_FME_BRIDGE) += dfl-fme-br.o
obj-$(CONFIG_FPGA_DFL_FME_REGION) += dfl-fme-region.o
obj-$(CONFIG_FPGA_DFL_AFU) += dfl-afu.o
dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o
dfl-fme-objs := dfl-fme-main.o dfl-fme-pr.o dfl-fme-error.o
dfl-afu-objs := dfl-afu-main.o dfl-afu-region.o dfl-afu-dma-region.o
dfl-afu-objs += dfl-afu-error.o
# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o

View File

@ -0,0 +1,230 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Driver for FPGA Accelerated Function Unit (AFU) Error Reporting
*
* Copyright 2019 Intel Corporation, Inc.
*
* Authors:
* Wu Hao <hao.wu@linux.intel.com>
* Xiao Guangrong <guangrong.xiao@linux.intel.com>
* Joseph Grecco <joe.grecco@intel.com>
* Enno Luebbers <enno.luebbers@intel.com>
* Tim Whisonant <tim.whisonant@intel.com>
* Ananda Ravuri <ananda.ravuri@intel.com>
* Mitchel Henry <henry.mitchel@intel.com>
*/
#include <linux/uaccess.h>
#include "dfl-afu.h"
#define PORT_ERROR_MASK 0x8
#define PORT_ERROR 0x10
#define PORT_FIRST_ERROR 0x18
#define PORT_MALFORMED_REQ0 0x20
#define PORT_MALFORMED_REQ1 0x28
#define ERROR_MASK GENMASK_ULL(63, 0)
/* mask or unmask port errors by the error mask register. */
static void __afu_port_err_mask(struct device *dev, bool mask)
{
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_ERROR);
writeq(mask ? ERROR_MASK : 0, base + PORT_ERROR_MASK);
}
static void afu_port_err_mask(struct device *dev, bool mask)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
mutex_lock(&pdata->lock);
__afu_port_err_mask(dev, mask);
mutex_unlock(&pdata->lock);
}
/* clear port errors. */
static int afu_port_err_clear(struct device *dev, u64 err)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
struct platform_device *pdev = to_platform_device(dev);
void __iomem *base_err, *base_hdr;
int ret = -EBUSY;
u64 v;
base_err = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_ERROR);
base_hdr = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
mutex_lock(&pdata->lock);
/*
* clear Port Errors
*
* - Check for AP6 State
* - Halt Port by keeping Port in reset
* - Set PORT Error mask to all 1 to mask errors
* - Clear all errors
* - Set Port mask to all 0 to enable errors
* - All errors start capturing new errors
* - Enable Port by pulling the port out of reset
*/
/* if device is still in AP6 power state, can not clear any error. */
v = readq(base_hdr + PORT_HDR_STS);
if (FIELD_GET(PORT_STS_PWR_STATE, v) == PORT_STS_PWR_STATE_AP6) {
dev_err(dev, "Could not clear errors, device in AP6 state.\n");
goto done;
}
/* Halt Port by keeping Port in reset */
ret = __afu_port_disable(pdev);
if (ret)
goto done;
/* Mask all errors */
__afu_port_err_mask(dev, true);
/* Clear errors if err input matches with current port errors.*/
v = readq(base_err + PORT_ERROR);
if (v == err) {
writeq(v, base_err + PORT_ERROR);
v = readq(base_err + PORT_FIRST_ERROR);
writeq(v, base_err + PORT_FIRST_ERROR);
} else {
ret = -EINVAL;
}
/* Clear mask */
__afu_port_err_mask(dev, false);
/* Enable the Port by clear the reset */
__afu_port_enable(pdev);
done:
mutex_unlock(&pdata->lock);
return ret;
}
static ssize_t errors_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 error;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_ERROR);
mutex_lock(&pdata->lock);
error = readq(base + PORT_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)error);
}
static ssize_t errors_store(struct device *dev, struct device_attribute *attr,
const char *buff, size_t count)
{
u64 value;
int ret;
if (kstrtou64(buff, 0, &value))
return -EINVAL;
ret = afu_port_err_clear(dev, value);
return ret ? ret : count;
}
static DEVICE_ATTR_RW(errors);
static ssize_t first_error_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 error;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_ERROR);
mutex_lock(&pdata->lock);
error = readq(base + PORT_FIRST_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)error);
}
static DEVICE_ATTR_RO(first_error);
static ssize_t first_malformed_req_show(struct device *dev,
struct device_attribute *attr,
char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 req0, req1;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_ERROR);
mutex_lock(&pdata->lock);
req0 = readq(base + PORT_MALFORMED_REQ0);
req1 = readq(base + PORT_MALFORMED_REQ1);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%016llx%016llx\n",
(unsigned long long)req1, (unsigned long long)req0);
}
static DEVICE_ATTR_RO(first_malformed_req);
static struct attribute *port_err_attrs[] = {
&dev_attr_errors.attr,
&dev_attr_first_error.attr,
&dev_attr_first_malformed_req.attr,
NULL,
};
static umode_t port_err_attrs_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
struct device *dev = kobj_to_dev(kobj);
/*
* sysfs entries are visible only if related private feature is
* enumerated.
*/
if (!dfl_get_feature_by_id(dev, PORT_FEATURE_ID_ERROR))
return 0;
return attr->mode;
}
const struct attribute_group port_err_group = {
.name = "errors",
.attrs = port_err_attrs,
.is_visible = port_err_attrs_visible,
};
static int port_err_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
afu_port_err_mask(&pdev->dev, false);
return 0;
}
static void port_err_uinit(struct platform_device *pdev,
struct dfl_feature *feature)
{
afu_port_err_mask(&pdev->dev, true);
}
const struct dfl_feature_id port_err_id_table[] = {
{.id = PORT_FEATURE_ID_ERROR,},
{0,}
};
const struct dfl_feature_ops port_err_ops = {
.init = port_err_init,
.uinit = port_err_uinit,
};

View File

@ -22,14 +22,17 @@
#include "dfl-afu.h"
/**
* port_enable - enable a port
* __afu_port_enable - enable a port by clear reset
* @pdev: port platform device.
*
* Enable Port by clear the port soft reset bit, which is set by default.
* The AFU is unable to respond to any MMIO access while in reset.
* port_enable function should only be used after port_disable function.
* __afu_port_enable function should only be used after __afu_port_disable
* function.
*
* The caller needs to hold lock for protection.
*/
static void port_enable(struct platform_device *pdev)
void __afu_port_enable(struct platform_device *pdev)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
void __iomem *base;
@ -52,13 +55,14 @@ static void port_enable(struct platform_device *pdev)
#define RST_POLL_TIMEOUT 1000 /* us */
/**
* port_disable - disable a port
* __afu_port_disable - disable a port by hold reset
* @pdev: port platform device.
*
* Disable Port by setting the port soft reset bit, it puts the port into
* reset.
* Disable Port by setting the port soft reset bit, it puts the port into reset.
*
* The caller needs to hold lock for protection.
*/
static int port_disable(struct platform_device *pdev)
int __afu_port_disable(struct platform_device *pdev)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(&pdev->dev);
void __iomem *base;
@ -104,9 +108,9 @@ static int __port_reset(struct platform_device *pdev)
{
int ret;
ret = port_disable(pdev);
ret = __afu_port_disable(pdev);
if (!ret)
port_enable(pdev);
__afu_port_enable(pdev);
return ret;
}
@ -274,32 +278,134 @@ power_state_show(struct device *dev, struct device_attribute *attr, char *buf)
}
static DEVICE_ATTR_RO(power_state);
static ssize_t
userclk_freqcmd_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
u64 userclk_freq_cmd;
void __iomem *base;
if (kstrtou64(buf, 0, &userclk_freq_cmd))
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
mutex_lock(&pdata->lock);
writeq(userclk_freq_cmd, base + PORT_HDR_USRCLK_CMD0);
mutex_unlock(&pdata->lock);
return count;
}
static DEVICE_ATTR_WO(userclk_freqcmd);
static ssize_t
userclk_freqcntrcmd_store(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
u64 userclk_freqcntr_cmd;
void __iomem *base;
if (kstrtou64(buf, 0, &userclk_freqcntr_cmd))
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
mutex_lock(&pdata->lock);
writeq(userclk_freqcntr_cmd, base + PORT_HDR_USRCLK_CMD1);
mutex_unlock(&pdata->lock);
return count;
}
static DEVICE_ATTR_WO(userclk_freqcntrcmd);
static ssize_t
userclk_freqsts_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
u64 userclk_freqsts;
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
mutex_lock(&pdata->lock);
userclk_freqsts = readq(base + PORT_HDR_USRCLK_STS0);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)userclk_freqsts);
}
static DEVICE_ATTR_RO(userclk_freqsts);
static ssize_t
userclk_freqcntrsts_show(struct device *dev, struct device_attribute *attr,
char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
u64 userclk_freqcntrsts;
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
mutex_lock(&pdata->lock);
userclk_freqcntrsts = readq(base + PORT_HDR_USRCLK_STS1);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n",
(unsigned long long)userclk_freqcntrsts);
}
static DEVICE_ATTR_RO(userclk_freqcntrsts);
static struct attribute *port_hdr_attrs[] = {
&dev_attr_id.attr,
&dev_attr_ltr.attr,
&dev_attr_ap1_event.attr,
&dev_attr_ap2_event.attr,
&dev_attr_power_state.attr,
&dev_attr_userclk_freqcmd.attr,
&dev_attr_userclk_freqcntrcmd.attr,
&dev_attr_userclk_freqsts.attr,
&dev_attr_userclk_freqcntrsts.attr,
NULL,
};
ATTRIBUTE_GROUPS(port_hdr);
static umode_t port_hdr_attrs_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
struct device *dev = kobj_to_dev(kobj);
umode_t mode = attr->mode;
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, PORT_FEATURE_ID_HEADER);
if (dfl_feature_revision(base) > 0) {
/*
* userclk sysfs interfaces are only visible in case port
* revision is 0, as hardware with revision >0 doesn't
* support this.
*/
if (attr == &dev_attr_userclk_freqcmd.attr ||
attr == &dev_attr_userclk_freqcntrcmd.attr ||
attr == &dev_attr_userclk_freqsts.attr ||
attr == &dev_attr_userclk_freqcntrsts.attr)
mode = 0;
}
return mode;
}
static const struct attribute_group port_hdr_group = {
.attrs = port_hdr_attrs,
.is_visible = port_hdr_attrs_visible,
};
static int port_hdr_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
dev_dbg(&pdev->dev, "PORT HDR Init.\n");
port_reset(pdev);
return device_add_groups(&pdev->dev, port_hdr_groups);
}
static void port_hdr_uinit(struct platform_device *pdev,
struct dfl_feature *feature)
{
dev_dbg(&pdev->dev, "PORT HDR UInit.\n");
device_remove_groups(&pdev->dev, port_hdr_groups);
return 0;
}
static long
@ -330,7 +436,6 @@ static const struct dfl_feature_id port_hdr_id_table[] = {
static const struct dfl_feature_ops port_hdr_ops = {
.init = port_hdr_init,
.uinit = port_hdr_uinit,
.ioctl = port_hdr_ioctl,
};
@ -361,32 +466,37 @@ static struct attribute *port_afu_attrs[] = {
&dev_attr_afu_id.attr,
NULL
};
ATTRIBUTE_GROUPS(port_afu);
static umode_t port_afu_attrs_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
struct device *dev = kobj_to_dev(kobj);
/*
* sysfs entries are visible only if related private feature is
* enumerated.
*/
if (!dfl_get_feature_by_id(dev, PORT_FEATURE_ID_AFU))
return 0;
return attr->mode;
}
static const struct attribute_group port_afu_group = {
.attrs = port_afu_attrs,
.is_visible = port_afu_attrs_visible,
};
static int port_afu_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
struct resource *res = &pdev->resource[feature->resource_index];
int ret;
dev_dbg(&pdev->dev, "PORT AFU Init.\n");
ret = afu_mmio_region_add(dev_get_platdata(&pdev->dev),
DFL_PORT_REGION_INDEX_AFU, resource_size(res),
res->start, DFL_PORT_REGION_READ |
DFL_PORT_REGION_WRITE | DFL_PORT_REGION_MMAP);
if (ret)
return ret;
return device_add_groups(&pdev->dev, port_afu_groups);
}
static void port_afu_uinit(struct platform_device *pdev,
struct dfl_feature *feature)
{
dev_dbg(&pdev->dev, "PORT AFU UInit.\n");
device_remove_groups(&pdev->dev, port_afu_groups);
return afu_mmio_region_add(dev_get_platdata(&pdev->dev),
DFL_PORT_REGION_INDEX_AFU,
resource_size(res), res->start,
DFL_PORT_REGION_MMAP | DFL_PORT_REGION_READ |
DFL_PORT_REGION_WRITE);
}
static const struct dfl_feature_id port_afu_id_table[] = {
@ -396,7 +506,27 @@ static const struct dfl_feature_id port_afu_id_table[] = {
static const struct dfl_feature_ops port_afu_ops = {
.init = port_afu_init,
.uinit = port_afu_uinit,
};
static int port_stp_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
struct resource *res = &pdev->resource[feature->resource_index];
return afu_mmio_region_add(dev_get_platdata(&pdev->dev),
DFL_PORT_REGION_INDEX_STP,
resource_size(res), res->start,
DFL_PORT_REGION_MMAP | DFL_PORT_REGION_READ |
DFL_PORT_REGION_WRITE);
}
static const struct dfl_feature_id port_stp_id_table[] = {
{.id = PORT_FEATURE_ID_STP,},
{0,}
};
static const struct dfl_feature_ops port_stp_ops = {
.init = port_stp_init,
};
static struct dfl_feature_driver port_feature_drvs[] = {
@ -408,6 +538,14 @@ static struct dfl_feature_driver port_feature_drvs[] = {
.id_table = port_afu_id_table,
.ops = &port_afu_ops,
},
{
.id_table = port_err_id_table,
.ops = &port_err_ops,
},
{
.id_table = port_stp_id_table,
.ops = &port_stp_ops,
},
{
.ops = NULL,
}
@ -694,9 +832,9 @@ static int port_enable_set(struct platform_device *pdev, bool enable)
mutex_lock(&pdata->lock);
if (enable)
port_enable(pdev);
__afu_port_enable(pdev);
else
ret = port_disable(pdev);
ret = __afu_port_disable(pdev);
mutex_unlock(&pdata->lock);
return ret;
@ -748,9 +886,17 @@ static int afu_remove(struct platform_device *pdev)
return 0;
}
static const struct attribute_group *afu_dev_groups[] = {
&port_hdr_group,
&port_afu_group,
&port_err_group,
NULL
};
static struct platform_driver afu_driver = {
.driver = {
.name = DFL_FPGA_FEATURE_DEV_PORT,
.name = DFL_FPGA_FEATURE_DEV_PORT,
.dev_groups = afu_dev_groups,
},
.probe = afu_probe,
.remove = afu_remove,

View File

@ -79,6 +79,10 @@ struct dfl_afu {
struct dfl_feature_platform_data *pdata;
};
/* hold pdata->lock when call __afu_port_enable/disable */
void __afu_port_enable(struct platform_device *pdev);
int __afu_port_disable(struct platform_device *pdev);
void afu_mmio_region_init(struct dfl_feature_platform_data *pdata);
int afu_mmio_region_add(struct dfl_feature_platform_data *pdata,
u32 region_index, u64 region_size, u64 phys, u32 flags);
@ -97,4 +101,9 @@ int afu_dma_unmap_region(struct dfl_feature_platform_data *pdata, u64 iova);
struct dfl_afu_dma_region *
afu_dma_region_find(struct dfl_feature_platform_data *pdata,
u64 iova, u64 size);
extern const struct dfl_feature_ops port_err_ops;
extern const struct dfl_feature_id port_err_id_table[];
extern const struct attribute_group port_err_group;
#endif /* __DFL_AFU_H */

View File

@ -0,0 +1,359 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Driver for FPGA Management Engine Error Management
*
* Copyright 2019 Intel Corporation, Inc.
*
* Authors:
* Kang Luwei <luwei.kang@intel.com>
* Xiao Guangrong <guangrong.xiao@linux.intel.com>
* Wu Hao <hao.wu@intel.com>
* Joseph Grecco <joe.grecco@intel.com>
* Enno Luebbers <enno.luebbers@intel.com>
* Tim Whisonant <tim.whisonant@intel.com>
* Ananda Ravuri <ananda.ravuri@intel.com>
* Mitchel, Henry <henry.mitchel@intel.com>
*/
#include <linux/uaccess.h>
#include "dfl.h"
#include "dfl-fme.h"
#define FME_ERROR_MASK 0x8
#define FME_ERROR 0x10
#define MBP_ERROR BIT_ULL(6)
#define PCIE0_ERROR_MASK 0x18
#define PCIE0_ERROR 0x20
#define PCIE1_ERROR_MASK 0x28
#define PCIE1_ERROR 0x30
#define FME_FIRST_ERROR 0x38
#define FME_NEXT_ERROR 0x40
#define RAS_NONFAT_ERROR_MASK 0x48
#define RAS_NONFAT_ERROR 0x50
#define RAS_CATFAT_ERROR_MASK 0x58
#define RAS_CATFAT_ERROR 0x60
#define RAS_ERROR_INJECT 0x68
#define INJECT_ERROR_MASK GENMASK_ULL(2, 0)
#define ERROR_MASK GENMASK_ULL(63, 0)
static ssize_t pcie0_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 value;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
value = readq(base + PCIE0_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)value);
}
static ssize_t pcie0_errors_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
int ret = 0;
u64 v, val;
if (kstrtou64(buf, 0, &val))
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
writeq(GENMASK_ULL(63, 0), base + PCIE0_ERROR_MASK);
v = readq(base + PCIE0_ERROR);
if (val == v)
writeq(v, base + PCIE0_ERROR);
else
ret = -EINVAL;
writeq(0ULL, base + PCIE0_ERROR_MASK);
mutex_unlock(&pdata->lock);
return ret ? ret : count;
}
static DEVICE_ATTR_RW(pcie0_errors);
static ssize_t pcie1_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 value;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
value = readq(base + PCIE1_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)value);
}
static ssize_t pcie1_errors_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
int ret = 0;
u64 v, val;
if (kstrtou64(buf, 0, &val))
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
writeq(GENMASK_ULL(63, 0), base + PCIE1_ERROR_MASK);
v = readq(base + PCIE1_ERROR);
if (val == v)
writeq(v, base + PCIE1_ERROR);
else
ret = -EINVAL;
writeq(0ULL, base + PCIE1_ERROR_MASK);
mutex_unlock(&pdata->lock);
return ret ? ret : count;
}
static DEVICE_ATTR_RW(pcie1_errors);
static ssize_t nonfatal_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
return sprintf(buf, "0x%llx\n",
(unsigned long long)readq(base + RAS_NONFAT_ERROR));
}
static DEVICE_ATTR_RO(nonfatal_errors);
static ssize_t catfatal_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
return sprintf(buf, "0x%llx\n",
(unsigned long long)readq(base + RAS_CATFAT_ERROR));
}
static DEVICE_ATTR_RO(catfatal_errors);
static ssize_t inject_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 v;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
v = readq(base + RAS_ERROR_INJECT);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n",
(unsigned long long)FIELD_GET(INJECT_ERROR_MASK, v));
}
static ssize_t inject_errors_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u8 inject_error;
u64 v;
if (kstrtou8(buf, 0, &inject_error))
return -EINVAL;
if (inject_error & ~INJECT_ERROR_MASK)
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
v = readq(base + RAS_ERROR_INJECT);
v &= ~INJECT_ERROR_MASK;
v |= FIELD_PREP(INJECT_ERROR_MASK, inject_error);
writeq(v, base + RAS_ERROR_INJECT);
mutex_unlock(&pdata->lock);
return count;
}
static DEVICE_ATTR_RW(inject_errors);
static ssize_t fme_errors_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 value;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
value = readq(base + FME_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)value);
}
static ssize_t fme_errors_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 v, val;
int ret = 0;
if (kstrtou64(buf, 0, &val))
return -EINVAL;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
writeq(GENMASK_ULL(63, 0), base + FME_ERROR_MASK);
v = readq(base + FME_ERROR);
if (val == v)
writeq(v, base + FME_ERROR);
else
ret = -EINVAL;
/* Workaround: disable MBP_ERROR if feature revision is 0 */
writeq(dfl_feature_revision(base) ? 0ULL : MBP_ERROR,
base + FME_ERROR_MASK);
mutex_unlock(&pdata->lock);
return ret ? ret : count;
}
static DEVICE_ATTR_RW(fme_errors);
static ssize_t first_error_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 value;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
value = readq(base + FME_FIRST_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)value);
}
static DEVICE_ATTR_RO(first_error);
static ssize_t next_error_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
u64 value;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
value = readq(base + FME_NEXT_ERROR);
mutex_unlock(&pdata->lock);
return sprintf(buf, "0x%llx\n", (unsigned long long)value);
}
static DEVICE_ATTR_RO(next_error);
static struct attribute *fme_global_err_attrs[] = {
&dev_attr_pcie0_errors.attr,
&dev_attr_pcie1_errors.attr,
&dev_attr_nonfatal_errors.attr,
&dev_attr_catfatal_errors.attr,
&dev_attr_inject_errors.attr,
&dev_attr_fme_errors.attr,
&dev_attr_first_error.attr,
&dev_attr_next_error.attr,
NULL,
};
static umode_t fme_global_err_attrs_visible(struct kobject *kobj,
struct attribute *attr, int n)
{
struct device *dev = kobj_to_dev(kobj);
/*
* sysfs entries are visible only if related private feature is
* enumerated.
*/
if (!dfl_get_feature_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR))
return 0;
return attr->mode;
}
const struct attribute_group fme_global_err_group = {
.name = "errors",
.attrs = fme_global_err_attrs,
.is_visible = fme_global_err_attrs_visible,
};
static void fme_err_mask(struct device *dev, bool mask)
{
struct dfl_feature_platform_data *pdata = dev_get_platdata(dev);
void __iomem *base;
base = dfl_get_feature_ioaddr_by_id(dev, FME_FEATURE_ID_GLOBAL_ERR);
mutex_lock(&pdata->lock);
/* Workaround: keep MBP_ERROR always masked if revision is 0 */
if (dfl_feature_revision(base))
writeq(mask ? ERROR_MASK : 0, base + FME_ERROR_MASK);
else
writeq(mask ? ERROR_MASK : MBP_ERROR, base + FME_ERROR_MASK);
writeq(mask ? ERROR_MASK : 0, base + PCIE0_ERROR_MASK);
writeq(mask ? ERROR_MASK : 0, base + PCIE1_ERROR_MASK);
writeq(mask ? ERROR_MASK : 0, base + RAS_NONFAT_ERROR_MASK);
writeq(mask ? ERROR_MASK : 0, base + RAS_CATFAT_ERROR_MASK);
mutex_unlock(&pdata->lock);
}
static int fme_global_err_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
fme_err_mask(&pdev->dev, false);
return 0;
}
static void fme_global_err_uinit(struct platform_device *pdev,
struct dfl_feature *feature)
{
fme_err_mask(&pdev->dev, true);
}
const struct dfl_feature_id fme_global_err_id_table[] = {
{.id = FME_FEATURE_ID_GLOBAL_ERR,},
{0,}
};
const struct dfl_feature_ops fme_global_err_ops = {
.init = fme_global_err_init,
.uinit = fme_global_err_uinit,
};

View File

@ -127,31 +127,10 @@ static struct attribute *fme_hdr_attrs[] = {
&dev_attr_socket_id.attr,
NULL,
};
ATTRIBUTE_GROUPS(fme_hdr);
static int fme_hdr_init(struct platform_device *pdev,
struct dfl_feature *feature)
{
void __iomem *base = feature->ioaddr;
int ret;
dev_dbg(&pdev->dev, "FME HDR Init.\n");
dev_dbg(&pdev->dev, "FME cap %llx.\n",
(unsigned long long)readq(base + FME_HDR_CAP));
ret = device_add_groups(&pdev->dev, fme_hdr_groups);
if (ret)
return ret;
return 0;
}
static void fme_hdr_uinit(struct platform_device *pdev,
struct dfl_feature *feature)
{
dev_dbg(&pdev->dev, "FME HDR UInit.\n");
device_remove_groups(&pdev->dev, fme_hdr_groups);
}
static const struct attribute_group fme_hdr_group = {
.attrs = fme_hdr_attrs,
};
static long fme_hdr_ioctl_release_port(struct dfl_feature_platform_data *pdata,
unsigned long arg)
@ -199,8 +178,6 @@ static const struct dfl_feature_id fme_hdr_id_table[] = {
};
static const struct dfl_feature_ops fme_hdr_ops = {
.init = fme_hdr_init,
.uinit = fme_hdr_uinit,
.ioctl = fme_hdr_ioctl,
};
@ -213,6 +190,10 @@ static struct dfl_feature_driver fme_feature_drvs[] = {
.id_table = fme_pr_mgmt_id_table,
.ops = &fme_pr_mgmt_ops,
},
{
.id_table = fme_global_err_id_table,
.ops = &fme_global_err_ops,
},
{
.ops = NULL,
},
@ -359,9 +340,16 @@ static int fme_remove(struct platform_device *pdev)
return 0;
}
static const struct attribute_group *fme_dev_groups[] = {
&fme_hdr_group,
&fme_global_err_group,
NULL
};
static struct platform_driver fme_driver = {
.driver = {
.name = DFL_FPGA_FEATURE_DEV_FME,
.name = DFL_FPGA_FEATURE_DEV_FME,
.dev_groups = fme_dev_groups,
},
.probe = fme_probe,
.remove = fme_remove,

View File

@ -35,5 +35,8 @@ struct dfl_fme {
extern const struct dfl_feature_ops fme_pr_mgmt_ops;
extern const struct dfl_feature_id fme_pr_mgmt_id_table[];
extern const struct dfl_feature_ops fme_global_err_ops;
extern const struct dfl_feature_id fme_global_err_id_table[];
extern const struct attribute_group fme_global_err_group;
#endif /* __DFL_FME_H */

View File

@ -271,11 +271,13 @@ static int dfl_feature_instance_init(struct platform_device *pdev,
struct dfl_feature *feature,
struct dfl_feature_driver *drv)
{
int ret;
int ret = 0;
ret = drv->ops->init(pdev, feature);
if (ret)
return ret;
if (drv->ops->init) {
ret = drv->ops->init(pdev, feature);
if (ret)
return ret;
}
feature->ops = drv->ops;

View File

@ -120,6 +120,10 @@
#define PORT_HDR_CAP 0x30
#define PORT_HDR_CTRL 0x38
#define PORT_HDR_STS 0x40
#define PORT_HDR_USRCLK_CMD0 0x50
#define PORT_HDR_USRCLK_CMD1 0x58
#define PORT_HDR_USRCLK_STS0 0x60
#define PORT_HDR_USRCLK_STS1 0x68
/* Port Capability Register Bitfield */
#define PORT_CAP_PORT_NUM GENMASK_ULL(1, 0) /* ID of this port */
@ -355,6 +359,11 @@ static inline bool dfl_feature_is_port(void __iomem *base)
(FIELD_GET(DFH_ID, v) == DFH_ID_FIU_PORT);
}
static inline u8 dfl_feature_revision(void __iomem *base)
{
return (u8)FIELD_GET(DFH_REVISION, readq(base + DFH));
}
/**
* struct dfl_fpga_enum_info - DFL FPGA enumeration information
*

View File

@ -262,6 +262,8 @@ enum probe_type {
* @resume: Called to bring a device from sleep mode.
* @groups: Default attributes that get created by the driver core
* automatically.
* @dev_groups: Additional attributes attached to device instance once the
* it is bound to the driver.
* @pm: Power management operations of the device which matched
* this driver.
* @coredump: Called when sysfs entry is written to. The device driver
@ -296,6 +298,7 @@ struct device_driver {
int (*suspend) (struct device *dev, pm_message_t state);
int (*resume) (struct device *dev);
const struct attribute_group **groups;
const struct attribute_group **dev_groups;
const struct dev_pm_ops *pm;
void (*coredump) (struct device *dev);