Make sure to use dma_sync_single_for_device whenever updating the
contents of an extended page table. This performs the cache flush using
the correct function from the DMA-API. Also move this call outside of
the inner loop to improve performance.
Tested: Ran run_tests with inception, testfullyconnected with parameter
caching, and rnntransducerdecoderpie107m on AT Enterprise.
Change-Id: Ie3c63113dde8cafe448d515e8970810afc9564ac
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Add that number of page table entries and extended address bit offset
are configurable. Update example virtual address format to be more
consistent with typical usage.
Change-Id: I73f9674df74b6d8ee5ed6e5d95ab12dcc56e29cc
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Add a brief description and URL for more information on the Apex device,
an Edge TPU (Tensorflow Processing Unit) machine learning accelerator.
Change-Id: Ia7b40414e9f0745feb8ced2cb3aa7e9c6baa0c3d
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Add support for platform devices in the gasket framework.
Change-Id: I1e3625f9222b37925a2a672c9ea6eed31aefa35f
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Add interrupt type DEVICE_MANAGED, indicating that the device driver
manages interrupt setup and handling. Future non-PCI wire interrupts
will use this type, calling gasket_handle_interrupt(), which is now made
non-static, to call into the gasket framework in order to update sysfs
files for interrupt counts and other framework interfaces.
Change-Id: Ie0be3d950ed2706f7ada848c19ddf7017e9623b2
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Refactor gasket_components_to_dev_address to be faster and easier to
understand. The old implementation was unnecessarily complex and masked
the page_index for simple addresses but not extended ones. It makes the
most sense for this function to perform no such masking.
Change-Id: If5f68cb6f1bf3000a5d5b77aab89cb8e391c3cf4
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
The maximum number of entries in the page table is configurable at
initialization time and should be used in gasket_extended_lvl0_page_idx.
Change-Id: I39f2dba48083a3eef3951ecd9659ee9f4901275e
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Previously pages would have never been unmapped in this case.
Change-Id: I71b6447267933865ab23a3ba3830e7ce3d3e0e57
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Correctly clean up data structure state in gasket_alloc_coherent_memory
error path, to ensure no double free on the stale pointer value.
Change-Id: Ifbc426e857d6bd05a13a725f2c2fdb07dedc098c
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Only call dma_unmap_page if there was an associated dma_map_page.
Change-Id: I6234719f8ab6b1e5e069c21e2f639eee4f22b039
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
This allows for more precise dma_direction in the dma_map_page requests.
Also leaves room for adding more flags later.
Change-Id: I46feecc2638b7ad0d34548cc10837549a3928011
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Remove unnecessary ptes[i].status update in gasket_perform_unmapping.
The vlaue will be cleared in the following memset.
Change-Id: I5aeafa1085203b5dc24b3303037ece1f37acd6c6
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Extended page tables can be mapped as DMA_TO_DEVICE.
Calls to dma_unmap_page should match corresponding dma_map_page.
Change-Id: I2d09fd5d66c364e190b073fabe089ce5eaf426f6
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Signed-off-by: Todd Poynor <toddpoynor@google.com>
gasket_alloc_coherent_memory() has some unnecessary code related to out
of memory checking that will never hit the condition checked, remove.
Change-Id: Ifb09e7ff31d8ae41a486639a2abc6ce26c4d85ef
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Return -EINVAL on mapping failures, instead of -1, which triggers a
checkpatch error.
Change-Id: I91a62a72caa5ca09e07bd4897e72f4919495c040
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Add debug logs for device enable/disable events, remove logs for
callbacks (the called functions can generate their own logs if needed).
Change-Id: I861077c3210452d9ac6606e973bac5c14eeef859
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Devices not using MSIX don't use the msix_initialized field, don't
require it to be set in the interrupt system status check. The general
check for interrupts configured that follows can cover both MSIX and
device-managed interrupts.
Change-Id: I83a86cb4db7cea30c30377dd82b033ef372d69c0
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Flags should be specified for dma_alloc_coherent() call. Use
GFP_KERNEL, it's fine to sleep here.
Change-Id: Iec4ad9550e8829b67e99b22246e40e0b21600c54
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Actually free gasket_coherent_page_entries and just call
gasket_free_coherent_memory_all from gasket_free_coherent_memory.
Change-Id: I1d016abd5f66e0620a37f6851e43fcfd6c96868a
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Add module parameters to control default values for HW temperature
monitoring module.
Signed-off-by: Leonid Lobachev <leonidl@google.com>
Change-Id: I69f7cd9c3c0f55a3a36efba35f48a5da37b3b89e
Perform periodic polling of apex temperature and scale down
its frequency if temperature is above one of defined trip points.
Added sysfs nodes to control related settings on per device basis.
Added module parameters to set allow setting default value for all
devices in system.
Signed-off-by: Leonid Lobachev <leonidl@google.com>
Change-Id: I5065282cba9f0888abd4086601bda89174f24705
Port of change by Rob Springer to fix leak of eventfd context objects in
gasket framework interrupt code.
Bug: 131928433
Bug: 80495595
Reported-by: Jann Horn <jannh@google.com>
Reported-by: Neel Natu <neelnatu@google.com>
Test: Resnet50PB, PortraitSegmentation
Change-Id: I6bdea5ab4d3a7435e7f51c96817f0542d185b22c
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Add node to read current temperature and nodes
to set and enable/disable thermal warnings.
Change-Id: I8a12f6e33679573107686fe32be2e4a8bb0f9cc7
Signed-off-by: Leonid Lobachev <leonidl@google.com>
For driver build as module pci_fixup_header approach doesn't work.
This causes apex pci class to remain 0xff and BARs remain
unclaimed. Adding a retry with explicit fixup function call
workarounds the problem.
Bug: 129287752
Change-Id: Id5ff85ef427d3740706a214d3beb1c9bbf42ab5a
Signed-off-by: Leonid Lobachev <leonidl@google.com>
sysfs_show was incorrectly extracting the sysfs_attribute_type from the
gasket_sysfs_attribute. This prevented dispatch from working properly.
Change-Id: Ib962a10cf24bd4be416708e0a6294b6db431c76f
Signed-off-by: Nick Ewalt <nicholasewalt@google.com>
Add a compat_ioctl handler for arm32 and replace ulong to u64.
Bug: 115850337, 117133232
Test: label_image runs correctly.
Change-Id: Ie17244faff2aa1461397e740fba4581c6b112f30
Pass the gasket driver descriptor to the interrupt init function, rather
than exploding out separate parameters from various fields of that
structure. This allows us to make more localized changes to the types
of interrupts supported (MSIX vs. wire, etc.) without affecting the
calling sequence, and seems nicer for simplification purposes.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Split interrupt handler into PCI MSIX-specific and generic functions,
for adding non-MSIX handlers in the future. Move MSIX init code
together,, out of generic init path.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Rename lookup_internal_desc() to lookup_pci_internal_desc() to reflect
use for PCI devices only, in prep for non-PCI devices in the future.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Separate code for generic parts of gasket device removal sequence from
the PCI device removal code, in prep for non-PCI devices later.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Split out generic gasket device add code from the code for adding a PCI
gasket device, in prep for other gasket device types in the future.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Some explicit memory barriers in the page table code are not necessary,
either because:
(a) The barrier follows a non-relaxed MMIO access that already performs
a read or write memory barrier.
(b) The barrier follows DMA API calls for which the device-visible
effects of IOMMU programming are guaranteed to be flushed to the IOMMU
prior to the call returning, and doesn't need to sync with normal memory
access.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Use of readl() is deprecated; readl_relaxed() with appropriate memory
barriers is preferred. Switch to relaxed reads and writes for better
performance as well. Memory barriers required for I/O vs. normal
memory access on Apex devices have already been explicitly coded in the
page table routines.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Remove debug logs that only indicate the name of the entered function,
in favor of using ftrace for function tracing style logs.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
gasket_alloc_dev can retrieve the device name from the parent parameter,
a separate parameter isn't needed for this. Rename the variable to
better reflect its meaning, as the name of the parent device for which a
gasket device is being allocated.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Previous cleanups missed a case of multi-line function call with line
continuation parameters not aligned per kernel style.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
A debug log in gasket_alloc_dev() is issued regardless of whether the
device pointer used returned success or error. The log isn't that
useful anyway, remove it.
Signed-off-by: Todd Poynor <toddpoynor@google.com>
Reviewed-by: Rob Springer <rspringer@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Array cm_mappable_regions is defined but is never used hence it is
redundant and can be removed.
Cleans up clang warning:
warning: 'cm_mappable_regions' defined but not used [-Wunused-const-variable=]
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Remove null ptr check before kfree because kfree is null ptr safe.
Issue found by checkpatch.
Signed-off-by: Sumit Kumar <sumit686215@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
