Yafang Shao 765437d1f0 mm, oom: make the calculation of oom badness more accurate ... [ Upstream commit 9066e5cfb73cdbcdbb49e87999482ab615e9fc76 ] Recently we found an issue on our production environment that when memcg oom is triggered the oom killer doesn't chose the process with largest resident memory but chose the first scanned process. Note that all processes in this memcg have the same oom_score_adj, so the oom killer should chose the process with largest resident memory. Bellow is part of the oom info, which is enough to analyze this issue. [7516987.983223] memory: usage 16777216kB, limit 16777216kB, failcnt 52843037 [7516987.983224] memory+swap: usage 16777216kB, limit 9007199254740988kB, failcnt 0 [7516987.983225] kmem: usage 301464kB, limit 9007199254740988kB, failcnt 0 [...] [7516987.983293] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [7516987.983510] [ 5740] 0 5740 257 1 32768 0 -998 pause [7516987.983574] [58804] 0 58804 4594 771 81920 0 -998 entry_point.bas [7516987.983577] [58908] 0 58908 7089 689 98304 0 -998 cron [7516987.983580] [58910] 0 58910 16235 5576 163840 0 -998 supervisord [7516987.983590] [59620] 0 59620 18074 1395 188416 0 -998 sshd [7516987.983594] [59622] 0 59622 18680 6679 188416 0 -998 python [7516987.983598] [59624] 0 59624 1859266 5161 548864 0 -998 odin-agent [7516987.983600] [59625] 0 59625 707223 9248 983040 0 -998 filebeat [7516987.983604] [59627] 0 59627 416433 64239 774144 0 -998 odin-log-agent [7516987.983607] [59631] 0 59631 180671 15012 385024 0 -998 python3 [7516987.983612] [61396] 0 61396 791287 3189 352256 0 -998 client [7516987.983615] [61641] 0 61641 1844642 29089 946176 0 -998 client [7516987.983765] [ 9236] 0 9236 2642 467 53248 0 -998 php_scanner [7516987.983911] [42898] 0 42898 15543 838 167936 0 -998 su [7516987.983915] [42900] 1000 42900 3673 867 77824 0 -998 exec_script_vr2 [7516987.983918] [42925] 1000 42925 36475 19033 335872 0 -998 python [7516987.983921] [57146] 1000 57146 3673 848 73728 0 -998 exec_script_J2p [7516987.983925] [57195] 1000 57195 186359 22958 491520 0 -998 python2 [7516987.983928] [58376] 1000 58376 275764 14402 290816 0 -998 rosmaster [7516987.983931] [58395] 1000 58395 155166 4449 245760 0 -998 rosout [7516987.983935] [58406] 1000 58406 18285584 3967322 37101568 0 -998 data_sim [7516987.984221] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=3aa16c9482ae3a6f6b78bda68a55d32c87c99b985e0f11331cddf05af6c4d753,mems_allowed=0-1,oom_memcg=/kubepods/podf1c273d3-9b36-11ea-b3df-246e9693c184,task_memcg=/kubepods/podf1c273d3-9b36-11ea-b3df-246e9693c184/1f246a3eeea8f70bf91141eeaf1805346a666e225f823906485ea0b6c37dfc3d,task=pause,pid=5740,uid=0 [7516987.984254] Memory cgroup out of memory: Killed process 5740 (pause) total-vm:1028kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB [7516988.092344] oom_reaper: reaped process 5740 (pause), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB We can find that the first scanned process 5740 (pause) was killed, but its rss is only one page. That is because, when we calculate the oom badness in oom_badness(), we always ignore the negtive point and convert all of these negtive points to 1. Now as oom_score_adj of all the processes in this targeted memcg have the same value -998, the points of these processes are all negtive value. As a result, the first scanned process will be killed. The oom_socre_adj (-998) in this memcg is set by kubelet, because it is a a Guaranteed pod, which has higher priority to prevent from being killed by system oom. To fix this issue, we should make the calculation of oom point more accurate. We can achieve it by convert the chosen_point from 'unsigned long' to 'long'. [cai@lca.pw: reported a issue in the previous version] [mhocko@suse.com: fixed the issue reported by Cai] [mhocko@suse.com: add the comment in proc_oom_score()] [laoar.shao@gmail.com: v3] Link: http://lkml.kernel.org/r/1594396651-9931-1-git-send-email-laoar.shao@gmail.com Signed-off-by: Yafang Shao <laoar.shao@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Tested-by: Naresh Kamboju <naresh.kamboju@linaro.org> Acked-by: Michal Hocko <mhocko@suse.com> Cc: David Rientjes <rientjes@google.com> Cc: Qian Cai <cai@lca.pw> Link: http://lkml.kernel.org/r/1594309987-9919-1-git-send-email-laoar.shao@gmail.com Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Sasha Levin <sashal@kernel.org>		2021-09-03 10:08:12 +02:00
..
kasan	kasan: fix incorrect arguments passing in kasan_add_zero_shadow	2021-01-27 11:47:53 +01:00
Kconfig	mm/zsmalloc.c: drop ZSMALLOC_PGTABLE_MAPPING	2020-12-16 10:56:59 +01:00
Kconfig.debug	mm, page_owner, debug_pagealloc: save and dump freeing stack trace	2019-09-24 15:54:08 -07:00
Makefile	mm: silence -Woverride-init/initializer-overrides	2019-09-24 15:54:10 -07:00
backing-dev.c	bdi: add a ->dev_name field to struct backing_dev_info	2020-05-14 07:58:30 +02:00
balloon_compaction.c	mm/balloon_compaction: suppress allocation warnings	2019-09-04 07:42:01 -04:00
cleancache.c	Driver Core and debugfs changes for 5.3-rc1	2019-07-12 12:24:03 -07:00
cma.c	cma: don't quit at first error when activating reserved areas	2020-09-03 11:26:51 +02:00
cma.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
cma_debug.c	mm/cma_debug.c: fix the break condition in cma_maxchunk_get()	2019-05-14 09:47:45 -07:00
compaction.c	mm/compaction: fix misbehaviors of fast_find_migrateblock()	2021-03-04 10:26:39 +01:00
debug.c	mm/debug.c: always print flags in dump_page()	2020-03-05 16:43:51 +01:00
debug_page_ref.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
dmapool.c	mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options	2019-07-12 11:05:46 -07:00
early_ioremap.c	mm/early_ioremap: Fix boot hang with earlyprintk=efi,keep	2017-12-11 14:54:44 +01:00
fadvise.c	fs: Export generic_fadvise()	2019-08-30 22:43:58 -07:00
failslab.c	mm/failslab.c: by default, do not fail allocations with direct reclaim only	2019-07-12 11:05:43 -07:00
filemap.c	mm/filemap: fix storing to a THP shadow entry	2021-06-10 13:37:15 +02:00
frame_vector.c	mm: untag user pointers in get_vaddr_frames	2019-09-25 17:51:41 -07:00
frontswap.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 482	2019-06-19 17:09:52 +02:00
gup.c	mm/gup: fix gup_fast with dynamic page table folding	2020-10-01 13:18:24 +02:00
gup_benchmark.c	mm/gup: fix memory leak in __gup_benchmark_ioctl	2020-01-09 10:20:00 +01:00
highmem.c	mm: convert totalram_pages and totalhigh_pages variables to atomic	2018-12-28 12:11:47 -08:00
hmm.c	pagewalk: separate function pointers from iterator data	2019-09-07 04:28:04 -03:00
huge_memory.c	mm/huge_memory.c: don't discard hugepage if other processes are mapping it	2021-07-14 16:53:47 +02:00
hugetlb.c	mm, futex: fix shared futex pgoff on shmem huge page	2021-06-30 08:47:55 -04:00
hugetlb_cgroup.c	mm: hugetlb: switch to css_tryget() in hugetlb_cgroup_charge_cgroup()	2019-11-15 18:34:00 -08:00
hwpoison-inject.c	hwpoison-inject: no need to check return value of debugfs_create functions	2019-06-03 15:39:40 +02:00
init-mm.c	mm/init-mm.c: include <linux/mman.h> for vm_committed_as_batch	2019-10-19 06:32:32 -04:00
internal.h	mm/thp: fix vma_address() if virtual address below file offset	2021-06-30 08:47:52 -04:00
interval_tree.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 248	2019-06-19 17:09:08 +02:00
khugepaged.c	khugepaged: fix wrong result value for trace_mm_collapse_huge_page_isolate()	2021-05-19 10:08:27 +02:00
kmemleak-test.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 333	2019-06-05 17:37:06 +02:00
kmemleak.c	mm/kmemleak.c: use address-of operator on section symbols	2020-10-01 13:17:53 +02:00
ksm.c	ksm: fix potential missing rmap_item for stable_node	2021-05-19 10:08:27 +02:00
list_lru.c	mm: list_lru: set shrinker map bit when child nr_items is not zero	2020-12-11 13:23:31 +01:00
maccess.c	uaccess: Add non-pagefault user-space write function	2020-01-17 19:48:40 +01:00
madvise.c	mm: validate pmd after splitting	2020-10-01 13:18:21 +02:00
memblock.c	memblock: do not start bottom-up allocations with kernel_end	2021-02-10 09:25:28 +01:00
memcontrol.c	mm, memcg: avoid stale protection values when cgroup is above protection	2021-08-26 08:36:22 -04:00
memfd.c	mm: page cache: store only head pages in i_pages	2019-09-24 15:54:08 -07:00
memory-failure.c	mm/memory-failure: make sure wait for page writeback in memory_failure	2021-06-23 14:41:23 +02:00
memory.c	mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page()	2021-06-30 08:47:53 -04:00
memory_hotplug.c	mm: don't rely on system state to detect hot-plug operations	2020-10-07 08:01:30 +02:00
mempolicy.c	mm: mempolicy: fix potential pte_unmap_unlock pte error	2020-11-10 12:37:27 +01:00
mempool.c	docs/core-api/mm: fix return value descriptions in mm/	2019-03-05 21:07:20 -08:00
memremap.c	mm/memory_hotplug: shrink zones when offlining memory	2020-01-09 10:19:56 +01:00
memtest.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
migrate.c	mm, thp: use head page in __migration_entry_wait()	2021-06-30 08:47:52 -04:00
mincore.c	mm: untag user pointers passed to memory syscalls	2019-09-25 17:51:41 -07:00
mlock.c	mm: untag user pointers passed to memory syscalls	2019-09-25 17:51:41 -07:00
mm_init.c	treewide: Add SPDX license identifier for missed files	2019-05-21 10:50:45 +02:00
mmap.c	mm/mmap.c: initialize align_offset explicitly for vm_unmapped_area	2020-10-01 13:17:54 +02:00
mmu_context.c	mm: fix kthread_use_mm() vs TLB invalidate	2020-09-03 11:26:51 +02:00
mmu_gather.c	mm/mmu_gather: invalidate TLB correctly on batch allocation failure and flush	2020-02-11 04:35:42 -08:00
mmu_notifier.c	mm/mmu_notifiers: use the right return code for WARN_ON	2019-11-06 08:47:50 -08:00
mmzone.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mprotect.c	mm, numa: fix bad pmd by atomically check for pmd_trans_huge when marking page tables prot_numa	2020-03-12 13:00:19 +01:00
mremap.c	mm: Fix mremap not considering huge pmd devmap	2020-06-07 13:18:46 +02:00
msync.c	mm: untag user pointers passed to memory syscalls	2019-09-25 17:51:41 -07:00
nommu.c	x86/mm: split vmalloc_sync_all()	2020-03-25 08:25:58 +01:00
oom_kill.c	mm, oom: make the calculation of oom badness more accurate	2021-09-03 10:08:12 +02:00
page-writeback.c	mm/page-writeback.c: avoid potential division by zero in wb_min_max_ratio()	2020-01-23 08:22:41 +01:00
page_alloc.c	mm: don't wake kswapd prematurely when watermark boosting is disabled	2020-12-30 11:51:27 +01:00
page_counter.c	mm/page_counter.c: fix protection usage propagation	2020-08-21 13:05:27 +02:00
page_ext.c	mm, page_owner: fix off-by-one error in __set_page_owner_handle()	2019-10-14 15:04:00 -07:00
page_idle.c	mm/page_idle.c: fix oops because end_pfn is larger than max_pfn	2019-06-29 16:43:45 +08:00
page_io.c	swap: fix swapfile read/write offset	2021-03-07 12:20:49 +01:00
page_isolation.c	mm/memory_hotplug: drain per-cpu pages again during memory offline	2020-09-23 12:40:47 +02:00
page_owner.c	mm/page_owner: change split_page_owner to take a count	2020-10-29 09:57:52 +01:00
page_poison.c	mm/page_poison.c: fix a typo in a comment	2019-09-24 15:54:08 -07:00
page_vma_mapped.c	mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk()	2021-06-30 08:47:55 -04:00
pagewalk.c	mm: pagewalk: fix termination condition in walk_pte_range()	2020-10-01 13:17:30 +02:00
percpu-internal.h	percpu: convert chunk hints to be based on pcpu_block_md	2019-03-13 12:25:31 -07:00
percpu-km.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428	2019-06-05 17:37:16 +02:00
percpu-stats.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428	2019-06-05 17:37:16 +02:00
percpu-vm.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 428	2019-06-05 17:37:16 +02:00
percpu.c	percpu: fix first chunk size calculation for populated bitmap	2020-09-23 12:40:45 +02:00
pgtable-generic.c	mm/thp: fix __split_huge_pmd_locked() on shmem migration entry	2021-06-30 08:47:52 -04:00
process_vm_access.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 152	2019-05-30 11:26:32 -07:00
readahead.c	treewide: Add SPDX license identifier for missed files	2019-05-21 10:50:45 +02:00
rmap.c	mm/thp: fix page_address_in_vma() on file THP tails	2021-06-30 08:47:53 -04:00
rodata_test.c	treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441	2019-06-05 17:37:17 +02:00
shmem.c	mm/hugetlb: fix F_SEAL_FUTURE_WRITE	2021-05-19 10:08:29 +02:00
shuffle.c	mm/shuffle: don't move pages between zones and don't read garbage memmaps	2020-09-03 11:26:51 +02:00
shuffle.h	mm: maintain randomization of page free lists	2019-05-14 19:52:48 -07:00
slab.c	mm, debug_pagealloc: don't rely on static keys too early	2020-01-23 08:22:40 +01:00
slab.h	mm: slab: make page_cgroup_ino() to recognize non-compound slab pages properly	2019-11-06 08:47:50 -08:00
slab_common.c	mm: slab: fix kmem_cache_create failed when sysfs node not destroyed	2021-07-25 14:35:14 +02:00
slob.c	mm, sl[aou]b: guarantee natural alignment for kmalloc(power-of-two)	2019-10-07 15:47:20 -07:00
slub.c	mm/slub.c: include swab.h	2021-06-23 14:41:30 +02:00
sparse-vmemmap.c	mm/sparsemem: convert kmalloc_section_memmap() to populate_section_memmap()	2019-07-18 17:08:07 -07:00
sparse.c	mm/sparse: add the missing sparse_buffer_fini() in error branch	2021-05-14 09:44:32 +02:00
swap.c	mm: introduce MADV_COLD	2019-09-25 17:51:41 -07:00
swap_cgroup.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
swap_slots.c	mm, swap, get_swap_pages: use entry_size instead of cluster in parameter	2018-08-22 10:52:44 -07:00
swap_state.c	mm/swap_state: fix a data race in swapin_nr_pages	2020-10-01 13:18:08 +02:00
swapfile.c	swap: fix swapfile read/write offset	2021-03-07 12:20:49 +01:00
truncate.c	mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page()	2021-06-30 08:47:53 -04:00
usercopy.c	usercopy: Avoid HIGHMEM pfn warning	2019-09-17 15:20:17 -07:00
userfaultfd.c	hugetlbfs: hugetlb_fault_mutex_hash() cleanup	2021-03-30 14:35:19 +02:00
util.c	mm: add kvfree_sensitive() for freeing sensitive data objects	2020-06-17 16:40:23 +02:00
vmacache.c	mm: get rid of vmacache_flush_all() entirely	2018-09-13 15:18:04 -10:00
vmalloc.c	mm/vunmap: add cond_resched() in vunmap_pmd_range	2020-09-03 11:26:52 +02:00
vmpressure.c	mm/vmpressure.c: fix a signedness bug in vmpressure_register_event()	2019-10-07 15:47:19 -07:00
vmscan.c	mm: memcontrol: fix occasional OOMs due to proportional memory.low reclaim	2021-08-26 08:36:22 -04:00
vmstat.c	mm, vmstat: reduce zone->lock holding time by /proc/pagetypeinfo	2019-11-06 08:47:50 -08:00
workingset.c	mm: workingset: fix vmstat counters for shadow nodes	2019-08-13 16:06:52 -07:00
z3fold.c	mm/z3fold: fix potential memory leak in z3fold_destroy_pool()	2021-07-14 16:53:47 +02:00
zbud.c	treewide: Add SPDX license identifier for more missed files	2019-05-21 10:50:45 +02:00
zpool.c	zpool: add malloc_support_movable to zpool_driver	2019-09-24 15:54:12 -07:00
zsmalloc.c	zsmalloc: account the number of compacted pages correctly	2021-03-07 12:20:49 +01:00
zswap.c	zswap: do not map same object twice	2019-09-24 15:54:12 -07:00