Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a more clever formula for available_mem #34

Open
wants to merge 1 commit into
base: julia-uv2-1.44.2
Choose a base branch
from

Conversation

gbaraldi
Copy link
Member

@gbaraldi gbaraldi commented Feb 1, 2023

We might want to use a different combination of values from vm_info64, I chose this because that's what Stats uses but I'm not too inclined to it or against it.
@staticfloat

Fixes JuliaLang/julia#48473


if (host_statistics64(mach_host_self(), HOST_VM_INFO64,
(host_info64_t)&info, &count) != KERN_SUCCESS) {
return UV_EINVAL; /* FIXME(bnoordhuis) Translate error. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is bnoordhuis?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ben Noordhuis, the libuv maintainer

// Based on https://github.com/exelban/stats/blob/6c991de101957065f579a550ac2ae358d733d0c0/Modules/RAM/readers.swift#L47-L57
uint64_t used = info.active_count + info.inactive_count + info.compressor_page_count +info.speculative_count + info.wire_count;
uint64_t not_used = info.purgeable_count + info.external_page_count;
uint64_t available = uv_get_total_memory() - (used - not_used) * sysconf(_SC_PAGESIZE);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think speculative_count should be in used; since the kernel counts it as part of free: https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/osfmk/kern/host.c#L815

I think there's an easier way to figure out "available" memory:

uint64_t available = (info.free_count + info.inactive_count) * sysconf(_SC_PAGESIZE);

You could arguably include purgeable_count into that. This completely ignores the external page count (which I believe represents pages which have been swapped to disk), and I think that.... is probably the right thing to do? If macOS pages things out to make room for us, that's fine, and we can continue to use more physical memory.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if external is swapped pages, or pages that are on disk by default, but the os is caching on RAM.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But your code seems to give very similar results while looking cleaner.

@vustef
Copy link

vustef commented Dec 20, 2023

In case anyone needs to calculate this in Julia, here's a hacked up snippet:

mutable struct VmStatistics64
	free_count::UInt32
	active_count::UInt32
	inactive_count::UInt32
	wire_count::UInt32
	zero_fill_count::UInt64
	reactivations::UInt64
	pageins::UInt64
	pageouts::UInt64
	faults::UInt64
	cow_faults::UInt64
	lookups::UInt64
	hits::UInt64
	purges::UInt64
	purgeable_count::UInt32
	
	speculative_count::UInt32

	decompressions::UInt64
	compressions::UInt64
	swapins::UInt64
	swapouts::UInt64
	compressor_page_count::UInt32
	throttled_count::UInt32
	external_page_count::UInt32
	internal_page_count::UInt32
	total_uncompressed_pages_in_compressor::UInt64

	VmStatistics64() = new(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
end

vms = VmStatistics64()

mach_host_self = @ccall mach_host_self()::UInt32


count = UInt32(sizeof(vms) / sizeof(Int32))
ref_count = Ref(count)

@ccall host_statistics64(mach_host_self::UInt32, 4::Int64, pointer_from_objref(vms)::Ptr{Int64}, ref_count::Ref{UInt32})::Int64

# @show vms
# # values should correspond to:
# @info read(`vm_stat`, String)

page_size = Int(@ccall sysconf(29::UInt32)::UInt32)

used = vms.active_count + vms.inactive_count + vms.compressor_page_count +vms.speculative_count + vms.wire_count
not_used = vms.purgeable_count + vms.external_page_count
available = Sys.total_memory() - (used - not_used) * page_size
@show available / 2^20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pathological GC time on M1 mac
4 participants