Ballooning permanently "steals" memory from VMs
Hi everyone,
I'm trying to use Ballooning (Dynamic Memory Allocation) in Ubuntu 22.04 VMs on Proxmox 7.3-3. I've configured a few VMs to use their Memory with Ballooning and set values like 512M/2G for min/max. When I startup the VM, it initially gets the full max memory as configured in the hardware settings, but then it returns RAM to the host and decreases its overall RAM all the way down to min. So far so good, this just means that Ballooning works, right? But when my applications want to allocate more memory, they fail to do so, resulting in OOM kills and memory allocation failures inside the VMs.
To illustrate, I've rebooted the VM and immediately called free -h multiple times to show the decrease of the system's RAM. The two commands were run within less than a minute from each other.
# free -h
total used free shared buff/cache available
Mem: 644Mi 320Mi 67Mi 62Mi 256Mi 237Mi
# free -h
total used free shared buff/cache available
Mem: 393Mi 274Mi 6.0Mi 85Mi 111Mi 10MiI've tried to run sysbench in order to force the VM to allocate full memory on the VM (or at least more than the configured minimum), and it failed:
# sysbench memory --memory-block-size=1G --memory-total-size=1G --memory-oper=write --threads=1 run sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3) FATAL: Failed to allocate buffer!
My expectation of the Ballooning functionality was that unused memory is temporarily returned to the host for redistribution, but that the VM could at any time request more RAM until the max configured RAM limit is reached. However, that doesn't seem to be the case.
So I guess my question is: wtf is happening here, and what can I do to resolve this? Any help is very much appreciated.
Comments Section
Ballooning, as I understand it, is just a way of telling proxmox "if memory is tight, give this vm AT LEAST x amount of ram at any given point, up to x if we have free resources".
You can fine tune this behavior by setting weights for your VMs.
If your VM always needs 8GB ram available but shouldn't allocate all of it immediately, then just set both values to 8192. That way the VM still only allocates the ram it needs right now, but also has access to all of it immediately. That's at least how I do it. You can remove the tick at ballooning too, if I'm not completely mistaken. I didn't tho.
Thank you very much for taking the time.
Thing is, my VM does indeed get the minimum amount of RAM at any given time, but it never has a chance of getting more than that, even though I have configured different values for min and max and the host has only allocated around 7 of 16 GB active RAM to running VMs. Once the ballooning driver has removed the RAM from the VM, it just doesn't get it back, even though host-RAM is available. I'm really puzzled, especially because that's not the behaviour I know from ESXi.
I previously had my VMs configured for identical values of min and max with ballooning activated as you suggested, but experienced OOM kills of entire VMs by Proxmox due to memory shortage on the host, so I assumed that VMs do indeed allocate the whole configured RAM.
Ballooning works: (1) by pretending that the VM has 2 GB of emulated physical RAM, (2) the ballooning client allocating 1.5 GB of that RAM, giving it back to the hypervisor, (3) when the ballooning client sees memory pressure, it requests the hypervisor to again allocate some RAM to the VM, then freeing that part of RAM within the client. If you think of short selling, you're close enough.
Your observation on the initial phase where the VM is actually allocated the whole 2 GB is correct. This is the time before the ballooning client within the VM is initialized.
The question now is: what triggers (3). Turns out, it is a simple polling every few seconds—IIRC, in Proxmox this is 2 seconds by default. Essentially, every time non-ballooned memory within the VM is close to being full, the ballooning client requests a bit more memory to be released. This works great in workloads which slowly consume RAM, but not when you attempt to allocate lots of RAM in a single instant. Yet it is the latter that your sysbench does.
Alright, I get that. However, sysbench (or u/clickbg's linked C program) are only proxies for rela life workloads that do the same. I usually use these Ubuntu VMs as Docker container hosts and it's not uncommon to spin up another container or request larger chunks of RAM at once in real life scenarios. Shouldn't the ballooning driver be able to handle these kinds of situations? What can I do?
Also, I assumed that ballooning wouldn't have an impact on the amount of total memory in the VM (as shown by
freebelow), because the ballooning driver would just allocate all available RAM pages in the guest OS and report these back to the hypervisor as available; at least that's what I see in ESXi, where the majority of my experience is. From what I see in Proxmox, the ballooning driver seems to actually decrease the amount of total memory available to the VM, as in "unplugging physical RAM banks from the board", and that's what's puzzling me. Hardware that isn't there cannot be used by the VM.# free -h total used free shared buff/cache available Mem: 455Mi 117Mi 282Mi 0.0Ki 54Mi 314MiEDIT: formatting
Comment deleted by user
Thank you, I have no PCIe passthrough configured, mainly because I lack decent hardware. :-D Also, luckily, my VMs are for personal/internal use only, so I don't have any specific high performance requirements apart from "should work", so PCIe passthrough wasn't a necessity.
Still interesting to know though. Does that mean that no matter what kind of device I passthrough (NIC, disk controller, ...), I will then have to forgo ballooning on that VM?
Ballooning device does work as other have described it here, but it needs available memory to give the vm when asked for it. If there is no memory, there's nothing to give.
The various mechanisms on a proxmox system that cause memory pressure (zfs arc, oom on lxc containers, etc) and their bahaviours around memory pressure can get in the way of each other.
If you are using zfs, you could try temporarily lowering the arc max to keep it a safer distance from the ballooning threshold.
This is obviously based on a few assumptions that may be incorrect about your memory situation.
I use ZFS on the host, but not on the guest OSses. Is that what you meant?
Just ran into this issue on my Debian 12 VM. Had it happily configured at 2GB/16GB ballooning for a year, no issues until a few days ago. Suddenly started seeing a lot of OOM Killer events and the machine was completely frozen. In PVE, the machine was humming at 2GB/16GB memory utilization so I didn't first realize the issue. After some troubleshooting, I noticed Debian 12 'free -m' command output was saying there is no memory available, and system total memory is only 2GB. So whatever ballooning was doing, it was not giving the memory back to the VM. The host has no overcommitted memory, so I don't know why the memory is not released back to the VM's when they need it.
I disabled ballooning for this VM and now it has the full 16GB at its disposal when it needs it.