How to free some space in OpenShift

If your builds run into an error looking like:

error: build error: devmapper: Thin Pool has 11561 free data blocks which is
less than minimum required 11944 free data blocks. Create more free space in
thin pool or use dm.min_free_space option to change behavior

It is likely that one or several of the nodes has some disk space issues.

Check the disk space on the nodes

You can do this via ansible by simply running:

ansible -a 'lvs' os_nodes_stg

The output will look somthing like this:

  LV          VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root        GuestVolGroup00 -wi-ao---- <58.59g
  docker-pool vg-docker       twi-a-t---  58.32g             42.00  18.44

os-node02.stg.iad2.fedoraproject.org | CHANGED | rc=0 >>
  LV          VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root        GuestVolGroup00 -wi-ao---- <58.59g
  docker-pool vg-docker       twi-a-t--- <48.60g             32.37  14.81

os-node01.stg.iad2.fedoraproject.org | CHANGED | rc=0 >>
  LV          VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root        GuestVolGroup00 -wi-ao---- <58.59g
  docker-pool vg-docker       twi-a-t---  58.32g             40.75  17.38

os-node04.stg.iad2.fedoraproject.org | CHANGED | rc=0 >>
  LV          VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root        GuestVolGroup00 -wi-ao---- <58.59g
  docker-pool vg-docker       twi-a-t---  58.32g             90.32  28.35

In this case you can see that os-node04.stg has a disk full at 90%.

Free space on a node

There is a cron job running regularly to clean up docker images that aren’t needed anymore, but you can also run it manually to free some space.

The command to run is then:

docker rmi $(docker images --filter dangling=true -q)

This needs to be run on the os-node that is running out of space.