Environment.memory_consumption module

Tiny module to measure and work on memory consumption.

It defines a utility function to get the memory consumes in the current process or the current thread (getCurrentMemory()), and a function to pretty print memory size (sizeof_fmt()).

It also imports tracemalloc and define a convenient function that pretty print the most costly lines after a run.

>>> return_code = start_tracemalloc()
Starting to trace memory allocations...
>>> # ... run your application ...
>>> display_top_tracemalloc()
<BLANKLINE>
Top 10 lines ranked by memory consumption:
#1: python3.6/doctest.py:1330: 636 B
    compileflags, 1), test.globs)
#2: <doctest __main__[1]>:1: 568 B
    display_top_tracemalloc()
#3: python3.6/doctest.py:1346: 472 B
    if check(example.want, got, self.optionflags):
#4: python3.6/doctest.py:1374: 464 B
    self.report_success(out, test, example, got)
#5: python3.6/doctest.py:1591: 456 B
    got = self._toAscii(got)
#6: ./memory_consumption.py:168: 448 B
    snapshot = tracemalloc.take_snapshot()
#7: python3.6/doctest.py:1340: 440 B
    self._fakeout.truncate(0)
#8: python3.6/doctest.py:1339: 440 B
    got = self._fakeout.getvalue()  # the actual output
#9: python3.6/doctest.py:1331: 432 B
    self.debugger.set_continue() # ==== Example Finished ====
#10: python3.6/doctest.py:251: 89 B
    result = StringIO.getvalue(self)
2 others: 78 B
<BLANKLINE>
Total allocated size: 4.4 KiB
4523

Warning

This is automatically used (for main.py at least) when DEBUGMEMORY=True (cli env).

Warning

This is experimental and does not work as well on Mac OS X and Windows as it works on GNU/Linux systems.

Environment.memory_consumption.getCurrentMemory(thread=False, both=False)[source]

Get the current memory consumption of the process, or the thread.

  • Example, before and after creating a huge random matrix in Numpy, and asking to invert it:
>>> currentMemory = getCurrentMemory()
>>> print("Consumed {} memory".format(sizeof_fmt(currentMemory)))  # doctest: +SKIP
Consumed 16.8 KiB memory

>>> import numpy as np; x = np.random.randn(1000, 1000)  # doctest: +SKIP
>>> diffMemory = getCurrentMemory() - currentMemory; currentMemory += diffMemory
>>> print("Consumed {} more memory".format(sizeof_fmt(diffMemory)))  # doctest: +SKIP
Consumed 18.8 KiB more memory

>>> y = np.linalg.pinv(x)  # doctest: +SKIP
>>> diffMemory = getCurrentMemory() - currentMemory; currentMemory += diffMemory
>>> print("Consumed {} more memory".format(sizeof_fmt(diffMemory)))  # doctest: +SKIP
Consumed 63.9 KiB more memory

Warning

This is still experimental for multi-threaded code.

Warning

It can break on some systems, see for instance [the issue #142](https://github.com/SMPyBandits/SMPyBandits/issues/142).

Warning

FIXME even on my own system, it works for the last few policies I test, but fails for the first??

Warning

This returns 0 on Microsoft Windows, because the resource module is not available on non-UNIX systems (see https://docs.python.org/3/library/unix.html).

Environment.memory_consumption.sizeof_fmt(num, suffix='B', longsuffix=True, usespace=True, base=1024)[source]

Returns a string representation of the size num.

  • Examples:
>>> sizeof_fmt(1020)
'1020 B'
>>> sizeof_fmt(1024)
'1 KiB'
>>> sizeof_fmt(12011993)
'11.5 MiB'
>>> sizeof_fmt(123456789)
'117.7 MiB'
>>> sizeof_fmt(123456789911)
'115 GiB'

Options include:

  • No space before unit:
>>> sizeof_fmt(123456789911, usespace=False)
'115GiB'
  • French style, with short suffix, the “O” suffix for “octets”, and a base 1000:
>>> sizeof_fmt(123456789911, longsuffix=False, suffix='O', base=1000)
'123.5 GO'
Environment.memory_consumption.start_tracemalloc()[source]

Wrapper function around tracemalloc.start(), to log the start of tracing memory allocation.

Environment.memory_consumption.display_top_tracemalloc(snapshot=None, key_type='lineno', limit=10)[source]

Display detailed information on the limit most costly lines in this memory snapshot.