From b17c48ab512bdf04de0a08af19fcf0724f84b700 Mon Sep 17 00:00:00 2001 From: Tiziano Zito Date: Tue, 13 Aug 2024 13:44:19 +0200 Subject: [PATCH] add note about memmap --- numpy/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 numpy/README.md diff --git a/numpy/README.md b/numpy/README.md new file mode 100644 index 0000000..32d95de --- /dev/null +++ b/numpy/README.md @@ -0,0 +1,26 @@ +# Anatomy of a numpy array +## one dimension, float64 +![1d array](ndarray-memory-layout-1d.svg) + +## two dimensions, square, float64 +![2d array - square](ndarray-memory-layout-2d-square.svg) + +## two dimensions, rectangular, int32 +![2d array - rectangular](ndarray-memory-layout-2d-rectangular.svg) + +## what about Python lists? +![memory layout of a Python list](python-list-memory-layout.svg) + +## interesting attributes of numpy arrays + - `x.data`, `x.data.hex()`, `x.data.format`, `x.tobytes()` + - `x.flags`: + - `OWNDATA` + - `C_CONTIGUOUS` + - `F_CONTIGUOUS` + - more [flags](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.flags.html) + +## If your arrays are bigger than RAM + - `numpy.memmap`: an array-like object that maps memory to an array stored on disk, used for accessing small segments of large files on disk, without reading the entire file into memory. Use with caution! + + - `HDF5`: hierarchical data format, with aribitrary metadata and multilanguage support with `h5py` with an array-lie interface + - other projects, for example `xarray` and `zarr`