hats.io.size_estimates#

General utilities for estimating size of input and output.

Functions#

estimate_dir_size([path, divisor])

Estimate the disk usage of a directory, and recursive contents.

get_mem_size_per_row(data[, cols])

Estimate the memory footprint of each row when the data is loaded.

Module Contents#

estimate_dir_size(path: str | pathlib.Path | upath.UPath | None = None, *, divisor=1)[source]#

Estimate the disk usage of a directory, and recursive contents.

When divisor == 1, returns size in bytes.

get_mem_size_per_row(data, cols=None)[source]#

Estimate the memory footprint of each row when the data is loaded.

Sizes model the columnar arrow representation the data occupies once loaded into memory (i.e. Table.nbytes): each row costs its slice of the loaded column buffers.

  • fixed-width values (ints, floats, datetimes, …): the value’s byte width, with bit-packed types (booleans) counted at their true bit width (1/8 byte per value);

  • strings and binary: the value’s data bytes, plus one offset entry;

  • lists: element count x element byte width (variable-width elements are attributed their column-wide average), plus one offset entry;

  • structs (e.g. nested columns): the sum of their fields’ contributions;

  • validity bitmaps: one bit per row/element, counted when the validity buffer is present (arrow allocates it when nulls exist, but readers may also materialize an all-valid bitmap for nullable columns).

Column types outside this model have their total buffer size attributed uniformly across rows. Per-object Python overheads are deliberately excluded: they describe the cost of materializing rows as Python objects, not of loading the data. Summing a group of rows’ sizes therefore estimates the memory needed to load that group as a partition.

Args:

data (pd.DataFrame or pa.Table): the data chunk to measure cols (list[str] or None): if provided, only these columns are included in the

per-row size calculation

Returns:

np.ndarray: float64 array of memory sizes for each row in the chunk. Values may be fractional (bit-packed booleans, bitmap bits); sum before rounding.