hats.io.size_estimates#
General utilities for estimating size of input and output.
Functions#
|
Estimate the disk usage of a directory, and recursive contents. |
|
Estimate the memory footprint of each row when the data is loaded. |
Module Contents#
- estimate_dir_size(path: str | pathlib.Path | upath.UPath | None = None, *, divisor=1)[source]#
Estimate the disk usage of a directory, and recursive contents.
When divisor == 1, returns size in bytes.
- get_mem_size_per_row(data, cols=None)[source]#
Estimate the memory footprint of each row when the data is loaded.
Sizes model the columnar arrow representation the data occupies once loaded into memory (i.e.
Table.nbytes): each row costs its slice of the loaded column buffers.fixed-width values (ints, floats, datetimes, …): the value’s byte width, with bit-packed types (booleans) counted at their true bit width (1/8 byte per value);
strings and binary: the value’s data bytes, plus one offset entry;
lists: element count x element byte width (variable-width elements are attributed their column-wide average), plus one offset entry;
structs (e.g. nested columns): the sum of their fields’ contributions;
validity bitmaps: one bit per row/element, counted when the validity buffer is present (arrow allocates it when nulls exist, but readers may also materialize an all-valid bitmap for nullable columns).
Column types outside this model have their total buffer size attributed uniformly across rows. Per-object Python overheads are deliberately excluded: they describe the cost of materializing rows as Python objects, not of loading the data. Summing a group of rows’ sizes therefore estimates the memory needed to load that group as a partition.
- Args:
data (pd.DataFrame or pa.Table): the data chunk to measure cols (list[str] or None): if provided, only these columns are included in the
per-row size calculation
- Returns:
np.ndarray: float64 array of memory sizes for each row in the chunk. Values may be fractional (bit-packed booleans, bitmap bits); sum before rounding.