API Reference

Core

Read Parquet file(s) with predicate-pushdown filtering.

Wraps pyarrow.dataset to apply row-group-level predicate pushdown, avoiding unnecessary I/O and memory usage.

Parameters:

source (str, Path, or list) – File path, glob pattern (e.g., "data/*.parquet"), or list of paths and patterns.
filters (str, list, ExprNode, or None, optional) –
Filter specification. Accepts several formats:

Expression string – parsed via the built-in mini-language:
```
"vmag < 20"
"(a < 30 & b > 50) | c == 1"
"desig in 1,2,3"
"~(a > 5)"
"v is null"
```
List of 3-tuples (flat AND):
```
[("a", ">", 5), ("b", "<", 10)]
```
List of lists (DNF – OR of AND-groups):
```
[[("a", ">", 5)], [("b", "<", 10)]]
```
Pre-parsed AST node (FilterExpr, AndExpr, OrExpr, NotExpr).
columns (list of str, optional) – Columns to load. None includes every column found across the files.
output (str or Path, optional) – Save the result to this path (.parquet or .csv).
overwrite (bool, optional) – Allow overwriting output if it already exists.

Returns:

Filtered (and optionally column-selected) DataFrame.

Return type:

pandas.DataFrame

Notes

If combining files adds nulls to a 64-bit integer column, the result uses pandas Int64 or UInt64 to preserve large integers exactly. See Different Columns or Types for the common-type rules and limits.

Raises:

FileNotFoundError – No files matched source.
FileExistsError – output exists and overwrite is False.
ValueError – Invalid filter syntax.
TypeError – filters is not a supported type.

Examples

Simple filter:

df = pqfilt.read("data.parquet", filters="vmag < 20")

AND + OR expression:

df = pqfilt.read("data.parquet", filters="(a < 30 & b > 50) | c == 1")

Negation:

df = pqfilt.read("data.parquet", filters="~(a > 5)")

Null check:

df = pqfilt.read("data.parquet", filters="v is null")

Tuple syntax:

df = pqfilt.read("data.parquet", filters=[("a", ">", 5), ("b", "<", 10)])

Create a scanner that filters rows and selects output columns.

Parameters:

source (str, Path, or list) – File path, glob pattern, or list of paths and patterns.
filters (str, list, ExprNode, or None) – Filter specification accepted by to_ast().
columns (list of str, optional) – Columns to return. None includes every column found across the files.

Returns:

Scanner ready to read the filtered result as a table or record batches.

Return type:

pyarrow.dataset.Scanner

Notes

Files may have different columns or compatible column types. Missing fields become nulls. Integer and floating-point types follow Arrow’s numeric promotion rules; unsafe integer casts raise during scanning.

File metadata is read before scanning. Rows are still filtered and streamed in batches. See Different Columns or Types for examples and type limits.

Filter Parquet file(s) and write the result batch by batch.

Unlike read(), this function does not materialize the complete filtered result in memory. It writes Parquet by default and writes CSV when output has a .csv suffix. Output is staged beside the destination and published only after writing succeeds, preserving it on failure.

Parameters:

source (str, Path, or list) – File path, glob pattern, or list of paths and patterns.
output (str or Path) – Destination path for the filtered result.
filters (str, list, ExprNode, or None, optional) – Filter specification accepted by to_ast().
columns (list of str, optional) – Columns to write. None writes all columns.
overwrite (bool, optional) – Whether an existing output file may be replaced.

Returns:

Number of rows written.

Return type:

int

Raises:

FileExistsError – If output exists and overwrite is False.
ValueError – If output is one of the input files.

Filter an already-loaded pandas DataFrame using the pqfilt expression syntax.

Applies the same filter language as read() — expression strings, tuple lists, DNF, and pre-parsed AST nodes — directly to a DataFrame.

Parameters:

df (pandas.DataFrame) – Input data.
filters (str, list, or ExprNode) – Filter specification (same formats accepted by read()).

Returns:

Filtered rows with reset index.

Return type:

pandas.DataFrame

Notes

Comparisons on missing values use PyArrow’s three-valued semantics: an unknown comparison result is excluded from the returned rows. Membership filters follow PyArrow’s isin behavior, where a missing value matches a membership list containing a missing value.

Raises:

KeyError – A column named in the filter does not exist in df.
TypeError – filters is not a supported type.
ValueError – Invalid filter syntax.

Examples

import pandas as pd
import pqfilt

df = pd.DataFrame({"a": range(10), "b": range(0, 100, 10)})

pqfilt.filter_df(df, "a > 5")
pqfilt.filter_df(df, "a > 3 & b < 80")
pqfilt.filter_df(df, "~(a in 1,2,3)")
pqfilt.filter_df(df, [("a", ">", 5), ("b", "<", 90)])

Convert a supported filter specification to an AST.

Parameters:

filters (str, list, or ExprNode) – Expression string, flat list of filter tuples, DNF list of filter tuples, or a pre-parsed AST node.

Returns:

Parsed or supplied AST node.

Return type:

ExprNode

Raises:

TypeError – If filters is not a supported filter specification.
ValueError – If the supplied string or tuple filters are invalid.

Operators

pqfilt.SUPPORTED_OPERATORS = ('>', '>=', '<', '<=', '==', '!=', 'in', 'not in', 'is null', 'is not null')

Built-in immutable sequence.

If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable’s items.

If the argument is a tuple, the return value is the same object.

pqfilt.validate_operator(op: str, col: str | None = None) → None[source]

Validate that op is a supported filter operator.

Parameters:

op (str) – Operator string.
col (str, optional) – Column name for error-message context.

Raises:

ValueError – If op is not in SUPPORTED_OPERATORS.

Expression Parser

pqfilt.parse_expression(expr: str) → FilterExpr | AndExpr | OrExpr | NotExpr[source]

Parse a filter expression string into an AST.

Parameters:: expr (str) – Expression such as "a > 5 & b < 10 | c == 1".
Returns:: A FilterExpr, AndExpr, or OrExpr tree.
Return type:: ExprNode
Raises:: ValueError – On parse errors (unmatched parentheses, missing operator, etc.).

Examples

>>> parse_expression("vmag < 20")
FilterExpr(col='vmag', op='<', val=20)

>>> parse_expression("a > 5 & b < 10")
AndExpr(children=(FilterExpr(col='a', op='>', val=5), FilterExpr(col='b', op='<', val=10)))

Return a copy of an AST with each leaf transformed by fn.

Parameters:

node (ExprNode) – AST node to transform.
fn (callable) – Function that accepts a FilterExpr and returns an AST node. It may replace a leaf with a compound expression.

Returns:

Transformed AST. Compound nodes are reconstructed recursively.

Return type:

ExprNode

Raises:

TypeError – If fn does not return an AST node.

pqfilt.to_pyarrow_expr(node: FilterExpr | AndExpr | OrExpr | NotExpr) → Expression[source]

Convert a parsed AST into a pyarrow.compute.Expression.

Parameters:: node (ExprNode) – Output of parse_expression().
Returns:: Expression suitable for pyarrow.dataset.Dataset.to_table(filter=...).
Return type:: pyarrow.compute.Expression

AST Nodes

class pqfilt.FilterExpr(col: str, op: str, val: Any)[source]

Single comparison: col op val.

col

Column name.

Type:: str

op

Comparison operator (one of SUPPORTED_OPERATORS).

Type:: str

val

Comparison value (scalar or list for in / not in).

Type:: Any

class pqfilt.AndExpr(children: tuple[~pqfilt._parser.FilterExpr | ~pqfilt._parser.AndExpr | ~pqfilt._parser.OrExpr | ~pqfilt._parser.NotExpr, ...]=<factory>)[source]

Conjunction (AND) of child nodes.

children

All children must evaluate to True.

Type:: tuple of ExprNode

class pqfilt.OrExpr(children: tuple[~pqfilt._parser.FilterExpr | ~pqfilt._parser.AndExpr | ~pqfilt._parser.OrExpr | ~pqfilt._parser.NotExpr, ...]=<factory>)[source]

Disjunction (OR) of child nodes.

children

At least one child must evaluate to True.

Type:: tuple of ExprNode

class pqfilt.NotExpr(child: FilterExpr | AndExpr | OrExpr | NotExpr)[source]

Logical negation of a child node.

child

The expression to negate.

Type:: ExprNode