API Reference

Core

Read Parquet file(s) with predicate-pushdown filtering.

Wraps pyarrow.dataset to apply row-group-level predicate pushdown, avoiding unnecessary I/O and memory usage.

Parameters:

source (str, Path, or list) – File path, glob pattern (e.g., "data/*.parquet"), or explicit list of paths.
filters (str, list, ExprNode, or None, optional) –
Filter specification. Accepts several formats:

Expression string – parsed via the built-in mini-language:
```
"vmag < 20"
"(a < 30 & b > 50) | c == 1"
"desig in 1,2,3"
"~(a > 5)"
"v is null"
```
List of 3-tuples (flat AND):
```
[("a", ">", 5), ("b", "<", 10)]
```
List of lists (DNF – OR of AND-groups):
```
[[("a", ">", 5)], [("b", "<", 10)]]
```
Pre-parsed AST node (FilterExpr, AndExpr, OrExpr, NotExpr).
columns (list of str, optional) – Columns to load (projection pushdown). None loads all columns.
output (str or Path, optional) – Save the result to this path (.parquet or .csv).
overwrite (bool, optional) – Allow overwriting output if it already exists.

Returns:

Filtered (and optionally column-selected) DataFrame.

Return type:

pandas.DataFrame

Raises:

FileNotFoundError – No files matched source.
FileExistsError – output exists and overwrite is False.
ValueError – Invalid filter syntax.
TypeError – filters is not a supported type.

Examples

Simple filter:

df = pqfilt.read("data.parquet", filters="vmag < 20")

AND + OR expression:

df = pqfilt.read("data.parquet", filters="(a < 30 & b > 50) | c == 1")

Negation:

df = pqfilt.read("data.parquet", filters="~(a > 5)")

Null check:

df = pqfilt.read("data.parquet", filters="v is null")

Tuple syntax:

df = pqfilt.read("data.parquet", filters=[("a", ">", 5), ("b", "<", 10)])

Create an Arrow scanner with the pqfilt filter and projection policy.

Parameters:

source (str, Path, or list) – File path, glob pattern, or explicit list of paths.
filters (str, list, ExprNode, or None) – Filter specification accepted by to_ast().
columns (list of str, optional) – Columns to project from the input dataset.

Returns:

Lazily evaluated scanner configured with the requested filter and projection.

Return type:

pyarrow.dataset.Scanner

Filter Parquet file(s) and write the result batch by batch.

Unlike read(), this function does not materialize the complete filtered result in memory. It writes Parquet by default and writes CSV when output has a .csv suffix.

Parameters:

source (str, Path, or list) – File path, glob pattern, or explicit list of paths.
output (str or Path) – Destination path for the filtered result.
filters (str, list, ExprNode, or None, optional) – Filter specification accepted by to_ast().
columns (list of str, optional) – Columns to write. None writes all columns.
overwrite (bool, optional) – Whether an existing output file may be replaced.

Returns:

Number of rows written.

Return type:

int

Raises:

FileExistsError – If output exists and overwrite is False.
ValueError – If output is one of the input files.

Filter an already-loaded pandas DataFrame using the pqfilt expression syntax.

Applies the same filter language as read() — expression strings, tuple lists, DNF, and pre-parsed AST nodes — directly to a DataFrame.

Parameters:

df (pandas.DataFrame) – Input data.
filters (str, list, or ExprNode) – Filter specification (same formats accepted by read()).

Returns:

Filtered rows with reset index.

Return type:

pandas.DataFrame

Notes

Comparisons on missing values use PyArrow’s three-valued semantics: an unknown comparison result is excluded from the returned rows. Membership filters follow PyArrow’s isin behavior, where a missing value matches a membership list containing a missing value.

Raises:

KeyError – A column named in the filter does not exist in df.
TypeError – filters is not a supported type.
ValueError – Invalid filter syntax.

Examples

import pandas as pd
import pqfilt

df = pd.DataFrame({"a": range(10), "b": range(0, 100, 10)})

pqfilt.filter_df(df, "a > 5")
pqfilt.filter_df(df, "a > 3 & b < 80")
pqfilt.filter_df(df, "~(a in 1,2,3)")
pqfilt.filter_df(df, [("a", ">", 5), ("b", "<", 90)])

Convert a supported filter specification to an AST.

Parameters:

filters (str, list, or ExprNode) – Expression string, flat list of filter tuples, DNF list of filter tuples, or a pre-parsed AST node.

Returns:

Parsed or supplied AST node.

Return type:

ExprNode

Raises:

TypeError – If filters is not a supported filter specification.
ValueError – If the supplied string or tuple filters are invalid.

Operators

pqfilt.SUPPORTED_OPERATORS = ('>', '>=', '<', '<=', '==', '!=', 'in', 'not in', 'is null', 'is not null')

Built-in immutable sequence.

If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable’s items.

If the argument is a tuple, the return value is the same object.

pqfilt.validate_operator(op: str, col: str | None = None) → None[source]

Validate that op is a supported filter operator.

Parameters:

op (str) – Operator string.
col (str, optional) – Column name for error-message context.

Raises:

ValueError – If op is not in SUPPORTED_OPERATORS.

Expression Parser

pqfilt.parse_expression(expr: str) → FilterExpr | AndExpr | OrExpr | NotExpr[source]

Parse a filter expression string into an AST.

Parameters:: expr (str) – Expression such as "a > 5 & b < 10 | c == 1".
Returns:: A FilterExpr, AndExpr, or OrExpr tree.
Return type:: ExprNode
Raises:: ValueError – On parse errors (unmatched parentheses, missing operator, etc.).

Examples

>>> parse_expression("vmag < 20")
FilterExpr(col='vmag', op='<', val=20)

>>> parse_expression("a > 5 & b < 10")
AndExpr(children=(FilterExpr(col='a', op='>', val=5), FilterExpr(col='b', op='<', val=10)))

Return a copy of an AST with each leaf transformed by fn.

Parameters:

node (ExprNode) – AST node to transform.
fn (callable) – Function that accepts a FilterExpr and returns an AST node. It may replace a leaf with a compound expression.

Returns:

Transformed AST. Compound nodes are reconstructed recursively.

Return type:

ExprNode

Raises:

TypeError – If fn does not return an AST node.

pqfilt.to_pyarrow_expr(node: FilterExpr | AndExpr | OrExpr | NotExpr) → Expression[source]

Convert a parsed AST into a pyarrow.compute.Expression.

Parameters:: node (ExprNode) – Output of parse_expression().
Returns:: Expression suitable for pyarrow.dataset.Dataset.to_table(filter=...).
Return type:: pyarrow.compute.Expression

AST Nodes

class pqfilt.FilterExpr(col: str, op: str, val: Any)[source]

Single comparison: col op val.

col

Column name.

Type:: str

op

Comparison operator (one of SUPPORTED_OPERATORS).

Type:: str

val

Comparison value (scalar or list for in / not in).

Type:: Any

class pqfilt.AndExpr(children: tuple[~pqfilt._parser.FilterExpr | ~pqfilt._parser.AndExpr | ~pqfilt._parser.OrExpr | ~pqfilt._parser.NotExpr, ...]=<factory>)[source]

Conjunction (AND) of child nodes.

children

All children must evaluate to True.

Type:: tuple of ExprNode

class pqfilt.OrExpr(children: tuple[~pqfilt._parser.FilterExpr | ~pqfilt._parser.AndExpr | ~pqfilt._parser.OrExpr | ~pqfilt._parser.NotExpr, ...]=<factory>)[source]

Disjunction (OR) of child nodes.

children

At least one child must evaluate to True.

Type:: tuple of ExprNode

class pqfilt.NotExpr(child: FilterExpr | AndExpr | OrExpr | NotExpr)[source]

Logical negation of a child node.

child

The expression to negate.

Type:: ExprNode