CSV vs Parquet: Why Data Teams Switched
The csv vs parquet question comes up in almost every data team’s first growth spurt: CSVs are everywhere, they open in Excel, and they feel safe — until a query scanning ten million rows starts taking minutes instead of seconds. This guide covers what is different, why the switch matters for analytical workloads, and when CSV is still the right call.
CSV vs Parquet in one sentence
CSV is a row-oriented plain-text file; Parquet is a columnar binary file with a built-in schema. The two formats were designed for different goals: CSV for maximum portability and human readability, Parquet for efficient machine querying at scale.
How CSV stores data (and where it hurts)
Open any CSV and you’ll see every row written out in order: order_id,customer_id,amount,country,timestamp. To sum the amount column, your system reads the entire file — all five columns, every row. There is no way to skip the columns you don’t need.
Three problems compound that read cost:
- No data types. Everything is a string at rest. Your pipeline must infer or cast
"99.50"into a decimal and"2026-07-06"into a date. Mismatches slip through silently. - No compression. Plain text is verbose. The string
"United Kingdom"takes 14 bytes on every row it appears, with no savings for repetition. - Limited parallel reads. Row boundaries are just newlines, and quoted fields can contain embedded newlines, making clean splits for distributed engines awkward.
None of this makes CSV a bad format. It makes it a format optimized for exchange, not for query.
How Parquet stores data
Parquet organizes a file into row groups — horizontal slices of typically 128 MB. Within each row group, every column’s values are stored contiguously, so a query needing 3 columns from a 40-column table reads only those 3 and skips the rest.
Several features build on that columnar foundation:
- Per-column compression and encoding. Values in a column share the same type and often repeat, so compression ratios are high; dictionary encoding turns repeated strings like
"US"into compact integer codes. - Embedded schema and types. The file footer records that
amountisDOUBLEandorder_idisINT64. No guesswork, no silent cast errors. - Column statistics & predicate pushdown. Each row group stores the min, max, and null count per column, so the engine can skip groups that cannot contain a matching value without reading a single row.
- Splittable row groups. Row groups are self-contained, so distributed engines like Spark or Trino assign each to a different worker for parallel reads.
CSV vs Parquet side by side
Here is how the two formats compare across the dimensions that matter most:
| Dimension | CSV | Parquet |
|---|---|---|
| Human-readable | Yes — open in any text editor | No — binary; appears as unreadable bytes in a text editor |
| File size | Larger (uncompressed plain text) | Smaller (per-column compression; often several times smaller) |
| Query speed (analytical) | Slow (full-file scan, all columns) | Fast (column pruning & row-group skipping) |
| Data typing | None — everything is a string at rest | Built-in per column (INT64, DOUBLE, STRING…) |
| Schema | None embedded — inferred or supplied externally | Embedded in the file footer |
| Splittable | Limited (awkward for parallel reads) | Yes — row groups are independently readable |
| Tooling | Universal — any language, tool, or spreadsheet | Pandas, PyArrow, DuckDB, Spark, most cloud warehouses |
Why data teams switched
Three mechanisms make Parquet faster and cheaper for analytical queries, and they compound.
Column pruning. When a query needs 4 columns from a 50-column table, Parquet reads just those 4 and skips the other 46 — a fraction of what CSV scans, cutting both query time and cloud compute cost.
Compression. After per-column encoding, Parquet files are often several times smaller than the equivalent CSV, though the ratio depends on the data. Smaller files mean faster transfers, lower storage costs, and less I/O per query.
Predicate pushdown. Row-group statistics let the engine skip chunks of the file based on a WHERE clause. A filter like WHERE country = 'DE' on a billion-row table can skip most row groups entirely without decompressing them.
The caveat: these gains apply to analytical queries reading a subset of columns over many rows. For a small reference table or a file you exchange once a month, Parquet adds no meaningful advantage.
When CSV is still the right choice
CSV remains the right tool in three situations. Sending a few hundred rows to a business partner or a SaaS import wizard? CSV opens in Excel with no extra software. When a human needs to inspect or edit data by hand, CSV is the only practical choice — Parquet is binary and unreadable in a text editor. And when the receiving system only accepts CSV, there is no decision to make.
- Files are small (thousands of rows, not millions)
- A human needs to read, inspect, or edit the data
- The destination only accepts CSV
- Maximum portability matters more than query speed
- Analytical queries scan a few columns across many rows
- Storage costs or query compute costs are a concern
- You need reliable data types and an embedded schema
- Parallel or distributed reads are part of the pipeline
How to convert CSV to Parquet
The right conversion tool depends on file size. Pandas & PyArrow handle files that fit in memory with a single to_parquet() call. DuckDB is faster for larger files, converting in a single local SQL statement with no DataFrame overhead. Apache Spark is the right choice when data is too large for one machine or when you are already running a distributed pipeline.
Two things matter regardless of tool: set the schema explicitly rather than relying on type inference, and partition the output by a common filter column like date or country so future queries can skip whole directories.
One boundary worth knowing: Parquet is a file format, not a table format. If you need ACID transactions or row-level updates, look at Delta Lake, Apache Iceberg, or Apache Hudi — table formats that layer those features on top of Parquet files.
Frequently asked questions
Is Parquet always better than CSV?
No. Parquet wins on large analytical workloads where queries read a few columns across millions of rows. For small files, human-readable data, or interchange compatibility, CSV is simpler and perfectly adequate. Follow the workload, not a blanket rule.
Can I open a Parquet file in Excel?
Not directly — Parquet is binary and appears as unreadable bytes in a text editor or Excel. To inspect it you need a reader: pandas and PyArrow in Python, DuckDB via SQL, or a Parquet viewer extension for VS Code.
Is Parquet compressed?
Yes. Parquet applies compression per column. Common codecs are Snappy (fast, moderate ratio) and Zstandard (better ratio, still fast). Even without a codec, the columnar layout and dictionary encoding produce files substantially smaller than equivalent CSV.
How is Parquet different from a CSV.gz?
A gzip-compressed CSV is smaller but still row-oriented text — you still decompress and parse every row for any query. Parquet’s columnar layout lets the engine skip columns and row groups based on statistics. Compression alone does not give you column pruning or predicate pushdown.
Understanding the csv vs parquet tradeoff is one of those foundational data engineering concepts that keeps paying off. For the full picture of how file formats fit into a working analytics system, see the modern data stack overview. If you are deciding where to store Parquet files, data warehouse vs data lake covers that storage trade-off in depth.
Last updated: July 6, 2026
Thanks for sharing! A solid grasp of data engineering fundamentals makes learning advanced data technologies much easier.
ReplyDelete