Pandas Should Go Extinct
Points and comments are a snapshot, not live.
Author argues pandas DataFrame library should be replaced by Polars and DuckDB for most workloads.
The post argues that most analytics workloads are "Medium Data" (~100GB) and do not need distributed systems like Spark. Citing Amazon Redshift fleet data, the author claims 94.68% of tables are under 100GB and 86.9% of queries process ≤80GB. Benchmarks on a 1-billion-row CSV show Pandas takes 4m28s using 38GB RAM, while Polars (5.04s, 18GB) and DuckDB (5.19s, 1.93GB) are significantly faster and more memory-efficient. Key advantages cited are automatic multi-threading, lazy evaluation, and streaming.
What commenters are saying
Most commenters agree that Pandas is overused and that Polars or DuckDB are better for new projects. Several cite real-world performance gains from switching. However, some push back: Pandas still excels for ETL and integrates seamlessly with sklearn and matplotlib, while Polars has corner-case API issues and memory bloat in lazy mode. Two camps form: those who see Pandas as entrenched but inferior, and those who value its ecosystem and ergonomics for small-to-medium tasks. A few note the article initially had a broken link, which was quickly fixed.