Apache Spark Data Source V2 is Spark’s way to support native table formats, not only…
This is a dev post classified by AGK Intelligence as Databases (a free resource), kept by the Dev Radar because it carries real work, not commentary.
Apache Spark Data Source V2 is Spark’s way to support native table formats, not only files. Szehon Ho walks through why that exists. Early Spark read and wrote file formats such as Parquet and used a thin Hive Metastore catalog. Apache Iceberg and Delta Lake added a much richer metadata layer so engines can run large, transactional operations. DSV2 expands that surface so a data source can plug that metadata into Spark, including through catalogs. 🎥 http://youtube.com/watch?v=MOQ3G8Ul4Gk&feature=youtu.be #ApacheSpark #OpenSource
Posted by Apache Spark (44.8k followers) 3 h ago · 3 likes · 544 views · view the original post on X. Kept by the Dev Radar as Databases.
More dev work like this
- #SQL Query Execution Order! #BigData #Analytics #DataScience #AI #MachineLearning #IoT… — @gp_pulipaka
- AI agents don’t stop at one query. One prompt can trigger 50 SQL queries, each needing… — @ClickHouseDB
- Your collection is growing. But is it time to shard, and which shard key fits your… — @MongoDB
- Your collection is growing. But is it time to shard, and which shard key fits your… — @MongoDB
- Our CTO @gakonst and team hit Tempo with about 35,000 TPS and it handled nearly 24,000. — @tempo
- HyperIndex handlers can now read entities stored only in @ClickHouseDB, so you can keep… — @envio_indexer
- https://pgx.dev is growing! 🚀 — @alxshp
- New Postgres, MySQL, MongoDB and Redis databases are private by default — @Railway
Every post is read and classified by AGK Intelligence: what it is, which market it belongs to, and whether the link is a real tool. 31.5k posts from 5k X accounts over the last 21 days, 9.1k tools, 12 markets. New posts are collected several times an hour and the full ranking is recomputed every hour — last update 2026-11-03 07:47 UTC. Full method.