Did you know you can now put the text of almost all of arXiv on one cheap SSD.
This is a dev post classified by Jev as AI dev tools (a tool drop), kept by the Dev Radar because it carries real work, not commentary.
Did you know you can now put the text of almost all of arXiv on one cheap SSD. Someone just packaged an enormous arXiv snapshot on Hugging Face 📚 3,148,796 papers 📄 PDFs for 99.47% 💽 Full archive: 16.08 TB Sounds ridiculous for Local AI… Until you see that … 🔥 paper_text = 70GB That contains the resolved TeX text for … 🧠 2,856,227 papers 📊 90.7% of all arXiv And you can download just that dataset by itself. No 16TB NAS is necessary. So theoretically you could build a completely local research system using ThumbLLM.exe from a thumb drive!!! 💾 70GB arXiv text corpus 🔎 local search / emb
Posted by David Hendrickson (11.2k followers) 1 h ago · 9 likes · 491 views · view the original post on X. Kept by the Dev Radar as AI dev tools.
More dev work like this
- Introducing the best value coding subscription ever — @jahooma
- 😂笑死，智普创始人唐杰亲自上小红书点举报，这波公关直接深入一线了！ — @NFT_Chen
- Keeping multiple coding agents moving gets messy fast. This dashboard gives you a… — @DanKornas
- Coding agents are changing how software gets built. — @freeCodeCamp
- Generates citation-backed reports with source credibility scoring and automated… — @tom_doerr
- Running several coding agents shouldn’t mean hunting across terminal tabs. — @DanKornas
- Join Happy Horse and create your own paper-cutting world. — @alibaba_cloud
- Removes AI tells from prose by teaching Claude or any LLM to detect and eliminate… — @tom_doerr
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 17.7k posts from 4.9k X accounts over the last 21 days, 2k tools, 12 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-21 04:44 UTC. Full method.