We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing…
This is a dev post classified by AGK Intelligence as Other (a tool drop), kept by the Dev Radar because it carries real work, not commentary.
We introduce Depth Hybrid Attention (DHA), which incorporates depth-wise value mixing into hybrid architectures with very little additional compute and cache overhead. By allowing layers to mix information across depth, DHA broadens each layer’s view and increases the model’s expressive capacity. Our DHA method adapts to multiple hybrid architectures, including Depth Window Attention (DWA) for GQA + SWA and Depth Delta Attention (DDA) for MLA + KDA. Across loss curves, short-context benchmark results, and long-context retrieval evaluations, both variants outperform their corresponding full-a
Posted by Nanbeige (5.7k followers) 3 h ago · 88 likes · 3.4k views · view the original post on X. Kept by the Dev Radar as Other. Tools mentioned: depth-hybrid-attention.
More dev work like this
- 🔥厉害了！微信视觉团队直接开源2B/4B视觉大模型 WeVisOne-VL，一个模型搞定检测+抠图！真正做到物体级 + 像素级统一感知！ — @NFT_Chen
- 「拍片哥女装」整理了 1407 张 AI 服装参考图，图片里不带人物，按春夏秋冬、派对等 8 类排列，打开网页就能翻。 — @GitHub_Daily
- We brought the MOSS family. You made it feel like a community. Thank you, @LocalLLaMAsub… — @MOSS_Global
- Starting the week at Devcon, sharing insights and a grasp of our general methodology in… — @miga_labs
- Japan’s JGB repo market is estimated at about $1.6 trillion, a market where faster… — @ZenithFdn
- VLM that says "it's not there" is now open source. — @TencentAI_News
- welcome to my code — @steveruizok
- NVIDIA NemotronLabs RE-USE is on ModelScope: a 9.6M-parameter speech restoration model… — @ModelScope2022
Every post is read and classified by AGK Intelligence: what it is, which market it belongs to, and whether the link is a real tool. 31.4k posts from 5k X accounts over the last 21 days, 9.1k tools, 12 markets. New posts are collected several times an hour and the full ranking is recomputed every hour — last update 2026-11-03 14:22 UTC. Full method.