Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering

Hangrui Xu; Zhengxian Wu; Yunyao Yu; Zhuohong Chen; Rui Cong; Xiangwen Deng; Zhifang Liu; Peng Jiao; Haoqian Wang

doi:10.1145/3767308.3836227

Accepted at ACM Multimedia 2026 KB-VQA

Beyond Visual Similarity. Entity-Aligned Retrieval for Knowledge-Based VQA

KBMR teaches multimodal language models to retrieve knowledge by entity identity—not merely visual resemblance—so the right evidence survives viewpoint changes and deceptive look-alikes.

Hangrui Xu^*Zhengxian Wu^*Yunyao Yu^*Zhuohong ChenRui CongXiangwen DengZhifang LiuPeng JiaoHaoqian Wang^†

Tsinghua University

University of Arizona

^* Equal contribution. ^† Corresponding author.

Read the paper View code Cite KBMR

arXiv 2608.21450
DOI 10.1145/3767308.3836227
Code Apache-2.0

Comparison of EVA-CLIP and KBMR retrieval for a hotel image: visual similarity leads to a wrong entity, while entity-aware retrieval finds Strategic Hotels and Resorts. — Figure 1 Appearance-led retrieval can favor a convincing look-alike. KBMR retrieves evidence aligned with the queried entity. Reproduced from Xu et al. (2026), CC BY 4.0.

+14.7R@1 points on InfoSeek

+11.4R@1 points on E-VQA

3KB-VQA benchmarks

100KWikipedia-entry retrieval KB

Why entity alignment?

A visually plausible result can still be the wrong answer.

First-stage retrieval determines what downstream reasoning is allowed to see. When that pool is built only around appearance, the correct Wikipedia entity can disappear before reranking or generation even begins.

Visual collision

Different entities can look alike.

Architecture, species, products, and landmarks often share near-identical visual patterns while pointing to completely different evidence.

Entity continuity

The same identity changes appearance.

A single entity may be photographed from another viewpoint, in another season or era, or rendered in a different visual style.

KBMR target

Retrieve the identity, unlock the evidence.

KBMR reshapes the embedding space around entity-level correspondence, supplying a more faithful candidate pool to any downstream KB-VQA pipeline.

The representation shift

From “does it look similar?” to “does it refer to the same entity?”

Rather than stacking another filter on a CLIP candidate pool, KBMR changes the first-stage retriever itself.

Method · four connected moves

Turn semantic judgment into a retrieval space.

A separate MLLM first supplies graded entity-consistency signals. KBMR uses those signals to mine difficult evidence and distill a fast embedding retriever.

KBMR pipeline showing a trainable MLLM retriever and continuous semantic distillation on the left, and EVA-CLIP candidate mining, semantic discrimination, filtering, and hard-negative sampling on the right. — Figure 2 EVA-CLIP candidate mining, MLLM semantic discrimination, stratified hard-negative construction, and continuous semantic distillation. Reproduced from Xu et al. (2026), CC BY 4.0.

01encode

Embed semantic identity

Prompt an MLLM with the image, take its final-token hidden state, and normalize it into the retrieval embedding.

<Image> Summary above image in one word:
02mine

Find visual hard cases

EVA-CLIP-8B retrieves the Top-50 visually close candidates after known positives are excluded.
03score

Measure entity agreement

The Semantic Discriminator converts its Yes/No logits into a continuous consistency weight.

w_i = σ((z_yes − z_no) / γ) γ = 1.1
04distill

Learn the soft neighborhood

Four difficulty strata yield eight hard negatives. Symmetric KL aligns retrieval similarity with the semantic prior.

ℒ_CSD = ½ [ KL(p ∥ s) + KL(s ∥ p) ]

Fast at inference

The teacher stays in training.

The Semantic Discriminator constructs supervision offline. At deployment, the trained KBMR retriever embeds images and performs ordinary similarity search—without a pairwise MLLM judge in the retrieval loop.

Query→KBMR vector→Search→Evidence

Paper-reported results

Better retrieval reaches the final answer.

Entity alignment raises first-stage recall across benchmarks, then transfers into stronger downstream KB-VQA when KBMR replaces the original retriever.

E-VQAR@1

24.7%

+11.4 points

vs. EVA-CLIP-8B at 13.3%

InfoSeekR@1

60.3%

+14.7 points

vs. EVA-CLIP-8B at 45.6%

OK-VQAPseudo R@5

80.4%

KBMR · LLaVA-OV-7B

Entity retrieval performance

OK-VQAVQA score*

79.3%

+12.7 points

OMGM with KBMR retrieval

One query, one decisive shift

Corvus albicollis moves from the edge of the pool to the first result.

In the paper's qualitative example, EVA-CLIP fills the top ranks with visually similar bird species. KBMR restores the correct entity to rank one.

EVA-CLIP #5 correct entity rank

KBMR #1 correct entity rank

Retrieval

Entity retrieval on E-VQA and InfoSeek

Recall (%), higher is better.

Paper-reported retrieval comparison for the strongest visual baseline and both KBMR backbones.
Retriever	E-VQA R@1	E-VQA R@5	InfoSeek R@1	InfoSeek R@5
EVA-CLIP-8B visual baseline	13.3	31.3	45.6	67.1
KBMR Qwen2-VL-7B	22.6	43.1	55.8	71.1
KBMR LLaVA-OV-7B	24.7	48.7	60.3	74.7

Scope. E-VQA and InfoSeek use the stated 100K-entry Wikipedia knowledge base; these values should not be read as full-Wikipedia retrieval results.

Availability. The public repository documents the Qwen2-VL-7B implementation. LLaVA-OV-7B values are paper-reported and do not imply a released LLaVA checkpoint.

Downstream · OMGM

E-VQA and InfoSeek

Downstream answer accuracy after replacing OMGM's retriever with KBMR.
Pipeline	E-VQA All	InfoSeek All
OMGM	50.2	43.5
OMGM + KBMR	54.7	50.8
Absolute change	+4.5	+7.3

Downstream · OK-VQA

Plug-in retrieval impact

OK-VQA comparison after replacing OMGM's retriever.
Setting	Pseudo R@5	VQA score
OMGM	73.4	66.6
OMGM + KBMR LLaVA-OV-7B	80.4	79.3*
Absolute change	+7.0	+12.7

* KBMR replaces OMGM's retriever while the downstream pipeline is retained. This is not a stand-alone retriever VQA score. All values on this page are reported by the paper.

Citation

Build on entity-aligned retrieval.

If KBMR helps your research, please cite the paper and explore the complete open-source implementation.

Venue: ACM Multimedia 2026
arXiv: 2608.21450
DOI: 10.1145/3767308.3836227
Code: realHarryX/KBMR

BibTeX

@misc{xu2026kbmr,
  title = {Beyond Visual Similarity: Entity-Aligned Retrieval
           for Knowledge-Based Visual Question Answering},
  author = {Hangrui Xu and Zhengxian Wu and Yunyao Yu and
            Zhuohong Chen and Rui Cong and Xiangwen Deng and
            Zhifang Liu and Peng Jiao and Haoqian Wang},
  year = {2026},
  eprint = {2608.21450},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi = {10.1145/3767308.3836227},
  url = {https://arxiv.org/abs/2608.21450}
}

Ready to copy.