Hugging Face has deployed a hybrid search system for Papers with Code using Qwen3-Embedding-0.6B, PostgreSQL full-text search and pgvector. Available through the website and the pwc search command, it maintains embeddings for more than 110,000 current papers sourced from arXiv and Daily Papers.
Each paper is encoded from its normalized title and abstract under a versioned embedding contract. Hugging Face records the exact model revision, output dimension, input format, query or document role, normalization method and a hash of the source text. The production configuration generates 256-dimensional L2-normalized vectors, using the document prompt for papers and the query prompt for searches.
Full-corpus encoding runs as a GPU batch workload through Hugging Face Jobs. Storage Buckets hold database snapshots, manifests and vector artifacts, while Inference Endpoints handle live queries and small document updates. Before vectors enter PostgreSQL, the importer checks schemas, checksums, dimensions, normalization, unique paper IDs and whether content hashes still match current records.
In Hugging Face’s 5,000-paper pilot, an NVIDIA L4 encoded about 75 papers per second at 1,024 dimensions. The 256-dimensional HNSW index delivered 0.9955 Recall@20 against exact search, with median lookup latency of 1.31 milliseconds and p95 latency of 2.21 milliseconds. Its table and index required about 27% of the storage used by the 1,024-dimensional version. These are vendor-reported results from the stated pilot.
For each query, the lexical and semantic branches retrieve up to 50 candidates apiece. Their rankings are combined through reciprocal rank fusion with equal weights and k=60. Additional rules keep exact titles and arXiv identifiers at the top, recognize certain navigational searches, tolerate incomplete titles and bounded spelling errors, and abstain from forcing ambiguous fuzzy matches.
The interactive embedding endpoint is limited to one replica and can scale to zero when idle. Its client applies a one-second production timeout, validates vector dimensions, finiteness and norm, and opens a circuit breaker after repeated failures. If the endpoint is starting, overloaded or returns an invalid response, Papers with Code skips semantic retrieval and immediately serves lexical results.
Practical context: Separating batch corpus encoding from live query processing reduces the search path’s practical dependence on GPU availability, because expensive catalog processing does not occur during a user request and lexical retrieval remains available as a fallback. Reproducibility also depends on preserving the revision, instructions, dimensions, normalization and input format—not merely the model name. The published pilot metrics characterize this configuration but do not assess relevance across a broad range of scientific queries.
| Component | Role | Workload |
|---|---|---|
| Hugging Face Jobs | Full-corpus encoding, new model generations and large backfills | GPU batch processing |
| Hugging Face Storage Buckets | Storage for snapshots, manifests and vector artifacts | Durable transfer between systems |
| Hugging Face Inference Endpoints | Query encoding and small document updates | Interactive and incremental processing |
| PostgreSQL and pgvector | Full-text and nearest-vector retrieval | Production candidate retrieval |
Sources
Event date: 2026-08-21. Primary source date: 2026-08-21.
