How this site works: the work explorer and AI assistant
The work explorer brings my projects, research, experience, and writing into one view. You can search for a topic, inspect a piece of work, and follow its connections to other entries. The AI assistant offers another way in: ask a question about my background and review the source material used to answer it.
Both features use embeddings, which turn written descriptions into numbers that can be compared for similarity. This guide explains how to use the explorer, what its map means, and how the assistant finds information.
Start with a topic
Use the search box to look for a subject, project, or organization. The filters narrow the results to Projects, Research, Experience, or Writing. Map view shows the connections between entries; List view makes it easy to scan their titles and summaries.
- For example, search for "tokenization" and select Projects to find JOLT, my work on learning tokenization jointly with a language model.
- Select JOLT to read its summary and open the case study. The Related work panel also connects it to the corresponding research publication.
- Choose that related publication to continue exploring. If it falls outside your current search or type filter, the explorer clears those filters so the entry becomes visible.
Your selection stays open while you inspect other points. With a keyboard, use the arrow keys to move between map points and Enter or Space to select one. List view provides the same details and related-work links.
What embeddings add
Each entry has a written description drawn from the site's content. Google's gemini-embedding-001 model converts that description into a vector: a list of numbers representing patterns in the text. Comparing these vectors helps identify entries with similar subjects, including entries that use different wording.
The explorer's search box matches words in titles and descriptions. Embeddings power the map positions and related-work suggestions, and help the assistant find passages relevant to a question. Keeping these roles distinct makes the interface easier to interpret: a search narrows the visible entries, while a connection suggests where to explore next.
Projects, research, experience, and writing
Numerical representations of the descriptions
Map positions and related work
Search for relevant source passages
How to read the map
Each point represents one entry. Its color and shape identify the type of work, and the labeled axes provide a broad thematic orientation:
- Left to right: research to engineering, from an emphasis on methods and experiments toward building and operating systems.
- Top to bottom: applied machine learning to language models, from work on domain-specific data toward text, tokenization, and language-model behavior.
Each axis is defined by two reference descriptions embedded with the same model as the work. The difference between those reference vectors defines a direction; projecting an entry onto it gives that entry a coordinate. These chosen directions make the map interpretable, although many projects naturally span both ends of an axis.
The coordinates are scaled to fit the available space, and nearby points are spread slightly to keep them selectable. Distances on screen are therefore approximate. Use the map to recognize themes and discover connections; a position does not measure the quality, impact, or maturity of the work.
How related work is chosen
Connections come from cosine similarity, a measure of how closely two embedding vectors point in the same direction. Each entry contributes links to its three closest matches, and duplicate links are combined. An entry can have more than three lines because other entries may also choose it as a close match.
The Related work panel shows the three strongest connections for the selected entry, ranked by similarity in the original embedding space. That ranking is independent of the spacing adjustments used to draw the map. For JOLT, this brings the project and its research publication together even though they appear in different sections of the portfolio.
How the assistant finds an answer
The assistant searches a collection of source passages: the explorer's entries, individual sections of blog posts, and profile facts. Category summaries help it answer broad questions such as "What has Deepanshu published?" Its search index uses a compact, 256-number version of the embeddings generated for the site.
When you ask a question, the site embeds it and compares it with the stored passages. Candidates must meet a minimum similarity threshold. Matching words in the passage and its title then contribute small ranking bonuses, and up to four passages are supplied to the language model alongside core profile facts.
The model is instructed to answer from that material and acknowledge missing information. This approach is called retrieval-augmented generation, or RAG: find relevant information first, then use it to write an answer.
- Your question
Find passages using meaning and matching words.
- Source material
Supply up to four passages and core profile facts.
- Answer and sources
Read the response and inspect the retrieved context.
Source links below an answer let you open the relevant work. The expandable context view shows the retrieved passages and their scores, and matching map entries briefly highlight. These are the sources supplied to the model; they do not independently verify every sentence it writes. The scores indicate retrieval relevance, rather than a probability that the answer is correct.
Keeping the content current
The map and assistant index are generated from the site's content and stored with the code. After a content edit, a generation script rebuilds both. Development checks compare fingerprints of the current text with those stored in the generated files and warn when they differ.
Because the map data is prepared in advance, browsing, filtering, and following connections require no live AI call. The assistant does make external calls. If question embedding fails, it can use the full profile knowledge base as context; if answer generation fails, it returns an error message.
What to keep in mind
- Connections reflect the descriptions included on this site. The explorer does not independently analyze every linked paper or code repository.
- Similarity is a useful navigation signal. Related entries may share a method or topic while differing substantially in purpose and results.
- The assistant can miss relevant material or make mistakes. Review the linked work for detail, and contact me when a question needs clarification.
Automated checks cover filtering, connection ranking, map navigation, and data consistency. A separate evaluation suite exercises the assistant on documented facts, unknown information, unrelated questions, and attempts to override its instructions. These checks help catch regressions as the content and interface evolve.