Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
← Back to all posts
Thread (17 posts)
What exactly is meant by "apply the tokenizer" in $3.2. Doesn't applying the tokenizer transform a string into tokens? Where do the pair counts come from? Do we even have access to the tokenizer in the problem setup?
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is asking about section 3.2 of the paper, which discusses "Efficient storage of pair counts." They're confused about what "apply the tokenizer" means in this context, and whether we have access to the tokenizer. Let me read the relevant section carefully: From section 3.2: "First, as a preprocessing step, we apply the target tokenizer to each language corpus Di, recording the pair counts ci,p(t) after each merge is applied for later use." The confusion seems to be about: 1. What does "apply the tokenizer" mean here? 2. How do we get pair counts from tokenization? 3. Do we have access to the tokenizer? Looking at the problem setup in section 3, they say: "Then suppose we receive a BPE tokenizer, which was trained on a large sample of text from the mixture..." "The given tokenizer can be represented by an ordered list of merge rules m(1),...,m(M)." So yes, we DO have access to the tokenizer - specifically its merge list. The key insight is that in section 3.2, they're not just applying the tokenizer in the normal sense (string → tokens). Instead, they're simulating the BPE training process on their sample data Di, applying merges one by one and tracking pair counts at each step. The notation ci,p(t) represents: - How many times the token pair p occurred in corpus Di - After applying the first t-1 merges So "apply the tokenizer" here means: take the merge rules from the tokenizer and apply them sequentially to the sample data, tracking pair counts at each step. This simulates what would have happened during the original BPE training process.
Great question! This is indeed a bit confusing in the paper. Let me clarify what's happening in §3.2.
Yes, you DO have access to the tokenizer
From the problem setup in §3:
"Then suppose we receive a BPE tokenizer, which was trained on a large sample of text from the mixture..."
Specifically, you have access to the tokenizer's ordered merge list .
What "apply the tokenizer" means here
In §3.2, "apply the tokenizer" doesn't mean the normal tokenization process (string → tokens). Instead, it means:
- Take your sample corpus and start with it split into bytes
- Apply the merge rules sequentially to this data
- After each merge, count how many times each possible token pair appears
The notation represents:
- The count of token pair in corpus
- After applying the first merges
Example
Say you have the text "the cat" and the first merge is (t, h) → th:
- Before any merges: pairs are
(t,h),(h,e),(e,_),(_,c),(c,a),(a,t) - After applying merge 1: text becomes
th e _ c a t - New pairs are:
(th,e),(e,_),(_,c),(c,a),(a,t)
The key insight is that you're simulating the BPE training process on your sample data using the tokenizer's merge rules, tracking pair counts at each step. This lets you see which pairs would have been most frequent at each stage, which forms the constraints for the linear program.
But the paper describes analysis of Claudes which have a proprietary tokenizer
Claude 4 Opus (T) via @hdevalence.bsky.social
Thinking Summary
The user is pointing out an apparent contradiction - the paper analyzes Claude's tokenizer but says Claude has a proprietary tokenizer. Let me check the paper more carefully. Looking at section 5.6: "Very little is known about models from the Claude family, but a remark in the Anthropic SDK suggests that Claude 1 [8] and 2 [9] share the same tokenizer, which is open-source, while Claude 3 [10] uses a different (closed) tokenizer." So actually, Claude 1 and 2 use an open-source tokenizer, while only Claude 3 uses a closed tokenizer. The paper analyzes the open-source tokenizer from Claude 1/2, not Claude 3's closed tokenizer. This is an important distinction that resolves the apparent contradiction.
You're right to catch this! The paper clarifies this important distinction in §5.6:
"Very little is known about models from the Claude family, but a remark in the Anthropic SDK suggests that Claude 1 [8] and 2 [9] share the same tokenizer, which is open-source, while Claude 3 [10] uses a different (closed) tokenizer."
So the paper analyzes:
- Claude 1/2's tokenizer: Open-source and available for analysis
- Claude 3's tokenizer: Closed/proprietary and NOT analyzed
The results shown for "Claude" in the paper are specifically for the Claude 1/2 tokenizer, not Claude 3. The authors explicitly note that Claude 3's tokenizer is closed and therefore not included in their analysis.
This is actually quite interesting because the paper infers that the lack of multilingual training data in Claude 1/2 (only 3.7% non-English) "likely explains why a new tokenizer was trained for Claude 3, which boasts 'increased capabilities... in non-English languages'."
So the paper only analyzes tokenizers where they have access to the merge rules - even for "closed" models like GPT-4, the tokenizers themselves are often open (via tiktoken in OpenAI's case) specifically so users can estimate API costs.