tok_max: Maximize Token Efficiency

Compress text to maximize information density and machine-parseable structure. The inverse of humanizer: strip personality, filler, and ambiguity. Replace with precise terminology, flat statements, and high-density templates.

Overview

This skill transforms verbose or human-style text into token-efficient, machine-legible output. It eliminates 24 categories of wasteful patterns — from filler phrases and hedging to promotional language and conversational artifacts — and replaces them with structured, unambiguous formats.

Assumptions:

• The user wants maximum information density, not readability for humans
• Facts, names, dates, and numbers must be preserved exactly
• The output may feel "sterile" or "robotic" — this is the intended outcome
• Source citations should be specific or removed, never vague

Workflow

1. Analyze input — scan for compression targets
2. Apply patterns — compress using the 24 patterns
3. Restructure — choose high-density output template
4. Verify — run tok_compress.py for quantitative check
5. Deliver — output compressed text + tag block

Step 1: Analyze Input

Scan the input text and classify each sentence:

Load references/compression-patterns.md for the full 24-pattern catalog. Load references/efficiency-glossary.md for the phrase-to-term lookup table.

Step 2: Apply Compression Patterns

Process text through the 24 patterns in order. For each pattern found:

1. Identify the pattern match in the original text
2. Determine the precise replacement (use efficiency-glossary.md)
3. Verify the replacement preserves all facts from the original
4. Record the change for the tag block

Critical rules during compression:

• Never fabricate. If the original says "Experts believe" and no expert is named, delete the claim or label it [attribution: unknown]. Do not invent a source.
• Never add evaluation. Remove "crucial", "pivotal", "stunning", "vibrant" unless attributed to a specific person making that evaluation.
• Never hedge. Replace "it could potentially be argued that" with the direct claim or an explicit uncertainty label: [confidence: low].
• Always preserve numbers. Exact counts, dates, percentages, dollar amounts — never round or approximate.
• Always preserve named entities. People, organizations, products, locations — never generalize.

Step 3: Restructure Using Templates

After prose compression, choose the output format. Two tiers:

Tier 1 (default): High-density templates for human or mixed consumption. Load references/token-templates.md for templates (spec tables, decision trees, hierarchical lists, tag blocks).

Tier 2 (machine-only): Advanced token-level compression when the consumer is a parser, not a human. Load references/advanced-compression.md for control-character delimiters, scientific notation, hex dates, abbreviated keys, and punctuation elimination.

Select Tier 2 only when: (1) the consumer is a machine parser, (2) a schema can be established before data transmission, (3) human readability is not required.

Template selection:

Rules for all templates:

• Use sentence case for all headings
• No em dashes — use commas or periods
• No bold for emphasis — use headers or tables for structure
• No emojis
• No curly quotes — ASCII straight quotes only
• One term per concept — no synonym cycling

Step 4: Verify with tok_compress.py

Run the quantitative verification:

scripts/tok_compress.py original.txt compressed.txt

The script outputs:

• Word count before/after with compression ratio
• Pattern detection: how many of the 24 patterns remain
• Fact preservation ratio
• Verdict: PASS, NEEDS WORK, or REVIEW

Target metrics:

• Compression ratio: 30-60% reduction
• Remaining patterns: 0-2
• Fact preservation: >= 80%

If NEEDS WORK or REVIEW: return to Step 2 and compress further.

Step 5: Deliver Output

Provide:

1. Compressed text in the selected template format
2. Tag block with metadata about the compression:

[tok_max: applied | patterns: 8/24 found | compression: 52% | facts: 12/12 preserved]
[confidence: high | method: pattern-based compression | template: specification_table]

Reflection

After delivering compressed text, reflect on the quality gate results:

1. Did tok_compress.py report PASS? If NEEDS WORK, which patterns remain?
2. Are all facts preserved at >= 80%? If not, which facts were lost?
3. Is the output unambiguous — could a machine parse every sentence without inference?
4. Did the user want compression (preserve all) or summarization (allow loss)? Was this respected?

Record findings and return to Step 2 if any check fails.

Confidence Calibration

• High confidence: Overview — 24 patterns are comprehensive and well-tested
• High confidence: Workflow — 5-step linear process, no branching
• High confidence: Pattern catalog — derived from humanizer inverse, validated on AI-generated text
• Medium confidence: Template selection — requires judgment on information type classification
• High confidence: Fact preservation — rules are explicit and conservative

Edge Cases & Limitations

Edge Case: Input is already compressed. If the input text has <= 2 of the 24 patterns and uses flat statements, verify with tok_compress.py. If ratio < 10%, deliver the original with tag [tok_max: already optimal].

Edge Case: Input is creative writing/fiction. tok_max is not appropriate for creative text. The user wants soul and voice there. Decline and explain: "This text appears to be creative writing. tok_max strips voice and personality, which would destroy the intended effect."

Edge Case: Lossy compression requested. If the user explicitly asks to summarize (not compress), some fact loss is acceptable. Use [facts: summarized] in the tag block and note what was omitted.

Edge Case: Multi-modal input. If the input includes images, code blocks, or tables — compress only the prose around them. Leave structured data intact.

Limitation: No semantic understanding. tok_max operates on pattern matching, not meaning. It will not restructure arguments for logical flow — only compress the prose around them.

Limitation: No context awareness. tok_max does not know the audience. A scientific paper and a marketing email get the same treatment. The user must specify if a different compression level is needed.

Limitation: Token count is approximate. tok_compress.py uses word count as a proxy for tokens. Actual token counts depend on the tokenizer. The ratio is directionally correct but not exact.

Assumptions

• Python 3.10+ is available for running tok_compress.py
• The user has write access to the working directory for temporary files
• Input text is UTF-8 encoded
• The user understands that output will feel mechanical — this is intentional

Resources

scripts/

See scripts/guide_scripts.md for executable utilities.

references/

See references/guide_references.md for detailed reference documentation. Load references only when working on the specific task they cover:

• Pattern compression (Step 2) → references/compression-patterns.md
• Template selection (Step 3) → references/token-templates.md
• Phrase lookup (Step 2) → references/efficiency-glossary.md

Iteration Guidance

After delivering compressed text:

1. User reviews and may request further compression or partial restoration
2. If further compression: return to Step 2, focus on patterns not yet eliminated
3. If partial restoration: identify which facts the user wants emphasized, restructure the template accordingly
4. Re-run tok_compress.py after each iteration to verify metrics
5. Update the tag block with new compression stats