A tokenizer that lets LLMs read 4x more (Demo inside)

2 points by bramblestudio 6 hours ago

I'm a solo developer working on semantic compression for LLMs.

I built a tokenizer that reduces up to 75% of token usage by extracting Subject-Verb-Object structures, adding contextual completion, and dropping redundant language.

Runs on CPU, no training required. Just clean, logical compression — results are stable, and it avoids hallucination. Great for boosting model throughput and reducing cost.

Tested on long academic paragraphs, averaging 4x token density gain for English text.

Live demo here: https://huggingface.co/spaces/Sibyl-V/BSE_demo

Looking to license or collaborate — ideally with people who can handle commercial execution, so I can keep focusing on core modules.

Any feedback welcome. Open to talk.