# Cutting Text to a Limit Without Breaking It: Code Units, Bytes and Tokens in JavaScript and Python

Most apps cut text somewhere. A preview gets 160 characters, a push notification has a byte limit, a database column has a size, and a prompt for a language model has a token budget. The FindUtils post on [why an emoji counts as 2 characters](https://findutils.com/blog/why-an-emoji-counts-as-2-characters-string-length-explained/) explains why one string has four different lengths. This post is about the next step: **cutting** the string to fit one of those lengths without leaving broken text behind.

Every output below was produced on 25 October 2026 with Node 20.19.2 (plus Node 22.23.4 and 24.22.1 for one timing test), Chrome 156 through Playwright, and Python 3.13.5 with the `regex` (2026.9.29) and `tiktoken` (0.14.0) packages. Four test strings come up again and again:

```text
"Great job 👍🏽 team"     thumbs-up + skin-tone modifier
"Trip 🇹🇷🇩🇪 planned"     two flags, each made of two regional-indicator letters
"Family 👨‍👩‍👧 photo"     three people joined by two zero-width joiners
"Cafe\u0301 au lait"     e + combining accent (looks like "Café")
```

## 1. A code-unit cut can leave half a character

JavaScript's `slice` counts UTF-16 code units, the same unit as `.length` and the HTML `maxlength` attribute. Most emoji take two units (a surrogate pair), so a cut can land between them:

```text
"Great job 👍🏽 team".slice(0, 11)   -> "Great job \ud83d"   isWellFormed() false
"Great job 👍🏽 team".slice(0, 12)   -> "Great job 👍"        skin tone lost
"Trip 🇹🇷🇩🇪 planned".slice(0, 7)    -> "Trip 🇹"              half a flag
"Family 👨‍👩‍👧 photo".slice(0, 10)   -> "Family 👨‍"            one person + a dangling joiner
"Cafe\u0301 au lait".slice(0, 4)     -> "Cafe"                 accent gone
```

The first result is the dangerous one. `\ud83d` is a *lone surrogate*: half of a pair, which is not a valid character in any encoding. JavaScript doesn't complain. What happens next depends on where the string goes:

```text
JSON.stringify          {"t":"Great job \ud83d"}        escaped, still "valid" JSON
TextEncoder / Buffer    ... 20 ef bf bd                 silently replaced with U+FFFD (�)
fetch() request body    ... 20 ef bf bd                 same: the server receives "�"
encodeURIComponent      URIError: URI malformed         throws
```

So in the browser the text quietly gains a `�`, unless it goes into a URL, where it throws. The JSON route is worse, because the problem travels. Python accepts that JSON and fails later, somewhere else:

```text
json.loads('{"t": "Great job \\ud83d"}')   ok, a str of length 11
t.encode("utf-8")                         UnicodeEncodeError: surrogates not allowed
sqlite3 insert of t                       UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' ...
json.dumps(t, ensure_ascii=False).encode  UnicodeEncodeError
```

The stack trace points at a database write in a worker, while the bug is a `slice` in a front end. Node 20 and current browsers have `String.prototype.isWellFormed()` and `toWellFormed()`, which are cheap guards at an API boundary. They don't fix the cut, though. They only turn half an emoji into `�`.

## 2. Code points are not enough

The usual next attempt is to cut by code points, with `[...text]` in JavaScript or plain slicing in Python, where strings are sequences of code points. That removes the lone surrogate, but not the problem:

```text
[..."Great job 👍🏽 team"].slice(0, 11).join("")    -> "Great job 👍"     (Python s[:11] gives the same)
[..."Family 👨‍👩‍👧 photo"].slice(0, 9).join("")    -> "Family 👨‍"        ends with U+200D
[..."Trip 🇹🇷🇩🇪 planned"].slice(0, 6).join("")     -> "Trip 🇹"
[..."Cafe\u0301 au lait"].slice(0, 4).join("")     -> "Cafe"
```

Nothing throws, and every result is valid Unicode. Every result is also different text from what the user wrote. The skin tone is gone, a family turned into one man, a flag turned into the letter T, and "Café" turned into "Cafe". The unit a reader sees is the **grapheme cluster** from [Unicode Standard Annex #29](https://www.unicode.org/reports/tr29/), and that is the only safe place to cut.

## 3. Cut on graphemes, measure in the budget's unit

The fix keeps two things apart: *where* you may cut (between graphemes) and *how* you count (whatever unit the limit uses). In JavaScript, `Intl.Segmenter` gives graphemes:

```js
const seg = new Intl.Segmenter(undefined, { granularity: 'grapheme' });

export function truncate(text, max, { ending = '…', measure = s => s.length } = {}) {
  if (measure(text) <= max) return text;           // fits: no ending
  const budget = max - measure(ending);            // the ending counts toward the limit
  let out = '';
  for (const { segment } of seg.segment(text)) {
    if (measure(out + segment) > budget) break;    // stop before the grapheme that doesn't fit
    out += segment;
  }
  return out.trimEnd() + ending;
}
```

The results, with no ending and then with `…` (one code unit):

```text
truncate("Great job 👍🏽 team", 12, {ending:""})  -> "Great job"      | with …: "Great job…"
truncate("Trip 🇹🇷🇩🇪 planned", 9, {ending:""})   -> "Trip 🇹🇷"        | with …: "Trip…"
truncate("Family 👨‍👩‍👧 photo", 12, {ending:""})  -> "Family"         | with …: "Family…"
truncate("Cafe\u0301 au lait", 5, {ending:""})    -> "Café"           | with …: "Caf…"
```

Look at the flag row. With a limit of 9 and no ending, "Trip 🇹🇷" fits exactly. With `…`, the budget drops to 8, so the Turkish flag no longer fits and the whole flag goes. That is correct: the result never goes over the limit and never contains a piece of a character. The result can be a few units shorter than the limit. A family emoji is 8 code units, so it either fits completely or it is left out.

Python's standard library has no grapheme segmentation, but the third-party `regex` module matches one grapheme with `\X`:

```python
import regex

def truncate(text, max_units, ending='…', measure=len):
    if measure(text) <= max_units:
        return text
    budget, out = max_units - measure(ending), ''
    for g in regex.findall(r'\X', text):
        if measure(out + g) > budget:
            break
        out += g
    return out.rstrip() + ending
```

`truncate("Trip 🇹🇷🇩🇪 planned", 7, '')` gives `"Trip 🇹🇷"`, and `truncate("Cafe\u0301 au lait", 4, '')` gives `"Caf"`. (Python's `len` counts code points. Pass a different `measure` if your limit is in UTF-16 units, bytes or tokens.)

I also ran the same strings through the FindUtils [Text Truncate](https://findutils.com/text/text-truncate/) tool in Chrome, with Break on word off and no ending. It never split a character either: limit 11 or 12 on the thumbs-up string gave "Great job " (10), the family string gave "Family " (7) at limits 10 and 12, and "Cafe\u0301" at limit 4 gave "Caf". It keeps the trailing space, while the function above trims it. The [Text Truncate guide](https://findutils.com/guides/text-truncate-guide/) covers the tool's settings. It is a quick way to get expected values for your own unit tests.

## 4. Byte budgets: the naive cut can come out longer

Byte limits are common: a column sized in bytes, a header, a push payload, a key-value store with a value size limit. The obvious approach is to encode, cut the bytes and decode:

```text
text = "Family 👨‍👩‍👧 photo from Kraków"   (44 UTF-8 bytes)

Buffer.from(text).subarray(0, 10).toString()  -> "Family �"            10 bytes
Buffer.from(text).subarray(0, 22).toString()  -> "Family 👨‍👩‍�"         24 bytes (!)
truncate(text, 22, {ending: "", measure: utf8Bytes}) -> "Family"     6 bytes
```

The second line is the surprise. The cut landed inside the third person (an emoji is 4 bytes in UTF-8), Node replaced the 2 leftover bytes with `�`, and `�` itself is 3 bytes. Re-encoded, the "22-byte" result is 24 bytes, which is over the limit you were trying to meet.

Python is stricter by default and fails loudly. Its lenient options still produce wrong text:

```text
b[:10].decode("utf-8")              UnicodeDecodeError
b[:10].decode("utf-8", "ignore")    "Family "
b[:22].decode("utf-8", "ignore")    "Family 👨‍👩‍"        two people and a dangling joiner
b[:22].decode("utf-8", "replace")   "Family 👨‍👩‍�"
```

`errors="ignore"` is a popular answer to "how do I cut UTF-8 safely", and it does give valid UTF-8. It doesn't give whole characters. The grapheme loop with a byte `measure` (`s => Buffer.byteLength(s, 'utf8')` in Node, `lambda s: len(s.encode('utf-8'))` in Python) returns "Family" here, because the family emoji alone is 18 bytes and doesn't fit in the remaining 15. A shorter result is the right answer. If you want to check a byte count by hand, the [URL Encoder](https://findutils.com/network/url-encoder-decoder/) shows each UTF-8 byte as a `%XX` group.

## 5. Token budgets: the same problem, one layer up

Language-model budgets are in tokens, and tokenizers work on bytes. With OpenAI's [tiktoken](https://github.com/openai/tiktoken) and the `o200k_base` encoding, the token pieces show where the boundaries fall:

```text
"Great job 👍🏽 team"   6 tokens: 'Great' | ' job' | ' 👍' | f0 9f 8f | bd | ' team'
"Family 👨‍👩‍👧 photo"   10 tokens: the family takes 8 of them, each person split across two
```

The skin-tone modifier is split across two tokens, so keeping the first four tokens and decoding gives half a character:

```text
enc.decode(ids[:4])  -> "Great job 👍�"
enc.decode(ids[:2])  -> "Family �"       (family string)
enc.decode(ids[:8])  -> "Family 👨‍👩‍�"
```

`decode` replaces the broken bytes by default, so this doesn't fail. It sends `�` to the model, or stores it in a summary. In a Turkish sample ("Merhaba, nasılsınız? Çok güzel!", 11 tokens) every token happened to end on a whole character. A test on a few English and Turkish strings will pass, and emoji are what break it.

The same grapheme loop works for tokens if `measure` counts tokens. There is one more catch, which I measured. **The token count of a prefix isn't monotonic**: across 202 grapheme prefixes of a mixed sample, a longer prefix had *fewer* tokens than the one before it 9 times, because the tokenizer merges pieces differently once more text follows. In practice, measure the final candidate together with its ending rather than adding up per-grapheme counts:

```python
def truncate_tokens(text, max_tokens, ending='…'):
    if len(enc.encode(text)) <= max_tokens:
        return text
    gs = regex.findall(r'\X', text)
    lo, hi = 0, len(gs)                      # binary search over grapheme prefixes
    while lo < hi:
        mid = (lo + hi + 1) // 2
        if len(enc.encode(''.join(gs[:mid]).rstrip() + ending)) <= max_tokens:
            lo = mid
        else:
            hi = mid - 1
    return ''.join(gs[:lo]).rstrip() + ending
```

Budgets of 6, 7 and 9 tokens on "Great job 👍🏽 team, see you tomorrow" gave "Great job 👍🏽…" (6), "Great job 👍🏽 team,…" (7) and "Great job 👍🏽 team, see…" (9). For budgets 6 to 9, a slow linear scan over every prefix picked the same cut each time. Because the count isn't strictly monotonic, a binary search can in principle stop one grapheme short of the best cut. It can never return something over the budget, since every candidate is measured with its ending.

## 6. A performance trap in Node 20

`Intl.Segmenter` is the right tool, but on Node 20 it slows down badly on long strings. I timed iterating every grapheme of the same repeated sentence:

```text
code units   Node 20.19.2   Node 22.23.4   Node 24.22.1   Chrome 156
10,000            20 ms         3.7 ms         1.0 ms        0.7 ms
40,000           195 ms         9.5 ms         2.4 ms        2.2 ms
80,000         8,660 ms        17.1 ms         8.8 ms        3.8 ms
```

The jump from 40,000 to 80,000 units on Node 20 was the same in two runs (8.9 s and 8.7 s). Iterating the same 80,000-unit string by code point took under 1 ms. The `truncate` above only walks as far as the limit, so cutting a 160,000-unit string to 280 took 50 ms even on Node 20. What hurts is code that segments the *whole* string first, for example to count graphemes or to cut from the end. Node 20 reached end-of-life on 30 April 2026 according to the [Node.js release schedule](https://github.com/nodejs/Release), so this is one more reason to move to 22 or 24.

## A checklist for any cut

1. **Name the unit of the limit**: UTF-16 units (`maxlength`, most UIs), code points (Python, MySQL `CHAR_LENGTH`), bytes (storage, headers, payloads) or tokens (models).
2. **Cut between graphemes, always**: `Intl.Segmenter` in JavaScript, `regex` `\X` in Python. Never call `slice`, `subarray` or `decode(errors="ignore")` on user text.
3. **Measure the candidate with its ending**, and let the ending count toward the limit.
4. **Expect results shorter than the limit.** A family emoji is 8 units, 18 bytes or several tokens, and it fits whole or not at all.
5. **Guard boundaries.** Use `isWellFormed()` before sending JSON to a stricter service. Normalize to NFC first if two visually identical strings should get the same cut.
6. **Test with the four strings above**: skin tone, flag, ZWJ family, combining accent. Plain English text will pass any implementation.
7. **Don't segment huge strings on Node 20.** Stop at the limit, or upgrade.

For quick checks while you write tests, the [Character Counter](https://findutils.com/text/character-counter/) counts the way JavaScript does and the [Word Counter](https://findutils.com/text/word-counter/) adds reading time for longer drafts.

---

*Need help with a project? The team behind FindUtils takes on freelance work on [Upwork](https://www.upwork.com/freelancers/~010a5875eab37b4bd7), including [adding an AI feature to your web app](https://www.upwork.com/services/product/development-it-an-ai-feature-in-your-web-app-chat-assistant-document-analysis-or-search-2112348761417603851), such as a chat assistant, document analysis or search.*