Most apps cut text somewhere. A preview gets 160 characters, a push notification has a byte limit, a database column has a size, and a prompt for a language model has a token budget. The FindUtils post on why an emoji counts as 2 characters explains why one string has four different lengths. This post is about the next step: cutting the string to fit one of those lengths without leaving broken text behind.
Every output below was produced on 25 October 2026 with Node 20.19.2 (plus Node 22.23.4 and 24.22.1 for one timing test), Chrome 156 through Playwright, and Python 3.13.5 with the regex (2026.9.29) and tiktoken (0.14.0) packages. Four test strings come up again and again:
"Great job ππ½ team" thumbs-up + skin-tone modifier
"Trip πΉπ·π©πͺ planned" two flags, each made of two regional-indicator letters
"Family π¨βπ©βπ§ photo" three people joined by two zero-width joiners
"Cafe\u0301 au lait" e + combining accent (looks like "CafΓ©")
1. A code-unit cut can leave half a character
JavaScript's slice counts UTF-16 code units, the same unit as .length and the HTML maxlength attribute. Most emoji take two units (a surrogate pair), so a cut can land between them:
"Great job ππ½ team".slice(0, 11) -> "Great job \ud83d" isWellFormed() false
"Great job ππ½ team".slice(0, 12) -> "Great job π" skin tone lost
"Trip πΉπ·π©πͺ planned".slice(0, 7) -> "Trip πΉ" half a flag
"Family π¨βπ©βπ§ photo".slice(0, 10) -> "Family π¨β" one person + a dangling joiner
"Cafe\u0301 au lait".slice(0, 4) -> "Cafe" accent gone
The first result is the dangerous one. \ud83d is a lone surrogate: half of a pair, which is not a valid character in any encoding. JavaScript doesn't complain. What happens next depends on where the string goes:
JSON.stringify {"t":"Great job \ud83d"} escaped, still "valid" JSON
TextEncoder / Buffer ... 20 ef bf bd silently replaced with U+FFFD (οΏ½)
fetch() request body ... 20 ef bf bd same: the server receives "οΏ½"
encodeURIComponent URIError: URI malformed throws
So in the browser the text quietly gains a οΏ½, unless it goes into a URL, where it throws. The JSON route is worse, because the problem travels. Python accepts that JSON and fails later, somewhere else:
json.loads('{"t": "Great job \\ud83d"}') ok, a str of length 11
t.encode("utf-8") UnicodeEncodeError: surrogates not allowed
sqlite3 insert of t UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' ...
json.dumps(t, ensure_ascii=False).encode UnicodeEncodeError
The stack trace points at a database write in a worker, while the bug is a slice in a front end. Node 20 and current browsers have String.prototype.isWellFormed() and toWellFormed(), which are cheap guards at an API boundary. They don't fix the cut, though. They only turn half an emoji into οΏ½.
2. Code points are not enough
The usual next attempt is to cut by code points, with [...text] in JavaScript or plain slicing in Python, where strings are sequences of code points. That removes the lone surrogate, but not the problem:
[..."Great job ππ½ team"].slice(0, 11).join("") -> "Great job π" (Python s[:11] gives the same)
[..."Family π¨βπ©βπ§ photo"].slice(0, 9).join("") -> "Family π¨β" ends with U+200D
[..."Trip πΉπ·π©πͺ planned"].slice(0, 6).join("") -> "Trip πΉ"
[..."Cafe\u0301 au lait"].slice(0, 4).join("") -> "Cafe"
Nothing throws, and every result is valid Unicode. Every result is also different text from what the user wrote. The skin tone is gone, a family turned into one man, a flag turned into the letter T, and "CafΓ©" turned into "Cafe". The unit a reader sees is the grapheme cluster from Unicode Standard Annex #29, and that is the only safe place to cut.
3. Cut on graphemes, measure in the budget's unit
The fix keeps two things apart: where you may cut (between graphemes) and how you count (whatever unit the limit uses). In JavaScript, Intl.Segmenter gives graphemes:
const seg = new Intl.Segmenter(undefined, { granularity: 'grapheme' });
export function truncate(text, max, { ending = 'β¦', measure = s => s.length } = {}) {
if (measure(text) <= max) return text; // fits: no ending
const budget = max - measure(ending); // the ending counts toward the limit
let out = '';
for (const { segment } of seg.segment(text)) {
if (measure(out + segment) > budget) break; // stop before the grapheme that doesn't fit
out += segment;
}
return out.trimEnd() + ending;
}
The results, with no ending and then with β¦ (one code unit):
truncate("Great job ππ½ team", 12, {ending:""}) -> "Great job" | with β¦: "Great jobβ¦"
truncate("Trip πΉπ·π©πͺ planned", 9, {ending:""}) -> "Trip πΉπ·" | with β¦: "Tripβ¦"
truncate("Family π¨βπ©βπ§ photo", 12, {ending:""}) -> "Family" | with β¦: "Familyβ¦"
truncate("Cafe\u0301 au lait", 5, {ending:""}) -> "CafΓ©" | with β¦: "Cafβ¦"
Look at the flag row. With a limit of 9 and no ending, "Trip πΉπ·" fits exactly. With β¦, the budget drops to 8, so the Turkish flag no longer fits and the whole flag goes. That is correct: the result never goes over the limit and never contains a piece of a character. The result can be a few units shorter than the limit. A family emoji is 8 code units, so it either fits completely or it is left out.
Python's standard library has no grapheme segmentation, but the third-party regex module matches one grapheme with \X:
import regex
def truncate(text, max_units, ending='β¦', measure=len):
if measure(text) <= max_units:
return text
budget, out = max_units - measure(ending), ''
for g in regex.findall(r'\X', text):
if measure(out + g) > budget:
break
out += g
return out.rstrip() + ending
truncate("Trip πΉπ·π©πͺ planned", 7, '') gives "Trip πΉπ·", and truncate("Cafe\u0301 au lait", 4, '') gives "Caf". (Python's len counts code points. Pass a different measure if your limit is in UTF-16 units, bytes or tokens.)
I also ran the same strings through the FindUtils Text Truncate tool in Chrome, with Break on word off and no ending. It never split a character either: limit 11 or 12 on the thumbs-up string gave "Great job " (10), the family string gave "Family " (7) at limits 10 and 12, and "Cafe\u0301" at limit 4 gave "Caf". It keeps the trailing space, while the function above trims it. The Text Truncate guide covers the tool's settings. It is a quick way to get expected values for your own unit tests.
4. Byte budgets: the naive cut can come out longer
Byte limits are common: a column sized in bytes, a header, a push payload, a key-value store with a value size limit. The obvious approach is to encode, cut the bytes and decode:
text = "Family π¨βπ©βπ§ photo from KrakΓ³w" (44 UTF-8 bytes)
Buffer.from(text).subarray(0, 10).toString() -> "Family οΏ½" 10 bytes
Buffer.from(text).subarray(0, 22).toString() -> "Family π¨βπ©βοΏ½" 24 bytes (!)
truncate(text, 22, {ending: "", measure: utf8Bytes}) -> "Family" 6 bytes
The second line is the surprise. The cut landed inside the third person (an emoji is 4 bytes in UTF-8), Node replaced the 2 leftover bytes with οΏ½, and οΏ½ itself is 3 bytes. Re-encoded, the "22-byte" result is 24 bytes, which is over the limit you were trying to meet.
Python is stricter by default and fails loudly. Its lenient options still produce wrong text:
b[:10].decode("utf-8") UnicodeDecodeError
b[:10].decode("utf-8", "ignore") "Family "
b[:22].decode("utf-8", "ignore") "Family π¨βπ©β" two people and a dangling joiner
b[:22].decode("utf-8", "replace") "Family π¨βπ©βοΏ½"
errors="ignore" is a popular answer to "how do I cut UTF-8 safely", and it does give valid UTF-8. It doesn't give whole characters. The grapheme loop with a byte measure (s => Buffer.byteLength(s, 'utf8') in Node, lambda s: len(s.encode('utf-8')) in Python) returns "Family" here, because the family emoji alone is 18 bytes and doesn't fit in the remaining 15. A shorter result is the right answer. If you want to check a byte count by hand, the URL Encoder shows each UTF-8 byte as a %XX group.
5. Token budgets: the same problem, one layer up
Language-model budgets are in tokens, and tokenizers work on bytes. With OpenAI's tiktoken and the o200k_base encoding, the token pieces show where the boundaries fall:
"Great job ππ½ team" 6 tokens: 'Great' | ' job' | ' π' | f0 9f 8f | bd | ' team'
"Family π¨βπ©βπ§ photo" 10 tokens: the family takes 8 of them, each person split across two
The skin-tone modifier is split across two tokens, so keeping the first four tokens and decoding gives half a character:
enc.decode(ids[:4]) -> "Great job ποΏ½"
enc.decode(ids[:2]) -> "Family οΏ½" (family string)
enc.decode(ids[:8]) -> "Family π¨βπ©βοΏ½"
decode replaces the broken bytes by default, so this doesn't fail. It sends οΏ½ to the model, or stores it in a summary. In a Turkish sample ("Merhaba, nasΔ±lsΔ±nΔ±z? Γok gΓΌzel!", 11 tokens) every token happened to end on a whole character. A test on a few English and Turkish strings will pass, and emoji are what break it.
The same grapheme loop works for tokens if measure counts tokens. There is one more catch, which I measured. The token count of a prefix isn't monotonic: across 202 grapheme prefixes of a mixed sample, a longer prefix had fewer tokens than the one before it 9 times, because the tokenizer merges pieces differently once more text follows. In practice, measure the final candidate together with its ending rather than adding up per-grapheme counts:
def truncate_tokens(text, max_tokens, ending='β¦'):
if len(enc.encode(text)) <= max_tokens:
return text
gs = regex.findall(r'\X', text)
lo, hi = 0, len(gs) # binary search over grapheme prefixes
while lo < hi:
mid = (lo + hi + 1) // 2
if len(enc.encode(''.join(gs[:mid]).rstrip() + ending)) <= max_tokens:
lo = mid
else:
hi = mid - 1
return ''.join(gs[:lo]).rstrip() + ending
Budgets of 6, 7 and 9 tokens on "Great job ππ½ team, see you tomorrow" gave "Great job ππ½β¦" (6), "Great job ππ½ team,β¦" (7) and "Great job ππ½ team, seeβ¦" (9). For budgets 6 to 9, a slow linear scan over every prefix picked the same cut each time. Because the count isn't strictly monotonic, a binary search can in principle stop one grapheme short of the best cut. It can never return something over the budget, since every candidate is measured with its ending.
6. A performance trap in Node 20
Intl.Segmenter is the right tool, but on Node 20 it slows down badly on long strings. I timed iterating every grapheme of the same repeated sentence:
code units Node 20.19.2 Node 22.23.4 Node 24.22.1 Chrome 156
10,000 20 ms 3.7 ms 1.0 ms 0.7 ms
40,000 195 ms 9.5 ms 2.4 ms 2.2 ms
80,000 8,660 ms 17.1 ms 8.8 ms 3.8 ms
The jump from 40,000 to 80,000 units on Node 20 was the same in two runs (8.9 s and 8.7 s). Iterating the same 80,000-unit string by code point took under 1 ms. The truncate above only walks as far as the limit, so cutting a 160,000-unit string to 280 took 50 ms even on Node 20. What hurts is code that segments the whole string first, for example to count graphemes or to cut from the end. Node 20 reached end-of-life on 30 April 2026 according to the Node.js release schedule, so this is one more reason to move to 22 or 24.
A checklist for any cut
- Name the unit of the limit: UTF-16 units (
maxlength, most UIs), code points (Python, MySQLCHAR_LENGTH), bytes (storage, headers, payloads) or tokens (models). - Cut between graphemes, always:
Intl.Segmenterin JavaScript,regex\Xin Python. Never callslice,subarrayordecode(errors="ignore")on user text. - Measure the candidate with its ending, and let the ending count toward the limit.
- Expect results shorter than the limit. A family emoji is 8 units, 18 bytes or several tokens, and it fits whole or not at all.
- Guard boundaries. Use
isWellFormed()before sending JSON to a stricter service. Normalize to NFC first if two visually identical strings should get the same cut. - Test with the four strings above: skin tone, flag, ZWJ family, combining accent. Plain English text will pass any implementation.
- Don't segment huge strings on Node 20. Stop at the limit, or upgrade.
For quick checks while you write tests, the Character Counter counts the way JavaScript does and the Word Counter adds reading time for longer drafts.
Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including adding an AI feature to your web app, such as a chat assistant, document analysis or search.
About the author
- 463 free browser tools. No sign-up, no install. https://findutils.com
Comments
Join the discussionNo comments yet. Be the first to comment.
