I wanted to see how much can change in an ordinary conversion, so I wrote one small, realistic config and pushed it through the tools people actually use: five YAML parsers (Python, JavaScript and Go), a TOML writer, and six .env loaders. Everything below was run on a Linux box this week. The outputs are copied from the terminal, not reconstructed from documentation.

The test file

This is a fictional shop service. Every line is something I've seen in real configs:

# shop service config (fictional)
app:
  name: shop
  version: 1.10
  country: NO
  debug: off
  launch_date: 2026-01-01
  timeout: 1:30
  file_mode: 0755
  zip: 01234
  port: "8080"
  sentry_dsn: ~
defaults: &defaults
  retries: 3
  region: eu-west-1
worker:
  <<: *defaults
  queue: emails

What a human means: version "1.10", Norway, debug turned off, a launch date, a timeout of one minute thirty, Unix permissions 0755, a ZIP code with a leading zero, port 8080, and no Sentry DSN yet.

Step 1: what each YAML parser actually read

I loaded the file with PyYAML 6.0.3, ruamel.yaml 0.19.1, js-yaml 4.3.2, yaml 2.9.1 (default mode and version: '1.1') and Go's gopkg.in/yaml.v3 v3.0.1, then printed each value with its type.

PyYAML follows YAML 1.1:

PyYAML 6.0.3
  version      1.1                          float
  country      False                        bool
  debug        False                        bool
  launch_date  datetime.date(2026, 1, 1)    date
  timeout      90                           int
  file_mode    493                          int
  zip          668                          int
  port         '8080'                       str
  sentry_dsn   None                         NoneType

ruamel.yaml follows YAML 1.2:

ruamel.yaml 0.19.1
  version      1.1                          float
  country      'NO'                         str
  debug        'off'                        str
  timeout      '1:30'                       str
  file_mode    755                          int
  zip          1234                         int

Put side by side, the same file gives these results:

• country: NO. It is False in PyYAML and in yaml with version 1.1. It is "NO" in ruamel, js-yaml, yaml default mode and Go yaml.v3.
• debug: off. It is False under 1.1 rules and the string "off" under 1.2. If your app checks if config.debug:, the string "off" is truthy, so debug is on.
• timeout: 1:30. PyYAML reads this as a base-60 number: 90. The 1.2 parsers keep the string "1:30".
• file_mode: 0755. It is 493 (octal) in PyYAML, in yaml 1.1 mode and also in Go yaml.v3. It is 755 (decimal) in ruamel, js-yaml and yaml default. That's three different meanings of the same four characters across popular libraries.
• zip: 01234. It is 668 (octal again) in PyYAML and Go, and 1234 everywhere else. No parser kept the leading zero.
• version: 1.10. It is 1.1 in every single parser. It's a float, so the trailing zero is gone before conversion even starts.
• launch_date. It is a real date in Python, a time.Time in Go, a JavaScript Date (midnight UTC) in js-yaml, and the plain string "2026-01-01" in yaml default mode.
• port: "8080". It's quoted, so it's a string everywhere. Quoting was the only thing that behaved the same in all of them.

The merge key (<<: *defaults) was expanded by all five, so worker ended up with retries, region and queue. That's good to know, but it also means the anchor is gone after conversion. If you edit defaults later, the TOML copy won't follow.

Lesson one: the conversion bug is often already in the parse step. By the time any converter sees your data, NO may already be false.

Step 2: writing TOML

Next I passed both parsed results to tomli_w 1.2.0, a strict TOML writer:

tomli_w.dumps as-is -> TypeError: Object of type 'NoneType' is not TOML serializable

TOML has no null. That's the one loud failure in this whole experiment, and it's a useful one. You have to decide what sentry_dsn: ~ should become: drop the key, use an empty string, or comment it out. Each choice means something different to the app reading it. "Key missing" and "key set to empty" often take different code paths.

After dropping the null, the PyYAML path produced:

[app]
name = "shop"
version = 1.1
country = false
debug = false
launch_date = 2026-01-01
timeout = 90
file_mode = 493
zip = 668
port = "8080"

The ruamel path produced the same file except country = "NO", debug = "off", timeout = "1:30", file_mode = 755 and zip = 1234. Both wrote the date as a real TOML local date, and reading it back with Python's tomllib returned datetime.date(2026, 1, 1).

Both files are valid TOML, and both are wrong in different places. Nothing warned me.

Step 3: diff the two TOML files

This is the step that actually catches it. I pasted the ruamel output as file A and the PyYAML output as file B into the FindUtils TOML Diff and pressed Compare. The summary read Added 0, Removed 0, Changed 5, Unchanged 9:

app.country    "NO"   -> false
app.debug      "off"  -> false
app.timeout    "1:30" -> 90
app.file_mode  755    -> 493
app.zip        1234   -> 668

That's exactly the five values the YAML version changed, and nothing else. A text diff of the same two files would have flagged the lines too, but it can't tell you whether "NO" → false is a type change or a value change, and it gets noisy as soon as the key order or whitespace differs. A structural diff compares parsed values by path, which is the question you actually care about after a conversion.

If you want the walkthrough, there's a TOML Diff guide.

My routine now: convert with the parser you'll run in production, convert again with a different one, and diff the two. Where they disagree, quote the value in the source YAML.

Step 4: flattening to .env, where quoting is a dialect

The last hop in a lot of pipelines is .env: Docker, Laravel, Node, Go services, CI variables. People talk about "the .env format" as if there were one. I wrote eight one-line files, each setting a single variable V, and loaded them with six loaders: python-dotenv 1.2.4, Node dotenv 17.4.2 (alone, and with dotenv-expand 12.0.3), Go godotenv 1.5.1, vlucas/phpdotenv 5.8.0 (the loader Laravel uses), and plain bash set -a; . ./file. HOME_DIR=/srv was set for the expansion cases.

Here are the results:

Case 1, unquoted V=p@ss$word#1:

• p@ss$word#1 in python-dotenv and godotenv
• p@ss$word in Node dotenv and phpdotenv (the #1 was read as a comment)
• p@ss in Node with dotenv-expand ($word expanded to nothing)
• p@ss#1 in bash

That's one password and four different values. Your app would fail to log in with four of the six loaders.

Case 2, double-quoted V="p@ss$word#1": the full value in python-dotenv, Node dotenv, godotenv and phpdotenv. dotenv-expand and bash still expand $word and give p@ss#1.

Case 3, single-quoted V='p@ss$word#1': the full value in every loader except Node with dotenv-expand, which returned p@ss#1. Single quotes protected the value from bash, but not from dotenv-expand. This one surprised me the most.

Case 4, V=secret #1: secret everywhere. A space before # starts a comment in all six.

Case 5, V=hello world: hello world in four loaders. phpdotenv refused the file with Encountered unexpected whitespace at [hello world], and bash tried to run world as a command.

Case 6, V=${HOME_DIR}/data: /srv/data everywhere except plain Node dotenv, which keeps it literal unless you add dotenv-expand.

Case 7, V=$HOME_DIR/data (no braces): this is expanded by godotenv, dotenv-expand and bash, but left as the literal $HOME_DIR/data by python-dotenv and phpdotenv. If you write the variable without braces, Python and PHP services won't expand it.

Case 8, V="line1\nline2": a real newline in all five dotenv libraries, and a literal backslash-n in bash.

What I do now, based on these results:

1. Double-quote any value that contains $, #, spaces or a backslash.
2. If your stack runs dotenv-expand, a literal $ in a secret needs escaping even inside single quotes. Better still, generate secrets without $.
3. Always write ${VAR} with braces if you want expansion. Never rely on bare $VAR.
4. Don't source a .env file in a shell script and expect it to match what your app sees.

Step 5: diff the .env files, not just the YAML

Once there's a .env on each side, say .env.example and the generated .env, compare them by parsed value, not by text. I pasted these two files into the FindUtils Env Diff:

# File A
APP_NAME="Shop"
APP_VERSION=1.10
DB_PASSWORD=p@ss$word#1
SMTP_FROM=shop@example.com #ops
QUEUE=emails

# File B
APP_NAME='Shop'
APP_VERSION=1.1
DB_PASSWORD="p@ss$word#1"
SMTP_FROM="shop@example.com"
REGION=eu-west-1

It reported 1 only in B (REGION), 1 only in A (QUEUE), 1 changed (APP_VERSION 1.10 → 1.1) and 3 unchanged. The differences in quoting didn't count as changes. "Shop" and 'Shop' are equal, and SMTP_FROM with a trailing #ops comment equals the quoted address. Its page says the same thing: values are compared after dotenv rules are applied, and it doesn't expand ${VAR} references. Secret-looking keys such as DB_PASSWORD are masked by default, so you can paste a real file and share the report.

One caveat that matches the table above: the tool treats the unquoted DB_PASSWORD=p@ss$word#1 as equal to the quoted one, which is how python-dotenv and godotenv read it. Node dotenv and phpdotenv would cut it at the #. So a clean Env Diff tells you that both files say the same thing. It can't promise every loader reads it the same way. That's why rule 1, quote anything with # or $, is still worth following. See the Env Diff guide for the options (masking, ignore key case, duplicate keys).

A checklist you can paste into a PR

FindUtils already has a longer config conversion review checklist and a piece on what survives a config format conversion. Here is the short version, shaped by what this experiment broke:

• Quote in the source. "NO", "off", "1:30", "0755", "01234", "1.10". If a value is an identifier, a code, a version or a mode, it is a string.
• Know your parser's YAML version. PyYAML is 1.1. Go yaml.v3 reads 0755 as octal. js-yaml and ruamel are 1.2.
• Decide what null becomes before the TOML writer decides for you, or crashes.
• Check dates. Is it a date, a midnight-UTC timestamp, or a string? Each one serializes differently downstream.
• Anchors are expanded on conversion. The TOML or .env copy is a snapshot.
• Run two parsers and diff the results with TOML Diff. Anything in "Changed" needs a quote or a comment.
• Diff .env files by value with Env Diff before deploying, and quote anything containing $, #, spaces or \.

Reproduce it

The setup is small: one YAML file, one script per language, and a folder of eight .env files. The versions I used were PyYAML 6.0.3, ruamel.yaml 0.19.1, tomli_w 1.2.0, python-dotenv 1.2.4, js-yaml 4.3.2, yaml 2.9.1, dotenv 17.4.2, dotenv-expand 12.0.3, gopkg.in/yaml.v3 v3.0.1, godotenv 1.5.1 and vlucas/phpdotenv 5.8.0, with Node 20, Go 1.27 and PHP 8.4. Pin your own versions. YAML libraries change their defaults more often than you'd expect.

The point isn't that any one library is wrong. Each one does what its spec or its README says. The problem is that a config passes through three or four of them on the way to production, and every hop can quietly change a meaning without an error. Two diffs take a minute and catch all of it.

Need help with a project? The team behind FindUtils takes on freelance work: hire us on Upwork. If your Laravel app needs an admin panel whose settings screens read and write config safely, see our Laravel Livewire admin panel service.

let remaining = 25 * 60;
setInterval(() => {
  remaining--;
  render(remaining);
}, 1000);

It looks right on screen. Then someone starts a 25-minute focus session, switches to their editor, comes back, and the timer still shows several minutes to go after the real 25 minutes have passed. The code assumes the callback runs once per second. The browser never promised that, and in a background tab it often doesn't.

I wanted numbers instead of a vague "background tabs are throttled", so I measured it. Everything below was run on 27 October 2026 in Google Chrome 156.0.8078.47 on Linux. It was a real browser window with default settings, not headless. Each test page was hidden by opening a new tab in front of it. A small script read the page's variables once a minute over the DevTools protocol.

The test page: five countdowns started at the same moment

const DURATION = 25 * 60;
const start = Date.now();
const end = start + DURATION * 1000;

// 1. naive: subtract one second per tick
let naive = DURATION;
setInterval(() => { naive--; }, 1000);

// 2. deadline: recompute from the clock on every tick
let deadline = DURATION;
setInterval(() => {
  deadline = Math.max(0, Math.round((end - Date.now()) / 1000));
}, 1000);

// 3. chained setTimeout, naive
let chained = DURATION;
(function tick() {
  setTimeout(() => { chained--; tick(); }, 1000);
})();

// 4. a requestAnimationFrame counter: does it run at all while hidden?
let frames = 0;
(function frame() { frames++; requestAnimationFrame(frame); })();

A second version of the page also drove a naive countdown from a dedicated Web Worker. A third left out a line that wrote the time into document.title every second. That line turned out to matter a lot.

Result 1: eight minutes in a background tab

These are the readings from the page without a worker, hidden 6 seconds after it loaded:

hidden for   real time left   naive    chained setTimeout   deadline
1 min        23:53            23:54    23:57                23:54
3 min        21:53            21:54    22:04                21:54
8 min        16:53            17:23    17:47                16:54

After eight minutes the naive countdown was 30 seconds behind and the chained setTimeout was 54 seconds behind. The deadline version was right; its one-second difference comes from rounding and the moment of its last tick. When I brought the tab back to the front, the two tick-counting versions stayed wrong. Nothing in their code knows how much time has really passed.

requestAnimationFrame stopped completely. The frame counter did not move once while the tab was hidden. Anything that updates a countdown from an animation loop simply freezes in the background.

On a different run, a page that also ran a worker was hidden for ten minutes. There the naive countdown ended 45 seconds behind and the chained one 80 seconds behind. The exact amount changes from run to run. The direction never does.

Why the chained setTimeout drifts from the first minute

The log of callback times explains the two kinds of drift. In the first five minutes the chained setTimeout had 17 gaps of 1,997 ms instead of 1,000 ms. The setInterval had none.

Chrome's article on timer throttling in Chrome 88 describes the first stage. When a page is hidden, timers are put in a group that "the browser will check … once per second". Timers with similar timeouts get batched together. A setInterval stays in step with those checks. A chained setTimeout asks for 1,000 ms from the moment its callback ran, which is slightly after a check. Now and then it just misses one and waits for the next, so a second goes missing. Those missed seconds make up most of the 11-second lag at the three-minute reading in the table.

Why both fall behind after five minutes

The setInterval log shows a single 30,000 ms gap at the five-minute mark. That is the second stage, which the same Chrome article calls intensive throttling. It applies when all of these are true:

• the page has been hidden for more than 5 minutes,
• the chain count is 5 or greater,
• the page has been silent for at least 30 seconds,
• WebRTC is not in use.

Then "the browser will check timers in this group once per minute". The article also defines the chain: with setInterval, "each iteration is part of the chain". A once-a-second interval reaches a chain count of 5 after five seconds and stays there for good.

So the naive countdown lost 30 seconds in one go. But why did it then go back to ticking once per second, instead of losing most of every minute?

Result 2: the line that wrote to document.title

The answer is in Chromium's page scheduler. When a page updates its title or favicon, intensive throttling is inhibited for same-origin frames for 3 seconds. The comment says this "enables alternating effects meant to grab the user's attention". My page updated the title every second, so each update kept the next one-second tick out of intensive throttling. Only the first tick after the five-minute mark had to wait for a once-a-minute check.

To test this, I ran the page with and without the title line side by side. I used the flag from the Chrome article, --enable-features=IntensiveWakeUpThrottling:grace_period_seconds/10, which starts intensive throttling 10 seconds after the page is hidden instead of 5 minutes. After about four minutes (247 s):

                                  interval ticks   naive shows   real time left
page that updates document.title  211              21:29         20:53
page that never touches it        62               23:58         20:53

Without the title update, the page got 62 interval callbacks in 247 seconds. The naive countdown was more than three minutes behind. That version's deadline countdown read the clock correctly every time it ran. But it ran only once a minute, so its stored value lagged by up to a minute. On the next tick after the tab came back it was correct again.

So whether a background countdown is 30 seconds off or three minutes off depends on whether the page happens to write its title. That makes a poor foundation for anything you rely on.

Result 3: a long task does the same thing in a visible tab

Throttling is not the only way to lose ticks. In a visible tab, I blocked the main thread for 5 seconds with a busy loop, the kind of long task that heavy rendering or a large JSON.parse can cause. The interval gaps were:

setInterval gaps (ms):     1000, 1000, 5224, 775
worker message gaps (ms):  1000, 1000, 5214, 0, 0, 0, 0, 786

setInterval does not make up for callbacks it missed. It ran once after the block and carried on, and the naive countdown was 4 seconds behind (24:55 against a real 24:51). The worker kept ticking during the block. Its messages queued up and arrived together when the main thread was free, so the worker-driven count stayed correct.

In my hidden-tab runs the worker's one-second interval was not throttled either: one message a second for ten minutes, with the longest gap 1,008 ms. That is an observation from one browser, not a documented guarantee. A worker doesn't help if what you need is the screen to repaint or a sound to play.

Result 4: the alarm at the end

A countdown has two jobs: show the time left, and do something when it reaches zero. I tested the second job on its own. The tab was hidden, intensive throttling started after 10 seconds (the same flag), and I set a 90-second alarm three ways:

const due = Date.now() + 90000;

// A. one long setTimeout (chain count 1)
setTimeout(fire, 90000);

// B. a 1-second polling loop with chained setTimeout
(function poll() {
  setTimeout(() => Date.now() >= due ? fire() : poll(), 1000);
})();

// C. setInterval polling every second
const id = setInterval(() => {
  if (Date.now() >= due) { clearInterval(id); fire(); }
}, 1000);

A. single setTimeout    790 ms late
B. chained polling      38,791 ms late
C. setInterval polling  38,791 ms late

The single setTimeout has a chain count of 1. It never qualifies for intensive throttling, only the once-per-second check. The polling loops reach a chain count of 5 within seconds and get the once-a-minute check. Both fired at the same minute-aligned moment, almost 39 seconds after the deadline.

A countdown that keeps time

Put together, the fix is short:

function startCountdown(seconds, { render, done }) {
  const end = Date.now() + seconds * 1000;
  const left = () => Math.max(0, Math.ceil((end - Date.now()) / 1000));

  // Display: recompute from the clock, never count ticks
  const id = setInterval(() => render(left()), 1000);

  // Repaint at once when the user comes back
  const onVisible = () => {
    if (document.visibilityState === 'visible') render(left());
  };
  document.addEventListener('visibilitychange', onVisible);

  // The end: one timer, not a polling loop
  const alarm = setTimeout(() => {
    clearInterval(id);
    document.removeEventListener('visibilitychange', onVisible);
    render(0);
    done();
  }, seconds * 1000);

  render(left());
  return () => {
    clearInterval(id);
    clearTimeout(alarm);
    document.removeEventListener('visibilitychange', onVisible);
  };
}

What each part is for:

• The display reads the clock. The interval is only a request to repaint. If ticks are dropped, batched or delayed, the next one still shows the right number. Pausing works the same way: save the time left when the user pauses, and set a new end when they resume.
• visibilitychange repaints at once. With a throttled interval, a hidden tab can show a stale value for up to a minute. The event fixes the display as soon as the user looks.
• The end is a single setTimeout. In my test it fired within a second. A polling loop fired 39 seconds late.
• Don't count frames. requestAnimationFrame is fine for smooth animation while visible, but it stops while hidden.

Date.now() follows the system clock, so it jumps if the user or NTP changes the clock in the middle of a countdown. For a focus timer that is rarely a problem, and it is the clock the user compares against anyway.

Other browsers

Chrome is not the strictest. MDN's setTimeout page says that "Firefox Desktop has a minimum timeout of 1 second for inactive tabs" and that "Firefox for Android has a minimum timeout of 15 minutes for inactive tabs and may unload them entirely." An unloaded tab runs no code at all. If a timer has to survive that, the end time needs to live somewhere that outlasts the page, such as localStorage, so a reloaded tab can work out where it should be. Anything that must alert the user at an exact moment on a phone is a job for a native notification, not a browser tab.

Testing your own timer in two minutes

You don't need to wait five minutes per experiment:

1. Start Chrome with --enable-features=IntensiveWakeUpThrottling:grace_period_seconds/10.
2. Start your countdown next to a stopwatch that reads the clock. The FindUtils Stopwatch does. In my run its display only updated once a minute while hidden, but it showed the correct elapsed time as soon as the tab was back in front.
3. Switch to another tab for two or three minutes, then come back and compare the two.
4. Repeat with a page that doesn't change its title. That is the version that shows the real worst case.

If you store end times as timestamps, the Unix Timestamp Converter turns a Date.now() value like 1793079000000 into a readable date and back. The post on seconds versus milliseconds covers what goes wrong when a millisecond value like this one meets code that expects seconds.

Summary

• setInterval(fn, 1000) means "about once a second while the page is visible and idle". Anything else is a bonus.
• Hidden for eight minutes, counting ticks put a countdown 30 seconds behind. A chained setTimeout was 54 seconds behind. Without a title update, the gap after four minutes was more than three minutes.
• Compute the time left from a deadline, repaint on visibilitychange, and use one setTimeout for the end.
• Test with the grace-period flag, and test the version of your page that doesn't touch document.title.

Need help with a project? The team behind FindUtils takes on freelance work. You can find us on Upwork. We also build React Native apps for iOS and Android connected to your web app, where the same JavaScript timer code has to survive the app going to the background.

Every output below was produced on 26 October 2026 with Node 20.19.2 (ICU 76.1), Python 3.13.5 with the regex package (2026.9.29), PHP 8.4.27 with the intl and mbstring extensions, Go 1.27.3 with rivo/uniseg v0.4.7 and golang.org/x/text v0.43.0, and Chrome 156. Five test strings come up again and again:

"👍🏽 ok"     thumbs-up + skin-tone modifier, a space, two letters
"noël"       written as n o e + combining diaeresis (U+0308) + l
"🇸🇪"         the Swedish flag: regional indicators S and E
"👨‍👩‍👧"         man, woman, girl joined by two zero-width joiners
"été"        first é precomposed (U+00E9), last é = e + combining acute (U+0301)

The last one looks contrived, but it is ordinary. Text pasted from different sources, or typed on different keyboards, can mix the two forms of é in one string.

1. What does s[0] give you?

The same expression returns four different things, because each language indexes a different unit:

                 JavaScript s[0]   Python s[0]   PHP $s[0]   Go s[0]
"👍🏽 ok"          "\ud83d"          '👍'          byte f0     byte 0xf0
"🇸🇪"             "\ud83c"          '🇸' (S)      byte f0     byte 0xf0
"été"            "é"               'é'           byte c3     byte 0xc3

JavaScript indexes UTF-16 code units, so the first item of "👍🏽 ok" is half of a surrogate pair. Python indexes code points, so you get a whole thumbs-up without its skin tone. PHP and Go index bytes of UTF-8, so you get the first byte of a four-byte or two-byte sequence. Lengths disagree for the same reason:

"👍🏽 ok"   JS .length 7 | Python len 5 | PHP strlen 11, mb_strlen 5, grapheme_strlen 4 | Go len 11, RuneCount 5, uniseg 4
"🇸🇪"      JS .length 4 | Python len 2 | PHP strlen 8,  mb_strlen 2, grapheme_strlen 1 | Go len 8,  RuneCount 2, uniseg 1
"été"     JS .length 4 | Python len 4 | PHP strlen 6,  mb_strlen 4, grapheme_strlen 3 | Go len 6,  RuneCount 4, uniseg 3

A reader sees 4, 1 and 3 characters. Those are grapheme clusters, defined in Unicode UAX #29. Only the grapheme count matches what's on screen, and it is the one no language gives you by default.

Go's for range loop is a useful reminder of what an index is. Over "été" (both é precomposed here) it yields i=0 é, i=2 t, i=3 é. The index is a byte offset, so it jumps from 0 to 2.

2. Four ways to reverse a string, and which ones break

Here are the usual one-liners, run on the test strings. "Code points" means [...s].reverse() in JavaScript, s[::-1] in Python, mb_str_split + array_reverse in PHP and a reversed []rune in Go. All four gave identical results.

input       code units / bytes             code points       graphemes
"👍🏽 ok"     JS: "ko \udffd\ud83c\udc4d\ud83d"  "ko 🏽👍"           "ko 👍🏽"
"noël"      JS: "l\u0308eon"  (l̈eon)        "l̈eon"            "lëon"
"🇸🇪"        JS: "\uddea\ud83c\uddf8\ud83c"    "🇪🇸"  (Spain)     "🇸🇪"
"👨‍👩‍👧"        JS: lone surrogates             "👧‍👩‍👨"            "👨‍👩‍👧"
"été"       JS: "\u0301eté"                 "\u0301eté"       "été"

Reversing code units (split('').reverse().join('') in JavaScript) swaps the halves of each surrogate pair. Three of the five results fail isWellFormed(). Reversing bytes is worse: PHP's strrev and a reversed []byte in Go produced invalid UTF-8 for all five inputs, including "noël" and "été", which contain no emoji at all. The PHP manual for strrev describes it as reversing a string, and it does, byte by byte.

Reversing code points avoids broken encoding. The text is still wrong:

• The skin tone now comes before the thumbs-up, so it shows on its own.
• The diaeresis moved from the e to the l ("l̈eon").
• The family became girl, woman, man. That order isn't in Unicode's list of recommended ZWJ sequences (man, woman, girl is), so you can't count on fonts drawing it as one family.
• In "été" the combining accent now starts the string, with nothing to sit on.
• The Swedish flag became the Spanish flag. Regional indicators S + E make 🇸🇪 and E + S make 🇪🇸, so the reversal still looks valid. Nothing throws, and nothing looks broken.

Only the grapheme column is what a reader would call "the same text backwards".

3. Palindrome checks fail in both directions

A palindrome check is s === reverse(s), so it inherits every problem above. It can fail in two ways: it can say yes to a string that isn't a palindrome, or no to one that is.

                         code units   code points   graphemes   NFC + graphemes
"racecar"                true         true          true        true
"🇸🇪🇪🇸" (Sweden, Spain)   false        true          false       false
"👍🏽👍🏽"                   false        false         true        true
"été" (mixed forms)      false        false         false       true
"😀x😀"                   false        true          true        true

The flag pair is a false positive. Its code points are S, E, E, S, which is a palindrome, but on screen it is Sweden then Spain, and backwards it is Spain then Sweden. Python's s == s[::-1] returned True for it, and so would any check that works on code points.

"👍🏽👍🏽" and "😀x😀" are false negatives. They read the same in both directions, but code units (and, for the skin tones, code points) don't.

The mixed "été" shows that graphemes alone aren't enough. The first é is one code point and the last is two, so the clusters differ until you normalize. After normalize('NFC'), both become U+00E9 and the check passes. Normalize before you compare, always.

4. Sentences: punctuation, case and the Turkish İ

Real palindrome checkers strip spaces and punctuation and ignore case. "A man, a plan, a canal: Panama" passed in every language once that was done. The Turkish classic "Ey Edip Adana'da pide ye" did too. In capitals it gets interesting:

"İ".toLowerCase()            -> "i\u0307"   (i + combining dot above), length 2
"İ".toLocaleLowerCase("tr")  -> "i"
Python "İ".lower()           -> 'i\u0307'   ("İ".casefold() is the same)
PHP mb_strtolower("İ")       -> 69 cc 87    (i + U+0307 in UTF-8)
Go strings.ToLower("İ")      -> "i"

JavaScript, Python and PHP apply Unicode's full default mapping from SpecialCasing.txt, which turns İ (dotted capital I) into a plain i followed by a combining dot above. Go's strings.ToLower uses the simple one-to-one mapping and returns "i". Only the Turkish and Azerbaijani rules map İ to i on purpose. So the same uppercase input gives different lowercase strings in different languages. Here is what that did to "EY EDİP ADANA'DA PİDE YE" in JavaScript:

drop spaces + punctuation:   "eyedi\u0307padanadapi\u0307deye"   code points: false   graphemes: true
keep only \p{L} and \p{N}:   "eyedipadanadapideye"               code points: true    graphemes: true
Turkish locale lowercase:    "eyedipadanadapideye"               code points: true

The first filter leaves the two combining dots in place. Reversing code points moves each dot onto the wrong letter, and a correct palindrome fails. The second filter happens to work, because \p{L} doesn't match the dot (it is a mark, \p{M}), but that is luck. A filter that keeps only letters also deletes every emoji, so "🇸🇪🇪🇸" turns into an empty string and passes as a palindrome. Grapheme comparison handled all of these correctly, with or without the Turkish locale. Use the locale when you know it, and compare graphemes either way.

The same default shows up in tools. The FindUtils Case Converter lowercased "EDİP" to "edi̇p" (five code points) in Chrome. That is the Unicode default, and it's worth knowing if you paste Turkish text into any converter. The case converter guide covers the other case styles.

5. The two-pointer loop, fixed

The classic version walks two indexes towards the middle. Over JavaScript code units, it returned false for "👍🏽👍🏽" and "😀x😀". On "😀x😀" the first comparison is \ud83d against \ude00: the left pointer sees the first half of an emoji and the right pointer sees the second half. The loop itself is fine. The array it walks is the wrong one. Segment first, then walk:

export function isPalindrome(text, locale = 'en') {
  const s = text.toLocaleLowerCase(locale).normalize('NFC')
                .replace(/[\s\p{P}]/gu, '');   // drop spaces + punctuation, keep emoji and marks
  const g = Array.from(new Intl.Segmenter(locale, { granularity: 'grapheme' }).segment(s), x => x.segment);
  for (let i = 0, j = g.length - 1; i < j; i++, j--) if (g[i] !== g[j]) return false;
  return true;
}

Results: the Panama sentence true; "EY EDİP ADANA'DA PİDE YE" true with both 'tr' and 'en'; mixed "été" true; "👍🏽👍🏽" and "😀x😀" true; "🇸🇪🇪🇸" false; "👍🏽👍🏿" (different skin tones) false.

The same idea in the other three languages:

import unicodedata, regex          # pip install regex; \X = one grapheme cluster
def is_palindrome(text):
    s = unicodedata.normalize('NFC', text.casefold())
    s = regex.sub(r'[\s\p{P}]', '', s)
    g = regex.findall(r'\X', s)
    return g == g[::-1]

function is_palindrome(string $text): bool {          // PHP 8.4+: grapheme_str_split
    $s = Normalizer::normalize(mb_strtolower($text, 'UTF-8'), Normalizer::FORM_C);
    $s = preg_replace('/[\s\p{P}]/u', '', $s);
    $g = grapheme_str_split($s);
    return $g === array_reverse($g);
}

In Go, the standard library stops at runes. Use uniseg.NewGraphemes(s) to get the clusters, golang.org/x/text/unicode/norm for NFC, and the same two-pointer loop over the cluster strings. The Python, PHP and Go versions returned the same results as JavaScript for every test string in this section, plus "Ey Edip Adana'da pide ye". grapheme_str_split is new in PHP 8.4. On older versions, preg_split('/(\X)/u', ...) does the same job.

6. Get the expected output before you write the test

To write unit tests for a reverse or palindrome function, you need expected values from a source you trust. I checked the browser's own segmenter through the FindUtils Text Reverse tool in Chrome 156, in Characters mode:

"👍🏽 ok"   -> "ko 👍🏽"
"noël"    -> "lëon"      (diaeresis still on the e)
"🇸🇪"      -> "🇸🇪"
"👨‍👩‍👧"      -> "👨‍👩‍👧"
"🇸🇪🇪🇸"    -> "🇪🇸🇸🇪"     (Spain, Sweden: not equal to the input, so not a palindrome)

Every result matched the grapheme column above. Words and Lines modes are there too, which helps when a log or changelog comes out in the wrong order. To see how many units a string has before you index into it, the Character Counter counts like JavaScript's .length (7 for "👍🏽 ok", 4 for "🇸🇪" and for the mixed "été" when I tried it today), and the FindUtils post on why an emoji counts as 2 characters explains why the numbers differ.

A checklist for any code that walks a string

1. Know what your index counts: UTF-16 units (JavaScript, Java, C#), code points (Python), or bytes (PHP, Go, Rust, C).
2. Never reverse bytes or code units of user text. PHP's strrev is for ASCII-only data.
3. Segment into graphemes before reversing, comparing positions or taking "the first character": Intl.Segmenter, regex \X, grapheme_str_split, uniseg.
4. Normalize to NFC before comparing, and lowercase with the user's locale when you know it.
5. Filter out what you don't want (spaces, punctuation) instead of keeping only \p{L}, which silently drops emoji.
6. Put the five test strings in your test suite. A flag pair, a skin tone, a ZWJ family, a combining mark and a mixed-form accent catch every bug in this post.

Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including PHP bug fixes, a PHP 8 upgrade or new features for your website, where strlen and strrev on UTF-8 text are a common find.

Every output below was produced on 25 October 2026 with Node 20.19.2 (plus Node 22.23.4 and 24.22.1 for one timing test), Chrome 156 through Playwright, and Python 3.13.5 with the regex (2026.9.29) and tiktoken (0.14.0) packages. Four test strings come up again and again:

"Great job 👍🏽 team"     thumbs-up + skin-tone modifier
"Trip 🇹🇷🇩🇪 planned"     two flags, each made of two regional-indicator letters
"Family 👨‍👩‍👧 photo"     three people joined by two zero-width joiners
"Cafe\u0301 au lait"     e + combining accent (looks like "Café")

1. A code-unit cut can leave half a character

JavaScript's slice counts UTF-16 code units, the same unit as .length and the HTML maxlength attribute. Most emoji take two units (a surrogate pair), so a cut can land between them:

"Great job 👍🏽 team".slice(0, 11)   -> "Great job \ud83d"   isWellFormed() false
"Great job 👍🏽 team".slice(0, 12)   -> "Great job 👍"        skin tone lost
"Trip 🇹🇷🇩🇪 planned".slice(0, 7)    -> "Trip 🇹"              half a flag
"Family 👨‍👩‍👧 photo".slice(0, 10)   -> "Family 👨‍"            one person + a dangling joiner
"Cafe\u0301 au lait".slice(0, 4)     -> "Cafe"                 accent gone

The first result is the dangerous one. \ud83d is a lone surrogate: half of a pair, which is not a valid character in any encoding. JavaScript doesn't complain. What happens next depends on where the string goes:

JSON.stringify          {"t":"Great job \ud83d"}        escaped, still "valid" JSON
TextEncoder / Buffer    ... 20 ef bf bd                 silently replaced with U+FFFD (�)
fetch() request body    ... 20 ef bf bd                 same: the server receives "�"
encodeURIComponent      URIError: URI malformed         throws

So in the browser the text quietly gains a �, unless it goes into a URL, where it throws. The JSON route is worse, because the problem travels. Python accepts that JSON and fails later, somewhere else:

json.loads('{"t": "Great job \\ud83d"}')   ok, a str of length 11
t.encode("utf-8")                         UnicodeEncodeError: surrogates not allowed
sqlite3 insert of t                       UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' ...
json.dumps(t, ensure_ascii=False).encode  UnicodeEncodeError

The stack trace points at a database write in a worker, while the bug is a slice in a front end. Node 20 and current browsers have String.prototype.isWellFormed() and toWellFormed(), which are cheap guards at an API boundary. They don't fix the cut, though. They only turn half an emoji into �.

2. Code points are not enough

The usual next attempt is to cut by code points, with [...text] in JavaScript or plain slicing in Python, where strings are sequences of code points. That removes the lone surrogate, but not the problem:

[..."Great job 👍🏽 team"].slice(0, 11).join("")    -> "Great job 👍"     (Python s[:11] gives the same)
[..."Family 👨‍👩‍👧 photo"].slice(0, 9).join("")    -> "Family 👨‍"        ends with U+200D
[..."Trip 🇹🇷🇩🇪 planned"].slice(0, 6).join("")     -> "Trip 🇹"
[..."Cafe\u0301 au lait"].slice(0, 4).join("")     -> "Cafe"

Nothing throws, and every result is valid Unicode. Every result is also different text from what the user wrote. The skin tone is gone, a family turned into one man, a flag turned into the letter T, and "Café" turned into "Cafe". The unit a reader sees is the grapheme cluster from Unicode Standard Annex #29, and that is the only safe place to cut.

3. Cut on graphemes, measure in the budget's unit

The fix keeps two things apart: where you may cut (between graphemes) and how you count (whatever unit the limit uses). In JavaScript, Intl.Segmenter gives graphemes:

const seg = new Intl.Segmenter(undefined, { granularity: 'grapheme' });

export function truncate(text, max, { ending = '…', measure = s => s.length } = {}) {
  if (measure(text) <= max) return text;           // fits: no ending
  const budget = max - measure(ending);            // the ending counts toward the limit
  let out = '';
  for (const { segment } of seg.segment(text)) {
    if (measure(out + segment) > budget) break;    // stop before the grapheme that doesn't fit
    out += segment;
  }
  return out.trimEnd() + ending;
}

The results, with no ending and then with … (one code unit):

truncate("Great job 👍🏽 team", 12, {ending:""})  -> "Great job"      | with …: "Great job…"
truncate("Trip 🇹🇷🇩🇪 planned", 9, {ending:""})   -> "Trip 🇹🇷"        | with …: "Trip…"
truncate("Family 👨‍👩‍👧 photo", 12, {ending:""})  -> "Family"         | with …: "Family…"
truncate("Cafe\u0301 au lait", 5, {ending:""})    -> "Café"           | with …: "Caf…"

Look at the flag row. With a limit of 9 and no ending, "Trip 🇹🇷" fits exactly. With …, the budget drops to 8, so the Turkish flag no longer fits and the whole flag goes. That is correct: the result never goes over the limit and never contains a piece of a character. The result can be a few units shorter than the limit. A family emoji is 8 code units, so it either fits completely or it is left out.

Python's standard library has no grapheme segmentation, but the third-party regex module matches one grapheme with \X:

import regex

def truncate(text, max_units, ending='…', measure=len):
    if measure(text) <= max_units:
        return text
    budget, out = max_units - measure(ending), ''
    for g in regex.findall(r'\X', text):
        if measure(out + g) > budget:
            break
        out += g
    return out.rstrip() + ending

truncate("Trip 🇹🇷🇩🇪 planned", 7, '') gives "Trip 🇹🇷", and truncate("Cafe\u0301 au lait", 4, '') gives "Caf". (Python's len counts code points. Pass a different measure if your limit is in UTF-16 units, bytes or tokens.)

I also ran the same strings through the FindUtils Text Truncate tool in Chrome, with Break on word off and no ending. It never split a character either: limit 11 or 12 on the thumbs-up string gave "Great job " (10), the family string gave "Family " (7) at limits 10 and 12, and "Cafe\u0301" at limit 4 gave "Caf". It keeps the trailing space, while the function above trims it. The Text Truncate guide covers the tool's settings. It is a quick way to get expected values for your own unit tests.

4. Byte budgets: the naive cut can come out longer

Byte limits are common: a column sized in bytes, a header, a push payload, a key-value store with a value size limit. The obvious approach is to encode, cut the bytes and decode:

text = "Family 👨‍👩‍👧 photo from Kraków"   (44 UTF-8 bytes)

Buffer.from(text).subarray(0, 10).toString()  -> "Family �"            10 bytes
Buffer.from(text).subarray(0, 22).toString()  -> "Family 👨‍👩‍�"         24 bytes (!)
truncate(text, 22, {ending: "", measure: utf8Bytes}) -> "Family"     6 bytes

The second line is the surprise. The cut landed inside the third person (an emoji is 4 bytes in UTF-8), Node replaced the 2 leftover bytes with �, and � itself is 3 bytes. Re-encoded, the "22-byte" result is 24 bytes, which is over the limit you were trying to meet.

Python is stricter by default and fails loudly. Its lenient options still produce wrong text:

b[:10].decode("utf-8")              UnicodeDecodeError
b[:10].decode("utf-8", "ignore")    "Family "
b[:22].decode("utf-8", "ignore")    "Family 👨‍👩‍"        two people and a dangling joiner
b[:22].decode("utf-8", "replace")   "Family 👨‍👩‍�"

errors="ignore" is a popular answer to "how do I cut UTF-8 safely", and it does give valid UTF-8. It doesn't give whole characters. The grapheme loop with a byte measure (s => Buffer.byteLength(s, 'utf8') in Node, lambda s: len(s.encode('utf-8')) in Python) returns "Family" here, because the family emoji alone is 18 bytes and doesn't fit in the remaining 15. A shorter result is the right answer. If you want to check a byte count by hand, the URL Encoder shows each UTF-8 byte as a %XX group.

5. Token budgets: the same problem, one layer up

Language-model budgets are in tokens, and tokenizers work on bytes. With OpenAI's tiktoken and the o200k_base encoding, the token pieces show where the boundaries fall:

"Great job 👍🏽 team"   6 tokens: 'Great' | ' job' | ' 👍' | f0 9f 8f | bd | ' team'
"Family 👨‍👩‍👧 photo"   10 tokens: the family takes 8 of them, each person split across two

The skin-tone modifier is split across two tokens, so keeping the first four tokens and decoding gives half a character:

enc.decode(ids[:4])  -> "Great job 👍�"
enc.decode(ids[:2])  -> "Family �"       (family string)
enc.decode(ids[:8])  -> "Family 👨‍👩‍�"

decode replaces the broken bytes by default, so this doesn't fail. It sends � to the model, or stores it in a summary. In a Turkish sample ("Merhaba, nasılsınız? Çok güzel!", 11 tokens) every token happened to end on a whole character. A test on a few English and Turkish strings will pass, and emoji are what break it.

The same grapheme loop works for tokens if measure counts tokens. There is one more catch, which I measured. The token count of a prefix isn't monotonic: across 202 grapheme prefixes of a mixed sample, a longer prefix had fewer tokens than the one before it 9 times, because the tokenizer merges pieces differently once more text follows. In practice, measure the final candidate together with its ending rather than adding up per-grapheme counts:

def truncate_tokens(text, max_tokens, ending='…'):
    if len(enc.encode(text)) <= max_tokens:
        return text
    gs = regex.findall(r'\X', text)
    lo, hi = 0, len(gs)                      # binary search over grapheme prefixes
    while lo < hi:
        mid = (lo + hi + 1) // 2
        if len(enc.encode(''.join(gs[:mid]).rstrip() + ending)) <= max_tokens:
            lo = mid
        else:
            hi = mid - 1
    return ''.join(gs[:lo]).rstrip() + ending

Budgets of 6, 7 and 9 tokens on "Great job 👍🏽 team, see you tomorrow" gave "Great job 👍🏽…" (6), "Great job 👍🏽 team,…" (7) and "Great job 👍🏽 team, see…" (9). For budgets 6 to 9, a slow linear scan over every prefix picked the same cut each time. Because the count isn't strictly monotonic, a binary search can in principle stop one grapheme short of the best cut. It can never return something over the budget, since every candidate is measured with its ending.

6. A performance trap in Node 20

Intl.Segmenter is the right tool, but on Node 20 it slows down badly on long strings. I timed iterating every grapheme of the same repeated sentence:

code units   Node 20.19.2   Node 22.23.4   Node 24.22.1   Chrome 156
10,000            20 ms         3.7 ms         1.0 ms        0.7 ms
40,000           195 ms         9.5 ms         2.4 ms        2.2 ms
80,000         8,660 ms        17.1 ms         8.8 ms        3.8 ms

The jump from 40,000 to 80,000 units on Node 20 was the same in two runs (8.9 s and 8.7 s). Iterating the same 80,000-unit string by code point took under 1 ms. The truncate above only walks as far as the limit, so cutting a 160,000-unit string to 280 took 50 ms even on Node 20. What hurts is code that segments the whole string first, for example to count graphemes or to cut from the end. Node 20 reached end-of-life on 30 April 2026 according to the Node.js release schedule, so this is one more reason to move to 22 or 24.

A checklist for any cut

1. Name the unit of the limit: UTF-16 units (maxlength, most UIs), code points (Python, MySQL CHAR_LENGTH), bytes (storage, headers, payloads) or tokens (models).
2. Cut between graphemes, always: Intl.Segmenter in JavaScript, regex \X in Python. Never call slice, subarray or decode(errors="ignore") on user text.
3. Measure the candidate with its ending, and let the ending count toward the limit.
4. Expect results shorter than the limit. A family emoji is 8 units, 18 bytes or several tokens, and it fits whole or not at all.
5. Guard boundaries. Use isWellFormed() before sending JSON to a stricter service. Normalize to NFC first if two visually identical strings should get the same cut.
6. Test with the four strings above: skin tone, flag, ZWJ family, combining accent. Plain English text will pass any implementation.
7. Don't segment huge strings on Node 20. Stop at the limit, or upgrade.

For quick checks while you write tests, the Character Counter counts the way JavaScript does and the Word Counter adds reading time for longer drafts.

Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including adding an AI feature to your web app, such as a chat assistant, document analysis or search.

Both choices sound conservative. After running the alternatives, I think both are right, and the reasons are good to know for any JavaScript code that touches big integers: IDs, hashes, money in minor units, or a prime check of your own.

Everything below was run on Node 20.19.2. The Go and Python cross-checks used Go 1.27.3 and Python 3.13.5.

Trial division, done properly

function isPrimeTD(n) {
  if (!Number.isSafeInteger(n) || n < 2) throw new RangeError(`not a safe integer >= 2: ${n}`);
  if (n % 2 === 0) return n === 2;
  if (n % 3 === 0) return n === 3;
  for (let i = 5; i * i <= n; i += 6) {        // 6k ± 1: skips multiples of 2 and 3
    if (n % i === 0 || n % (i + 2) === 0) return false;
  }
  return true;
}

Stopping at √n is safe: if n = a × b with a ≤ b, then a ≤ √n, so a composite number always has a factor no larger than its square root. The 6k ± 1 step skips two thirds of the candidates. Timings on a server CPU:

              561 composite      0.0 ms
       2147483647 prime          1.1 ms
    1000000000039 prime          2.7 ms
 9007199254740881 prime        281.5 ms
 9007199254740991 composite      0.0 ms

9,007,199,254,740,881 is the largest prime below 2^53, the guide's worst case, and it still finishes in under a third of a second. 2^53 − 1 itself falls instantly (it's divisible by 6,361). For anything a Number can hold, trial division is fast enough and exact.

The interesting line is the first one in the function.

The wall: above 2^53, every number is even

JavaScript numbers are 64-bit floats. They hold every integer exactly up to Number.MAX_SAFE_INTEGER, 2^53 − 1 (MDN). Past that, the gaps between representable numbers are 2, then 4, then 8. Between 2^53 and 2^54, only even numbers exist:

Number('9007199254740993')                 9007199254740992
9007199254740993 === 9007199254740992   -> true
Number('18446744073709551557') % 2      =  0

18,446,744,073,709,551,557 is the largest prime below 2^64. Parsed into a Number, it becomes an even number, and a naive check without the safety test answers confidently:

isPrimeNaive(Number('18446744073709551557')) -> false   (it IS prime)
isPrimeTD                                    -> not a safe integer >= 2: 18446744073709552000

The wrong answer comes with no error and no warning. The guard throws instead, and the error message prints the number JavaScript actually received, which isn't the one in the input. That is the whole case for refusing large inputs: the code never saw your number.

The second wall comes earlier: modular exponentiation

Fast primality tests use modular exponentiation, a^e mod m, computed by repeated squaring. Each step multiplies two numbers smaller than m. If m has 13 digits, that product has up to 26, far past 2^53, and it gets rounded before the %:

function modpowNum(b, e, m) {           // looks fine, isn't
  let r = 1; b %= m;
  while (e > 0) { if (e % 2 === 1) r = r * b % m; b = b * b % m; e = Math.floor(e / 2); }
  return r;
}

By Fermat's little theorem, 3^(p−1) mod p must be 1 for a prime p. With p = 1,000,000,000,039:

Number: 3^(m-1) mod m = 183510405447
BigInt: 3^(m-1) mod m = 1n

The modulus is a perfectly safe integer and the result is still garbage. The limit for multiply-then-mod with Number is about √(2^53) ≈ 94.9 million, not 2^53. Above that, use BigInt, which has no size limit:

function modpow(b, e, m) {
  let r = 1n; b %= m;
  while (e > 0n) { if (e & 1n) r = r * b % m; b = b * b % m; e >>= 1n; }
  return r;
}

Fermat's shortcut and why 561 breaks it

If n is prime, a^(n−1) mod n = 1 for every base a that shares no factor with n. The Fermat test checks a base or two and says "probably prime" when the result is 1. It's fast, and some composites pass it:

2^560 mod 561 = 1n            (561 = 3 x 11 x 17)
bases coprime to 561: 319, of which Fermat liars: 319
base-2 Fermat pseudoprimes below 2,000: 341, 561, 645, 1105, 1387, 1729, 1905
Carmichael numbers below 100,000: 561, 1105, 1729, 2465, 2821, 6601, 8911, 10585,
  15841, 29341, 41041, 46657, 52633, 62745, 63973, 75361

561 is a Carmichael number: every one of the 319 bases that share no factor with it says "prime". The guide explains why: 2, 10 and 16, one less than each prime factor, all divide 560. Picking more bases at random doesn't help unless you happen to pick a multiple of 3, 11 or 17.

Miller-Rabin with BigInt

Miller-Rabin fixes this. It writes n − 1 = d × 2^s, and instead of only checking that a^(n−1) ends at 1, it checks how the sequence a^d, a^(2d), … gets there. For a prime, that sequence either starts at 1 or passes through n − 1 (that is, −1) before it reaches 1. A composite that reaches 1 any other way has been caught, and Carmichael numbers get caught for most bases.

const BASES = [2n, 3n, 5n, 7n, 11n, 13n, 17n, 19n, 23n, 29n, 31n, 37n];
function isPrimeMR(n) {
  n = BigInt(n);
  if (n < 2n) return false;
  for (const p of BASES) { if (n === p) return true; if (n % p === 0n) return false; }
  let d = n - 1n, s = 0;
  while ((d & 1n) === 0n) { d >>= 1n; s++; }
  outer: for (const a of BASES) {
    let x = modpow(a, d, n);
    if (x === 1n || x === n - 1n) continue;
    for (let i = 1; i < s; i++) { x = x * x % n; if (x === n - 1n) continue outer; }
    return false;                       // a proves n composite
  }
  return true;
}

Pass big values as strings (isPrimeMR("18446744073709551557")) so they never go through a Number. I compared it with trial division on every integer from 0 to 999,999 and on 2,000 pseudo-random numbers between 10^12 and 3.2 × 10^12. There were zero mismatches. Then the big ones:

            9007199254740881 prime        0.11 ms
            9007199254740993 composite    0.01 ms
        18446744073709551557 prime        0.27 ms
        18446744073709551559 composite    0.02 ms
 618970019642690137449562111 prime        0.43 ms
   3317044064679887385961981 prime        0.31 ms

The largest prime below 2^53 takes 0.11 ms instead of 281 ms. 2^89 − 1, a known Mersenne prime, takes under half a millisecond.

Now look at the last line. 3,317,044,064,679,887,385,961,981 is not prime. A short Pollard rho script in Python factors it, and a second number like it, in about a second:

3317044064679887385961981 = 1287836182261 x 2575672364521
318665857834031151167461  = 399165290221 x 798330580441

Both are strong pseudoprimes to all twelve bases from 2 to 37. They are listed in OEIS A014233, the smallest numbers that fool Miller-Rabin with the first k primes as bases. 318,665,857,834,031,151,167,461 is the smallest odd composite that fools all twelve. The larger one fools base 41 as well. That gives the rule:

• With the first 12 primes as bases, Miller-Rabin is exact for n < 318,665,857,834,031,151,167,461, which covers every 64-bit integer (2^64 ≈ 1.8 × 10^19).
• Above that bound, a fixed set of bases is a "probably prime", not a proof. The 2^89 − 1 result above is right because that number really is prime, not because the test proved it.

Go's standard library shows what a more robust test looks like. big.Int.ProbablyPrime(0) runs a Baillie-PSW test, and its documentation says it is "100% accurate for inputs less than 2⁶⁴". It rejected both pseudoprimes:

                         561  ProbablyPrime(0)=false
    318665857834031151167461  ProbablyPrime(0)=false
   3317044064679887385961981  ProbablyPrime(0)=false
 618970019642690137449562111  ProbablyPrime(0)=true

Bonus: the same wall in LCM

The GCD and LCM post gives lcm(a, b) = a × b ÷ gcd(a, b). In floating point, the order of operations matters. The product overflows 2^53 long before the result does:

lcm(800900072081, 801100072099)  gcd=100000009
  a*b/gcd = 6416010477440890
  a/gcd*b = 6416010477440891
  BigInt  = 6416010477440891

The correct LCM is below 2^53, but a*b isn't, so multiplying first loses the last digit. Dividing by the GCD first keeps every intermediate value small. When the LCM itself is above 2^53, neither order helps: lcm(94906267, 94906265) prints 9007199326062756 either way, and only BigInt gives the true …755. Also watch for mixed types: BigInt(a) / g(a, b) throws TypeError: Cannot mix BigInt and other types if g returns a Number. That's annoying, but it's the language refusing to round your number silently, which is what you want.

Rules I use now

1. Validate first. Number.isSafeInteger(n) before any integer math. Above the limit, refuse, or take a string and switch to BigInt.
2. Multiply-then-mod is safe only below about 94.9 million with Number. Any modular arithmetic on bigger numbers goes in BigInt.
3. Trial division is exact and fast enough up to 2^53 (under 0.3 s in the worst case here). It's what a calculator should use there.
4. Miller-Rabin with the first 12 primes is a proof for 64-bit inputs and fast far beyond, but above 3.2 × 10^23 it only says "probably". Use Baillie-PSW, or a certified test, when "probably" isn't enough.
5. Never Fermat alone. 561 is the smallest of infinitely many Carmichael numbers.
6. Divide before you multiply in LCM-style formulas, and use BigInt when the result may pass 2^53.

If you just need an answer for one number, the Prime Checker runs trial division in your browser and shows the full factorization, like 360 = 2³ × 3² × 5. For anything above 2^53, you now know why it says no.

Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including fast React front ends for web apps and dashboards.

Versions need a version parser. The catch is that there are several, and they don't agree. Here is the string that shows it best:

1.0.0-1   vs   1.0.0

Every result in this article comes from a real run: Node 20.19 with semver 7.8.5, Python 3.13.5 with packaging 26.3, dpkg 1.22.22, GNU coreutils 9.7, PHP 8.4.26 with composer/semver 3.4.4, and Go 1.27.2 with golang.org/x/mod v0.42.0.

The baseline: text sort and sort -V

== sort (LC_ALL=C)       == sort -V
1.0.0                    1.0.0
1.0.0-rc.1               1.0.0-rc.1
1.10.0                   1.2.0
1.2.0                    1.9.0
1.9.0                    1.10.0

sort -V fixes the numbers but still puts the release before its release candidate. The right order is 1.0.0-rc.1, then 1.0.0.

npm semver: pre-release identifiers have two rules

const semver = require("semver");
semver.sort(["1.10.0", "1.9.0", "1.2.0", "1.0.0", "1.0.0-rc.1"]);
// [ '1.0.0-rc.1', '1.0.0', '1.2.0', '1.9.0', '1.10.0' ]

That's correct. Now the parts that surprise people:

alpha.2 vs alpha.10:   -1      (2 < 10)
alpha2  vs alpha10:     1      (alpha2 > alpha10)
+build.1 vs +build.2:   0      (equal)
valid('1.0'): null   valid('v1.0.0'): 1.0.0   coerce('1.0'): 1.0.0

The semver spec compares dot-separated identifiers that are all digits as numbers, and everything else as ASCII text. alpha.10 is two identifiers, so 10 beats 2. alpha10 is one identifier, so it's compared character by character, and "alpha2" sorts after "alpha10". Always put a dot before the number: rc.1, beta.12.

Build metadata after + is ignored for ordering, so +build.1 and +build.2 are equal. Don't put anything after + that has to sort.

Ranges have one more rule:

satisfies 1.3.0-beta.1 ^1.2.3:                      false
satisfies 1.3.0-beta.1 ^1.2.3 includePrerelease:    true
satisfies 1.2.3-beta.2 ^1.2.3-beta.1:               true

^1.2.3 won't match a pre-release of a different version, even though 1.3.0-beta.1 is numerically inside the range. A pre-release matches only when the range itself names a pre-release of the same major.minor.patch, or when you pass includePrerelease.

Python packaging (PEP 440): normalization first

from packaging.version import Version
sorted(["1.10.0", "1.9.0", "1.2.0", "1.0.0", "1.0.0rc1",
        "1.0.0.post1", "1.0.0.dev1", "1.0.0a1", "1!0.1"], key=Version)

['1.0.0.dev1', '1.0.0a1', '1.0.0rc1', '1.0.0', '1.0.0.post1', '1.2.0', '1.9.0', '1.10.0', '1!0.1']

PEP 440 has more phases than semver: dev releases come before alphas, post-releases after the final release, and an epoch (1!) beats everything without one. It is also forgiving about spelling, and that's where the 1.0.0-1 surprise comes from:

1.0 == 1.0.0: True
1.0.0-rc.1 -> 1.0.0rc1     1.0.0RC1 -> 1.0.0rc1     v1.0 -> 1.0
1.0.0-1    -> 1.0.0.post1  1.0.0_beta -> 1.0.0b0
1.0.0-alpha.2 -> 1.0.0a2   latest -> InvalidVersion

Local versions (1.0+ubuntu1) sort after 1.0, but ==1.0 still matches them.

One change worth knowing if you upgraded recently. Specifier checks on pre-releases changed in packaging 26.0. Its changelog lists "PEP 440 handling of prereleases for Specifier.contains, SpecifierSet.contains, and SpecifierSet.filter". Same expression, three versions:

packaging 24.2   "2.0rc1" in SpecifierSet(">=1.0")  ->  False
packaging 25.0   "2.0rc1" in SpecifierSet(">=1.0")  ->  False
packaging 26.0   "2.0rc1" in SpecifierSet(">=1.0")  ->  True

filter() in 26.3 still drops the release candidate when a final release also matches (filter(["1.5", "2.0rc1"]) gives ['1.5']), and keeps it when it's the only candidate. If your code uses in to decide whether to offer an upgrade, pin packaging or pass prereleases= explicitly. The packaging changelog has the details.

Debian dpkg: ~ sorts before nothing, - splits the revision

1.0~rc1   < 1.0
1.0-rc1   > 1.0
1.0~~     < 1.0~
1.0       < 1.0+b1
1:1.0     > 2.0
1.0-1     < 1.0-1ubuntu1
1.0.0     > 1.0
2:1.0     > 10.0

(dpkg --compare-versions a lt b for each pair.) The Debian policy defines the format as [epoch:]upstream_version[-debian_revision]. A tilde sorts before everything, even the end of the string, so 1.0~rc1 is the Debian way to write a release candidate. A hyphen starts the revision, so 1.0-rc1 is "upstream 1.0, package revision rc1", which is newer than 1.0. That's the opposite of what an npm user would read.

sort -V is close to these rules but not the same. It ignores epochs:

printf '10.0\n2:1.0\n' | sort -V    ->   2:1.0, 10.0
dpkg says                           ->   2:1.0 > 10.0

PHP: version_compare() and Composer don't agree with each other

version_compare('1.0', '1.0.0');          // -1  (1.0 is older than 1.0.0)
version_compare('1.0.0-rc.1', '1.0.0');   // -1
version_compare('1.0.0-dev', '1.0.0-alpha'); // -1
version_compare('1.0.0-pl1', '1.0.0');    //  1  (patch level, newer)
version_compare('1.0.0-foo', '1.0.0-dev'); // -1 (unknown words sort first)
version_compare('1.0.0-1', '1.0.0');      //  1
version_compare(PHP_VERSION, '8.1.0', '>='); // true on 8.4.26

The version_compare manual documents that order of words: anything not in its list, then dev, alpha/a, beta/b, RC/rc, #, pl/p. The words are matched by their start, which leads to the nastiest result of the whole run:

version_compare('1.0.0-preview1', '1.0.0') =  1   (newer than the release!)
version_compare('1.0.0-pre1', '1.0.0')     =  1
version_compare('1.0.0-beat1', '1.0.0')    = -1   (typo of beta, still older)
version_compare('1.0.0-foo', '1.0.0')      = -1

preview and pre begin with p, so they appear to be read as the patch-level marker and rank after 1.0.0. If you tag previews, use alpha, beta or RC. And 1.0 vs 1.0.0 is not a tie: the shorter one is older.

Composer's composer/semver (the library behind composer.json constraints) normalizes first:

normalize(1.0)          = 1.0.0.0
normalize(v1.0.0)       = 1.0.0.0
normalize(1.0.0-rc.1)   = 1.0.0.0-RC1
normalize(1.0.0-beta.2) = 1.0.0.0-beta2
normalize(1.0.0-1)      -> UnexpectedValueException: Invalid version string "1.0.0-1"
Comparator::equalTo('1.0', '1.0.0')                         -> false
Comparator::equalTo(normalize('1.0'), normalize('1.0.0'))   -> true

Compare normalized strings, never raw ones. Constraints have their own stability rule:

parseConstraints('^2.0')                   -> [>= 2.0.0.0-dev < 3.0.0.0-dev]
satisfies('2.0.0-RC1', '^1.2 || ^2.0')     -> true

The constraint matches the RC. What keeps it out of your vendor/ folder is minimum-stability, which, as the Composer schema docs put it, "defaults to stable" and ignores less stable versions "when resolving your project dependencies". So if you call Semver::satisfies() in your own code, for example to pick an upgrade or show "update available", check VersionParser::parseStability() as well. Here it returns RC for 2.0.0-RC1.

This matters during PHP upgrades: version_compare(PHP_VERSION, …) guards are often written by hand, and '8.1' and '8.1.0' are not equal.

Go: the v is required, and invalid compares equal

tags := []string{"v1.10.0", "v1.9.0", "v1.2.0", "v1.0.0", "v1.0.0-rc.1"}
semver.Sort(tags)  // [v1.0.0-rc.1 v1.0.0 v1.2.0 v1.9.0 v1.10.0]

IsValid("1.10.0"): false       IsValid("v1.2"): true
Compare("1.10.0", "1.9.0"): 0
sorted without the v: [1.10.0 1.9.0 1.0.0]
Compare("v1.2", "v1.2.0"): 0
IsValid("v1.2+build.5"): false   Canonical("v1.2"): v1.2.0
Compare("1.10.0", "v0.0.1"): -1

golang.org/x/mod/semver expects the v prefix that Go modules use. Without it every string is invalid, invalid strings compare equal to each other and below every valid one, and a sort quietly does nothing. The list came back in the order it went in, with no error. Check IsValid or add the v before sorting. Shorthand like v1.2 is valid and equals v1.2.0, but not with a pre-release or build suffix.

Cheat sheet

Rules I follow now

1. Parse versions with the library of the ecosystem the version comes from. A Debian package version in a Python script still goes through dpkg --compare-versions.
2. Write pre-releases so every parser agrees: 1.0.0-rc.1 in semver/npm/Go, 1.0.0rc1 in Python, 1.0.0~rc1 in Debian.
3. Never write 1.0.0-1 for anything. Three parsers call it newer, two older, one rejects it.
4. Validate before sorting. Go's sort and PHP's version_compare won't tell you when an input is nonsense, and in PHP a word like preview can even change the answer.
5. When you only need a sorted list of plain x.y.z releases in a browser tab, the Text Line Sorter won't do it either. The sorting post notes that Numeric mode reads v1.10 as the decimal 1.1. Paste the list into one of the scripts above instead.

Need help with a project? The team behind FindUtils takes on freelance work on Upwork, including PHP bug fixes and PHP 8 upgrades.

1. The one-pass "shortcut" formula returns 0, or NaN, on large values such as timestamps.
2. Averaging daily standard deviations, or storing the wrong totals in a summary table, gives a number that isn't the standard deviation of anything.
3. With n − 1, a group with one value becomes NaN, NULL or an exception, depending on the tool.

Everything below was run with Node 20.19, Python 3.13, NumPy 2.5.4, pandas 3.0.6 and DuckDB 1.5.6. Outputs are copied from those runs.

A quick reference point first. For 2, 4, 4, 4, 5, 5, 7, 9 the population standard deviation is 2 and the sample one is about 2.138. The FindUtils Statistics Calculator shows 2.1380899353 in its default Sample (n-1) mode, which is handy for checking your own code against.

Bug 1: the shortcut formula

Most people who write standard deviation by hand reach for this identity, because it needs only one pass and two running totals:

Σ(x − mean)² = Σx² − (Σx)² / n

On paper it's exact. In floating point, it subtracts two huge, nearly equal numbers, and the small difference you want can disappear in rounding. Here are three implementations side by side:

// stddev.mjs: three ways to compute a sample standard deviation

// 1. The "shortcut" one-pass formula: sum and sum of squares.
export function stdevNaive(xs) {
  let n = 0, sum = 0, sumSq = 0;
  for (const x of xs) { n++; sum += x; sumSq += x * x; }
  const variance = (sumSq - (sum * sum) / n) / (n - 1);
  return Math.sqrt(variance);
}

// 2. Two passes: mean first, then squared distances from it.
export function stdevTwoPass(xs) {
  const n = xs.length;
  const mean = xs.reduce((a, b) => a + b, 0) / n;
  let ss = 0;
  for (const x of xs) ss += (x - mean) ** 2;
  return Math.sqrt(ss / (n - 1));
}

// 3. Welford: one pass, keeps count, mean and M2 (sum of squared distances).
export class RunningStats {
  n = 0; mean = 0; m2 = 0;
  push(x) {
    this.n++;
    const d = x - this.mean;
    this.mean += d / this.n;
    this.m2 += d * (x - this.mean);
  }
  variance({ ddof = 1 } = {}) {
    return this.n - ddof > 0 ? this.m2 / (this.n - ddof) : NaN;
  }
  stdev(opts) { return Math.sqrt(this.variance(opts)); }
}

Now feed them something ordinary: eight request arrival times in Unix milliseconds, spread over about one second.

// stamps.mjs
import { stdevNaive, stdevTwoPass, RunningStats } from "./stddev.mjs";

const t0 = 1792479600000; // 2026-10-20T07:00:00Z
const arrivals = [0, 120, 135, 410, 455, 610, 870, 990].map((d) => t0 + d);
const w = new RunningStats();
arrivals.forEach((t) => w.push(t));
console.log("naive   :", stdevNaive(arrivals));
console.log("two-pass:", stdevTwoPass(arrivals));
console.log("welford :", w.stdev());

$ node stamps.mjs
naive   : 0
two-pass: 359.7295611189844
welford : 359.72959874234334

The shortcut says the arrivals have no spread at all. No error, no warning, just 0.

Why it happens. Each timestamp is about 1.79 × 10¹², so each square is about 3.2 × 10²⁴, and the sum of eight squares is about 2.57 × 10²⁵. A double has 53 bits of precision, so at that size the gap between neighbouring representable numbers is 2³², about 4.3 billion. The quantity we actually want, Σ(x − mean)², is about 905,838. It's smaller than a single rounding step of the totals, so the subtraction can't recover it.

It doesn't take timestamps to hit this. I added a constant offset to 2, 4, 4, 4, 5, 5, 7, 9 (same spread, bigger numbers) and ran the shortcut:

NaN shows up when rounding makes the "variance" negative and Math.sqrt refuses it. So you get either a confident 0 or a NaN that spreads through every number computed from it.

It's not a JavaScript problem. In Python, the same formula on floats also gives 0.0. Python's integers are exact, but s * s / n divides too early and drops back to a float, which also gives 0.0. Keeping the arithmetic in integers until the end works:

t0 = 1792479600000
xs = [t0 + d for d in [0, 120, 135, 410, 455, 610, 870, 990]]
n, s, sq = len(xs), sum(xs), sum(x * x for x in xs)
print(((n * sq - s * s) / (n * (n - 1))) ** 0.5)   # 359.7295611189844

SQL has the same trap, plus another one. In DuckDB, squaring a BIGINT timestamp fails outright:

Out of Range Error: Overflow in multiplication of INT64 (1792479600000 * 1792479600000)!

Cast to DOUBLE to get past the error, and the shortcut quietly returns 0.0 again.

Library functions are fine. statistics.stdev, np.std(..., ddof=1), pandas .std() and DuckDB's stddev_samp all returned 359.7… for the timestamps. The FindUtils Statistics Calculator gave 359.729561119 too. The bug lives in hand-rolled code: a quick script, a reduce in a dashboard, or a summary table built from SUM(x) and SUM(x*x).

Three fixes

• Two passes when you have all the data: compute the mean, then sum squared distances from it. It's exact to the last printed digit here.
• Welford's algorithm when you're streaming and can't keep the values. It updates the count, the mean and M2 one value at a time, and never forms the huge squares. Its result above, 359.72959874234334, differs from the exact value in the eighth significant digit. DuckDB's stddev_samp returned exactly the same 359.72959874234334.
• Subtract a reference value first. The spread doesn't change if you shift every value by the same constant. Subtract the first timestamp (or any fixed value close to the data), and even the shortcut works. In SQL:

SELECT sqrt((sum((ts_ms - 1792479600000)::DOUBLE ^ 2)
             - sum((ts_ms - 1792479600000)::DOUBLE)^2 / count(*)) / (count(*) - 1)) AS naive_shifted
FROM hits;
-- 359.7295611189844

Welford on the shifted values also gives 359.7295611189844. For timestamps, shifting first is a good habit anyway.

Bug 2: averaging daily standard deviations

Say you keep a summary per day instead of raw rows. The tempting move for a weekly number is to average the daily standard deviations. It's wrong, and not by a little.

The fix is to store three numbers per group (count, mean and M2) and combine them with the parallel formula from Chan, Golub and LeVeque, which merges two summaries exactly:

import statistics
from dataclasses import dataclass

@dataclass
class Summary:
    n: int = 0
    mean: float = 0.0
    m2: float = 0.0          # sum of squared distances from the mean

    def push(self, x):       # Welford's update
        self.n += 1
        d = x - self.mean
        self.mean += d / self.n
        self.m2 += d * (x - self.mean)

    def merge(self, other):  # combine two summaries (Chan et al.)
        n = self.n + other.n
        if n == 0:
            return Summary()
        d = other.mean - self.mean
        return Summary(n, self.mean + d * other.n / n,
                       self.m2 + other.m2 + d * d * self.n * other.n / n)

    def stdev(self, ddof=1):
        return (self.m2 / (self.n - ddof)) ** 0.5 if self.n > ddof else float("nan")

# Response times in ms for two days (a quiet Monday, a slower Tuesday)
monday  = [112, 98, 130, 105, 121, 99, 117, 108]
tuesday = [180, 240, 210, 195, 260, 205, 230, 188, 199, 221, 215, 240]

days = {}
for name, xs in [("monday", monday), ("tuesday", tuesday)]:
    s = Summary()
    for x in xs:
        s.push(x)
    days[name] = s
    print(f"{name:8} n={s.n:2}  mean={s.mean:7.2f}  stdev={s.stdev():6.2f}")

week = days["monday"].merge(days["tuesday"])
print(f"merged   n={week.n}  mean={week.mean:7.2f}  stdev={week.stdev():6.2f}")
print(f"raw data n={len(monday + tuesday)}  mean={statistics.mean(monday + tuesday):7.2f}  stdev={statistics.stdev(monday + tuesday):6.2f}")
print(f"average of the two daily stdevs: {(days['monday'].stdev() + days['tuesday'].stdev()) / 2:6.2f}")

$ python3 merge.py
monday   n= 8  mean= 111.25  stdev= 11.06
tuesday  n=12  mean= 215.25  stdev= 23.88
merged   n=20  mean= 173.65  stdev= 55.74
raw data n=20  mean= 173.65  stdev= 55.74
average of the two daily stdevs:  17.47

The merged summary matches the raw data exactly. The average of the daily values, 17.47, is a third of the real answer. It only measures how much each day varies around its own mean, and misses that Tuesday sat about 100 ms above Monday. That shift between days is usually the thing you wanted to see.

The same works in SQL with any number of groups. Total M2 is the sum of the group M2 values plus each group's count times its squared distance from the overall mean:

CREATE TABLE daily AS
SELECT day, count(*) AS n, avg(ms) AS mean, var_pop(ms) * count(*) AS m2
FROM requests GROUP BY day;

WITH g AS (SELECT sum(n * mean) / sum(n) AS mean FROM daily)
SELECT sum(n) AS n,
       round(g.mean, 2) AS mean,
       round(sqrt((sum(m2) + sum(n * (daily.mean - g.mean) ^ 2)) / (sum(n) - 1)), 2) AS stdev
FROM daily, g GROUP BY g.mean;
-- n = 20, mean = 173.65, stdev = 55.74 (stddev_samp over the raw rows: 55.74)

If your summary table already stores count, sum and sum of squares, it's fine for small values like these response times, but it brings back Bug 1 for anything large. Either store the mean and M2, or store the sums of shifted values.

To spot-check a merged number, paste the raw values into the Statistics Calculator. For these 20 response times it shows 55.7449785324.

Bug 3: n − 1 when n = 1

A sample standard deviation divides by n − 1, so a single value means dividing by zero. Every tool deals with that differently:

statistics.stdev([5]) : StatisticsError: stdev requires at least two data points
np.std([5], ddof=1)   : nan | warning: RuntimeWarning
pd.Series([5]).std()  : nan

DuckDB's stddev_samp over one row returned NULL, while stddev_pop returned 0.0. In pandas, a groupby where one route has a single request quietly puts NaN in that row:

route
/a    163.273799
/b           NaN

As the FindUtils post notes, the Statistics Calculator shows 0 for a single value, which you should read as "not enough data". The same applies to your own dashboards. Don't COALESCE a NULL standard deviation to 0, because 0 means "no spread", which is a claim you have no data for. Show "n/a", or filter with HAVING count(*) >= 2 and display the count next to the number. A standard deviation of three values is a very rough estimate either way.

Checklist

• Don't hand-roll Σx² − (Σx)²/n. Use two passes, Welford or a library function.
• Subtract a reference value before computing the spread of timestamps or other large numbers.
• Store count, mean and M2 in summary tables, and merge with the parallel formula. Never average standard deviations.
• Decide on purpose what one value means: NaN, NULL, an exception or "n/a". Never silently show 0.
• Keep a known answer around for tests. 2, 4, 4, 4, 5, 5, 7, 9 should give 2 (population) and 2.1380899353 (sample).

The divisor is the part everyone argues about. The arithmetic around it is what quietly turns a dashboard number into 0.

That's one idea with five spellings, and each one is right for the layer it lives in. Bugs show up at the borders: a key that didn't get renamed, a rename that lost a word, a type that describes a shape the code no longer has.

This post goes through each layer in turn: what its convention is and why, where I'd do the conversion, and a small TypeScript helper (tested with tsc --strict) that handles the edge cases most snippets skip.

The map

Layer 1: the database wants snake_case, and Postgres enforces it

The PostgreSQL docs are explicit: "unquoted names are always folded to lower case". If you write CREATE TABLE users (userProfileId int), the column is actually named userprofileid. The only way to keep the capital letters is to quote the name, "userProfileId", and then every query has to quote it too, forever.

So the database gets snake_case, and the first conversion can already happen in SQL if you want it there. A quoted alias keeps its case:

SELECT user_profile_id AS "userProfileId",
       created_at      AS "createdAt"
FROM users;

That works for one query. For a whole API, a single conversion step at the boundary is easier to keep consistent.

Layer 2: convert once, at the API boundary

JSON has no naming rules, so the choice is yours. What matters is that you make it once. Plenty of public APIs send snake_case and plenty send camelCase. The bug-prone setup is a mix: half the endpoints renamed in a serializer, half passing rows through untouched.

If your JSON is snake_case and your frontend is camelCase, convert in one function at the edge, and make the types follow it. Typing a raw response is quick with a generator. The JSON to TypeScript guide on FindUtils makes a point worth repeating here: generated types keep property keys exactly as they appear in the JSON, so created_at stays created_at. That's correct, because the types have to match what your code actually receives. But it also means that once you rename keys at runtime, the generated interface is describing the old shape.

TypeScript can do that rename at the type level, so you don't have to maintain two interfaces by hand:

type CamelCase<S extends string> =
  S extends `${infer Head}_${infer Tail}`
    ? `${Head}${Capitalize<CamelCase<Tail>>}`
    : S;

type CamelKeys<T> =
  T extends readonly (infer U)[] ? CamelKeys<U>[]
  : T extends object
    ? { [K in keyof T as K extends string ? CamelCase<K> : K]: CamelKeys<T[K]> }
    : T;

And the runtime half, which walks objects and arrays:

export function camelizeKeys<T>(value: T): CamelKeys<T> {
  if (Array.isArray(value)) return value.map(camelizeKeys) as CamelKeys<T>;
  if (value !== null && typeof value === "object") {
    return Object.fromEntries(
      Object.entries(value).map(([k, v]) => [toCamel(k), camelizeKeys(v)])
    ) as CamelKeys<T>;
  }
  return value as CamelKeys<T>;
}

interface UserRow {
  user_profile_id: number;
  created_at: string;
  last_login_ip: string | null;
  roles: { role_name: string }[];
}

const user = camelizeKeys(row as UserRow);
user.userProfileId;      // number
user.roles[0].roleName;  // string
user.user_profile_id;    // compile error: the old key is gone

Two warnings before you ship this, both of which I hit while testing:

1. Collisions. If a payload has both user_id and userId (it happens with legacy fields), both map to userId and the last one silently wins. If your data can contain both, check for duplicate output keys and throw.
2. Type and runtime can disagree. For a key like _id, the template-literal type produces Id (empty head, capitalized tail), while the runtime splitter drops the leading underscore and produces id. The types say one thing and the object holds another. Leave keys like that alone, or special-case them in both places.

The hard part: finding the words

Every conversion is really two steps: split the name into words, then join them in the new style. Joining is trivial. Splitting is where names get damaged.

Here's a splitter that handles separators, camel humps, digits and acronyms:

export function words(name: string): string[] {
  return name
    .replace(/([a-z0-9])([A-Z])/g, "$1 $2")     // userProfile -> user Profile
    .replace(/([A-Z]+)([A-Z][a-z])/g, "$1 $2")  // XMLHttp     -> XML Http
    .split(/[^A-Za-z0-9]+/)                      // spaces, _, -, .
    .filter(Boolean)
    .map((w) => w.toLowerCase());
}

const cap = (w: string) => w.charAt(0).toUpperCase() + w.slice(1);
export const toCamel = (s: string) => words(s).map((w, i) => (i ? cap(w) : w)).join("");
export const toSnake = (s: string) => words(s).join("_");
export const toKebab = (s: string) => words(s).join("-");
export const toConstant = (s: string) => toSnake(s).toUpperCase();

The output, straight from running it:

Look at the last row. userID becomes user_id, and user_id comes back as userId, not userID. Case conversion isn't reversible once acronyms are involved. The practical rule: choose one canonical spelling (usually the database column) and derive the others from it. Never derive the canonical form back from a derived one.

For one-off renames (a ticket, a doc, a migration script), a browser tool is quicker than writing code. The FindUtils Case Converter has ten cases, and its guide is upfront about the limits. Its snake_case and kebab-case buttons split at lower-to-upper humps, but camelCase and PascalCase lowercase the input first and need visible separators. So userProfileId → PascalCase gives Userprofileid unless you go through snake_case first. A run of capitals also stays one word, so XMLHttpRequest comes out as xmlhttp_request. That's the same trap the second regex above exists to avoid.

Layer 3: environment variables, and the Turkish i

Constants and environment variables are conventionally CONSTANT_CASE. POSIX is a little more nuanced than the folklore. It says the variables its standard utilities use consist "solely of uppercase letters, digits, and the ", and it reserves names containing lowercase letters for applications (POSIX.1-2017, chapter 8). In practice almost every framework and hosting panel expects MAX_RETRY_COUNT, so that's what I use. Keep in mind that names are case-sensitive on Linux and case-insensitive on Windows, so never rely on case alone to tell two variables apart.

The trap here is locale. If you generate env names in JavaScript, use toUpperCase(), never toLocaleUpperCase():

"user_id".toLocaleUpperCase("tr-TR"); // "USER_Ä°D"  (dotted capital I)
"user_id".toUpperCase();              // "USER_ID"
"TITLE".toLocaleLowerCase("tr-TR");   // "tıtle"    (dotless i)

In Turkish and Azerbaijani, i uppercases to Ä° and I lowercases to ı. That's correct for prose and wrong for identifiers. A build script running on a machine with a Turkish locale will quietly produce USER_Ä°D, which matches nothing. Python's str.upper() doesn't depend on the locale, but code that calls ICU or locale-aware APIs can.

Going the other way, loading MAX_RETRY_COUNT into a maxRetryCount config property, is the same toCamel from above, applied to Object.keys(process.env) with a prefix filter.

Layer 4: URLs, where encoding joins in

For paths and slugs, use kebab-case. Google's URL structure guidance recommends hyphens over underscores so people and search engines can tell the words apart.

Separators are never the problem in URLs. RFC 3986 lists -, ., _ and ~ as unreserved, so encodeURIComponent("user-profile_id.v2~x") returns the string unchanged. The trouble starts with values, not names:

encodeURIComponent("Café Menu");                       // "Caf%C3%A9%20Menu"
new URLSearchParams({ q: "Café & Co+1" }).toString();  // "q=Caf%C3%A9+%26+Co%2B1"

Encoding makes any value safe to put in a URL, but it doesn't make it readable. For a path that people will see, transliterate instead. A URL slug generator turns Café Menu into cafe-menu, and that's also the fix if you need an ASCII identifier from accented text, since the camelCase step above drops anything outside A–Z and 0–9. For a query value, encode it and don't transliterate. The FindUtils post on percent-encoding and query strings covers the %20 vs + split and double encoding in detail.

Query parameter names deserve one rule: they're case-sensitive in URLSearchParams and in most server frameworks, so ?userId=7 and ?userid=7 are different parameters. Pick the same convention you use for JSON keys and stick to it.

A checklist for your next field

• Choose one canonical spelling (normally the database column) and derive the rest from it.
• Don't fight Postgres. Use snake_case columns, or quote everywhere.
• Convert JSON keys once, in one function at the API edge, and derive the types (CamelKeys<T>) instead of keeping two interfaces.
• Guard against collisions (user_id + userId) and special keys (_id).
• Split acronyms deliberately, and remember that userID → user_id → userId doesn't round-trip.
• Use toUpperCase(), not toLocaleUpperCase(), for identifiers.
• Slugs get transliterated, query values get encoded, and parameter names stay as consistent as your JSON keys.

None of these conventions is better than the others. Each one is a local agreement. The code you write between layers is just translation, and translation goes wrong when it's done twice, done by hand, or done in a locale nobody expected.

Both go wrong in ways that are easy to miss. The shared file carries every meeting title, Zoom password and attendee email address in it. The spreadsheet shows one row for a weekly standup that actually ran fifty times.

This post explains why, using a small sample file, and then walks through a three-step pipeline: filter, scrub, expand. I'll use free browser tools from FindUtils for each step. Their tool pages say the calendar is read in your browser and nothing is uploaded. The iCalendar details apply whatever tool you use.

What's really inside an export

Here's one recurring event plus one moved occurrence, in the format Google Calendar, Apple Calendar and Outlook all export (RFC 5545):

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Example//Calendar//EN
X-WR-CALNAME:deniz@example.com
BEGIN:VEVENT
UID:standup-42@example.com
DTSTAMP:20261001T080000Z
DTSTART;TZID=Europe/Berlin:20261005T093000
DTEND;TZID=Europe/Berlin:20261005T094500
RRULE:FREQ=WEEKLY;BYDAY=MO,WE;COUNT=8
EXDATE;TZID=Europe/Berlin:20261014T093000
SUMMARY:Acme Corp standup (renewal talks)
DESCRIPTION:Join: https://zoom.us/j/81234567890?pwd=Qk9aU2xYdW1
 0ZzR4 Agenda: pricing for the Q1 renewal
LOCATION:Acme HQ\, Floor 3
ORGANIZER;CN=Deniz:mailto:deniz@example.com
ATTENDEE;CN=Sam Lee;PARTSTAT=ACCEPTED:mailto:sam@acme.example
CATEGORIES:Work
STATUS:CONFIRMED
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:Acme standup
TRIGGER:-PT10M
END:VALARM
END:VEVENT
BEGIN:VEVENT
UID:standup-42@example.com
RECURRENCE-ID;TZID=Europe/Berlin:20261021T093000
DTSTAMP:20261001T080000Z
DTSTART;TZID=Europe/Berlin:20261022T100000
DTEND;TZID=Europe/Berlin:20261022T101500
SUMMARY:Acme Corp standup (moved)
END:VEVENT
END:VCALENDAR

Three things stand out.

1. Most of the lines say what the event is, not when. Only DTSTART, DTEND, RRULE, EXDATE, RECURRENCE-ID (plus STATUS/TRANSP) describe time. The rest — SUMMARY, DESCRIPTION, LOCATION, ORGANIZER, ATTENDEE, the alarm text, and even the calendar name (in a Google export it's often the owner's email address) — is the private part.

2. Long lines are folded. RFC 5545 folds lines at 75 octets: a continuation line starts with a single space, and a reader joins it back without the space. The meeting password above is really Qk9aU2xYdW10ZzR4. A quick grep for it won't find it, and deleting the DESCRIPTION line by hand while leaving its continuation behind breaks the file. A fast way to see which properties a file carries:

# count top-level property names (folded continuation lines start with a space, so they're skipped)
grep -oE '^[A-Z-]+' work.ics | sort | uniq -c | sort -rn

On a real export, the ATTENDEE and DESCRIPTION counts are usually the eye-opener. (If the password lives in a URL's query string, the URL encoding explainer covers how ?pwd= and friends are put together.)

3. Eight meetings are stored as one rule. The series is one VEVENT with RRULE:FREQ=WEEKLY;BYDAY=MO,WE;COUNT=8. One date is removed by EXDATE, and the 21 October occurrence is replaced by a second VEVENT that shares the UID and names the occurrence it replaces in RECURRENCE-ID. A calendar app works all that out on display. A row-by-row converter doesn't.

If you want the full background on how these rules behave, the FindUtils post RRULE Explained: How Recurring Calendar Events Really Work goes through each part with real dates. The short version is below.

What those eight "occurrences" actually are

Worked out by hand (or with python-dateutil, see the end), the rule produces 5, 7, 12, 14, 19, 21, 26 and 28 October. Then:

• EXDATE removes 14 Oct. It does not add a ninth date at the end: COUNT counts the dates the rule generates, before exclusions. DTSTART counts as the first one.
• The RECURRENCE-ID component moves 21 Oct 09:30 to 22 Oct 10:00, with its own title.
• So seven meetings happen: 5, 7, 12, 19, 22 (moved), 26, 28 October.

There's also a time-zone trap. Berlin leaves summer time on 25 October 2026, so the meetings stay at 09:30 local time, but in UTC they move from 07:30Z to 08:30Z. RFC 5545 defines recurrence on the wall-clock time in the event's zone, which is what you want for a standup. A script that converts DTSTART to UTC first and then adds seven days will be an hour off after the change.

Two more rules that surprise people:

RRULE:FREQ=MONTHLY;BYMONTHDAY=31   -> Aug 31, Oct 31, Dec 31, Jan 31, Mar 31 ...  (months without a 31st are skipped)
RRULE:FREQ=MONTHLY;BYMONTHDAY=-1   -> Aug 31, Sep 30, Oct 31, Nov 30, Dec 31 ...  (last day of every month)

Step 1: Filter, so you only process what you need

A full export holds years of events, cancelled meetings and personal appointments next to work ones. Trim it first with ICS Filter:

• From / To dates, inclusive. An event is kept when any part of it touches those days.
• Keyword, matched case-insensitively in SUMMARY, DESCRIPTION and LOCATION (as one phrase, so "team lunch" won't match "lunch with the team").
• Category (comma-separated, any match) and Status (Confirmed, Tentative, Cancelled).
• Drop cancelled events is on by default.

Filters combine with AND. Two details matter for recurring events:

1. A series is kept or dropped as a whole. It stays if at least one occurrence (worked out from RRULE, RDATE and EXDATE) falls in the range, and the rule is copied unchanged.
2. You can download removed.ics as well, which is a record of exactly what the trim left out.

Because UIDs are unchanged, importing the filtered file back into the calendar it came from updates the existing events instead of duplicating them.

Step 2: Scrub before the file leaves your hands

If the file is going to someone else — a recruiter, an assistant, a developer reproducing a bug — run it through ICS Privacy Scrub. Each field gets a rule: Keep, Generalize or Remove. The defaults, as the page documents them:

Two switches are also on by default: New UIDs (so an imported copy doesn't overwrite the originals) and Mark private (CLASS:PRIVATE on every event). "Links" includes URL, CONFERENCE and the vendor meeting-link properties, so Meet and Teams links go too, not just Zoom.

A few things I like about how it's specified:

• With Generalize, attendees become numbered stand-ins (Attendee 1, the same address always gets the same number) and a link keeps only its scheme and host, such as https://zoom.us/.
• With New UIDs on, a recurring series keeps one shared UID, so the moved 22 October occurrence still attaches to its series.
• Shift dates by N days makes a demo calendar from a real one. Use a multiple of 7 for weekday meetings, or a Monday standup will drift to another weekday (the page warns you).

The page is honest about the limits: the result is not anonymous. With the defaults, the file still shows when you're busy, for how long, how often and in which category.

If all you want to share is availability, ICS Busy/Free is the stricter option. It writes either Busy-only events that keep their repeat rule, or a single VFREEBUSY block of busy periods in UTC (as RFC 5545 requires), with no per-field choices.

Step 3: Expand, when the destination can't read RRULE

A spreadsheet, a CRM or billing import, or an older importer doesn't read the rule, so it shows one row, or keeps only the first occurrence. ICS Expand RRULE turns each series into one event per occurrence inside a window, and writes .ics, JSON or CSV.

What it handles, per its documentation:

• FREQ=DAILY|WEEKLY|MONTHLY|YEARLY with INTERVAL, COUNT, UNTIL, BYDAY (including ordinals like 2TU and -1FR), BYMONTHDAY (negative counts from the month's end), BYMONTH and WKST.
• EXDATE removes, RDATE adds, and a RECURRENCE-ID component replaces the occurrence it names.
• Times keep their TZID or trailing Z; the arithmetic runs on wall-clock time, so the Berlin example stays at 09:30 across the change.
• A cap of 366 occurrences per series by default (max 2000) and 20,000 events in total. The notes say when a series was cut.

And what it doesn't do: a rule using BYSETPOS, BYYEARDAY, BYWEEKNO, BYHOUR, BYMINUTE, BYSECOND, or an hourly-or-finer frequency keeps only its first occurrence, and the notes name the part that wasn't expanded. Check the notes before trusting a count.

Two practical tips:

1. Import the expanded .ics into a separate calendar. Each occurrence gets its own UID (series UID plus the occurrence start), so importing it next to the original series duplicates every meeting.
2. CSV columns match ICS to CSV: summary, start, end, all_day, timezone, location, description, organizer, status, categories, rrule, uid, sorted by start. With "Skip cancelled series" on (the default), cancelled series and occurrences are left out and counted.

For the sample file and a window of 1–31 October, you should get the seven meetings listed above, with the 22 October one at 10:00.

Double-check the expansion in code

If you'd like to verify a tricky rule independently, python-dateutil does the core arithmetic. Here's the sample series, with the exclusion and the move applied by hand:

from datetime import datetime
from zoneinfo import ZoneInfo
from dateutil.rrule import rrulestr, rruleset

tz = ZoneInfo("Europe/Berlin")
start = datetime(2026, 10, 5, 9, 30, tzinfo=tz)

rs = rruleset()
rs.rrule(rrulestr("FREQ=WEEKLY;BYDAY=MO,WE;COUNT=8", dtstart=start))
rs.exdate(datetime(2026, 10, 14, 9, 30, tzinfo=tz))           # EXDATE

moved = {datetime(2026, 10, 21, 9, 30, tzinfo=tz):           # RECURRENCE-ID
         datetime(2026, 10, 22, 10, 0, tzinfo=tz)}           # new DTSTART

for occ in rs:
    occ = moved.get(occ, occ)
    print(occ.strftime("%a %d %b %H:%M %Z"), occ.astimezone(ZoneInfo("UTC")).strftime("%H:%MZ"))

Output:

Mon 05 Oct 09:30 CEST 07:30Z
Wed 07 Oct 09:30 CEST 07:30Z
Mon 12 Oct 09:30 CEST 07:30Z
Mon 19 Oct 09:30 CEST 07:30Z
Thu 22 Oct 10:00 CEST 08:00Z
Mon 26 Oct 09:30 CET 08:30Z
Wed 28 Oct 09:30 CET 08:30Z

That's a handy unit test for whatever does the expansion in your own pipeline. In practice the hard part is the plumbing around it: parsing folded lines, matching RECURRENCE-ID components to their series by UID, and handling UNTIL/EXDATE written in UTC for a zoned series.

Bonus: combining and comparing calendars

• Merge ICS folds up to twenty exports into one file (there's a guide). Its optional dedupe key is UID + RECURRENCE-ID, so a moved occurrence is never mistaken for a duplicate of its series.
• ICS Diff compares two exports by the same key and ignores DTSTAMP, LAST-MODIFIED and SEQUENCE by default, since apps rewrite them on every export.
• ICS Timezone Convert renames Outlook's Windows zone names (such as Eastern Standard Time) to IANA names, a common reason imported events show up an hour off.

The checklist

1. Filter to the date range, project or category you need. Keep removed.ics.
2. Scrub anything that leaves your hands. Use Busy/Free if availability is all they need.
3. Expand only for destinations that can't read RRULE, and import the result into a separate calendar.
4. Read the notes each tool prints: skipped months, capped series and unsupported rule parts are where silent errors hide.

A calendar export is a record of who you met, where, and how to join the call. Treat it like any other data export: trim it, redact it, and know exactly what you're handing over.

You could write a jq pipeline or a pandas script. Often, though, the person asking lives in a spreadsheet, and you need the answer in ten minutes. This post walks through a workflow I keep reusing. Every step runs in the browser with FindUtils tools, and each tool page says it processes the file in your tab and doesn't upload it:

1. NDJSON → CSV with nested fields flattened into columns

2. Dedupe by job_id, keeping the last event, which gives the latest state per job

3. Trim columns down to the ones that matter

4. Diff two runs by key, so you see added, removed and changed jobs

The sample log

Here is a small, made-up run-1.ndjson from a background export worker:

{"ts":"2026-10-12T09:00:01Z","job_id":"exp-101","status":"queued","attempt":1,"user":{"id":"u_17","plan":"pro"}}
{"ts":"2026-10-12T09:00:04Z","job_id":"exp-101","status":"failed","attempt":1,"user":{"id":"u_17","plan":"pro"},"error":{"code":"E_TIMEOUT"}}
{"ts":"2026-10-12T09:00:05Z","job_id":"exp-102","status":"done","attempt":1,"user":{"id":"u_42","plan":"free"},"tags":["csv","weekly"]}
{"ts":"2026-10-12T09:00:05Z","job_id":"exp-102","status":"done","attempt":1,"user":{"id":"u_42","plan":"free"},"tags":["csv","weekly"]}
{"ts":"2026-10-12T09:00:07Z","job_id":"exp-103","sta
{"ts":"2026-10-12T09:00:09Z","job_id":"exp-104","status":"failed","attempt":2,"user":{"id":"u_08","plan":"pro"},"error":{"code":"E_429"}}

It has all the usual problems: nested objects (user, error), an array (tags), a shipper retry that wrote line 4 twice, and a truncated line 5.

Why the usual tools stumble

• JSON.parse on the whole file fails. An NDJSON file isn't one JSON document, so parsing everything at once breaks after the first line. Each line has to be parsed on its own.

• A "JSON to CSV" converter that expects an array can't read it either. And a converter that does read it often puts user into a single cell as {"id":"u_17",...}.

• jq works, but you have to know the keys up front, and it stops at the first broken line:

jq -r '[.ts, .job_id, .status, .attempt, .user.id, (.error.code // "")] | @csv' run-1.ndjson
# prints rows 1-4, then: jq: parse error: Invalid string ... at line 6
# (the cut-off string runs into the next line). exp-104 is never printed.

That's fine for a quick look. It's less fine when line 5 is line 48,213 and you need every row after it.

Step 1: NDJSON → CSV, with nested fields as columns

Open the NDJSON CSV Converter and drop the file in. A few behaviours from its documentation matter here:

• Every line is parsed separately. A line that isn't valid JSON is skipped, counted, and listed in the notes as Line N: invalid JSON. The other lines still convert, so you lose one row instead of the whole job, and you know exactly which line to chase.

• The header is the union of every key in every row, in the order keys first appear. A key that only shows up on line 900 still gets a column.

• Flatten (the default) turns nested objects into dot paths (user.id, error.code) and arrays into indexed columns (tags[0], tags[1]). Choose JSON cell if you'd rather keep an object or array as JSON text in one cell.

• null and a missing key both become an empty cell. Numbers and booleans are written as text, and your spreadsheet decides the types when it opens the file.

• There are Separator (comma, semicolon, tab) and Excel BOM options for Excel setups that expect semicolons or show accented characters wrongly.

For the sample, you'd get:

ts,job_id,status,attempt,user.id,user.plan,error.code,tags[0],tags[1]
2026-10-12T09:00:01Z,exp-101,queued,1,u_17,pro,,,
2026-10-12T09:00:04Z,exp-101,failed,1,u_17,pro,E_TIMEOUT,,
2026-10-12T09:00:05Z,exp-102,done,1,u_42,free,,csv,weekly
2026-10-12T09:00:05Z,exp-102,done,1,u_42,free,,csv,weekly
2026-10-12T09:00:09Z,exp-104,failed,2,u_08,pro,E_429,,

The notes also tell you line 5 was skipped, so exp-103 is missing from this run. That's worth knowing before you report "3 jobs".

Big files: the converter's input cap is 10 MB. For a larger log, slice it first with standard tools and convert the slice:

# only the lines for one worker, or split into 50k-line parts
grep '"worker":"export"' app.ndjson > export-only.ndjson
split -l 50000 app.ndjson part-

Side note: when the source is a JSON array instead

If you're starting from an API response such as {"data":[...]} rather than a log file, use JSON Flatten. It finds the array under a data, results, items or records wrapper (or one level below, like data.items), and you can type any other location in Array path. Arrays can be joined with |, kept as JSON text, or spread into indexed columns, and a Depth setting limits how deep the dot paths go. One caveat from its tips applies to any JavaScript JSON parser: whole numbers above 9,007,199,254,740,991 lose precision, so have very long IDs sent as strings.

Step 2: keep the latest state per job (dedupe, keep last)

A log is a history. For "what state is each job in?", you want one row per job_id, specifically the last one.

Load the CSV into CSV Dedupe:

1. Pick job_id as the key column. With no key picked, only rows that match in every cell count as duplicates, so you'd catch the retry on line 4 and nothing else.

2. Under "When a row repeats, keep", choose Last row.

3. Leave Trim whitespace on (the default). Turn on Ignore case only if your IDs can differ in case.

The result:

ts,job_id,status,attempt,user.id,user.plan,error.code,tags[0],tags[1]
2026-10-12T09:00:04Z,exp-101,failed,1,u_17,pro,E_TIMEOUT,,
2026-10-12T09:00:05Z,exp-102,done,1,u_42,free,,csv,weekly
2026-10-12T09:00:09Z,exp-104,failed,2,u_08,pro,E_429,,

Two things make this safe to hand to someone else:

• You can download duplicates.csv as well as cleaned.csv. That's a record of every row that was removed, so you can check each decision.

• Kept rows stay in their original file order. "Keep last" means last in the file, so make sure the log is in time order (most shippers append in order, but merged files may not be).

Also note how blank keys behave: by default a row with an empty key is kept as unique. Ten events with no job_id won't collapse into one unless you choose Collapse into one.

Step 3: trim to the columns you want to compare

Before diffing two runs, drop the columns that change on every event but don't tell you anything, such as ts. Otherwise every job will look "changed". In CSV Column Extractor, keep job_id, status, attempt and error.code. Removing user.id here is also a small privacy win if the file is going outside your team.

Step 4: diff run 1 against run 2 by key

Do the same three steps for the re-run log (run-2.ndjson). Then open CSV Diff, put run 1 in File A (before) and run 2 in File B (after), and select job_id as the key column.

That last part matters. If you don't pick a key, the tool looks for a shared column named id, email, sku, uuid, key or code, and otherwise falls back to the first shared column whose values are unique in both files. job_id isn't on that list, so pick it yourself.

Because rows are matched by key rather than by line, a re-sorted file doesn't produce noise. For our two runs, the summary reads like this:

• Added: exp-103 (the job lost to the truncated line in run 1 completed in run 2)

• Changed: exp-101 (status failed → done, attempt 1 → 2, error.code cleared) and exp-104 (attempt 2 → 3, still E_429)

• Unchanged: exp-102

The csv-diff.csv download has a change_type column, the columns of File A plus any columns only File B has, and an old_ column for each column that changed somewhere (old_status, old_attempt, old_error.code). There's also a ZIP with added.csv, removed.csv and changed.csv, which is easy to paste into a ticket.

One tip from the tool's docs: pick a key that identifies the same real thing in both files. If you match on something that can itself change, an edited row turns into one "removed" plus one "added".

Step 5 (optional): back to NDJSON for a loader

If the cleaned table needs to go into something that reads JSON Lines, such as a warehouse loader or a fine-tune dataset, flip the same converter to CSV to NDJSON. Values stay strings unless you turn on Coerce types. With it on, true/false, null and plain numbers become JSON values, but a number with leading zeros like 007 stays text, and so does an integer of 2^53 or more. Dotted headers such as user.id stay flat keys. The tool deliberately doesn't rebuild nested objects.

Checklist

Logs carry request paths, user IDs and sometimes tokens. The tool pages say the files stay in your browser (you can confirm that in DevTools' Network panel while converting), but the CSV you export is a normal file. Redact it before it goes into a shared drive.

For more depth on each step, FindUtils has guides on NDJSON to CSV, deduping CSVs and diffing CSVs, plus a good read on why JSON → CSV → SQL loses data types.

This walkthrough takes one realistic JSON response and turns it into a Zod schema plus inferred TypeScript types in a few minutes. You will see the full schema, how z.infer replaces a duplicate interface, and how to call parse / safeParse in a fetch wrapper. A note on tools: if you prefer not to type the schema by hand, browser converters such as FindUtils' JSON to Zod Schema and JSON to TypeScript run locally in the tab and do not upload your paste — useful when the sample contains customer emails or staging tokens.

The sample payload

Imagine a billing API that returns a customer, their open invoices, and a nested address. Copy this into a file named sample.json (or keep it in a scratch note):

{
  "id": "cus_9f3a2c1b",
  "email": "ada@example.com",
  "created_at": "2026-09-12T14:22:01.000Z",
  "plan": "pro",
  "is_active": true,
  "address": {
    "line1": "42 Computing Lane",
    "city": "Istanbul",
    "country": "TR",
    "postal_code": "34000"
  },
  "invoices": [
    {
      "id": "inv_1001",
      "amount_cents": 2900,
      "currency": "usd",
      "status": "paid",
      "hosted_url": "https://pay.example.com/inv/1001"
    },
    {
      "id": "inv_1002",
      "amount_cents": 2900,
      "currency": "usd",
      "status": "open",
      "hosted_url": "https://pay.example.com/inv/1002"
    }
  ],
  "tags": ["beta", "eu"]
}

Why this shape matters for generators and hand-written schemas alike:

• Strings that look like email, URL, and ISO datetime should become refinements (z.string().email(), .url(), .datetime()), not bare z.string().
• Nested address becomes z.object({...}).
• invoices is an array of objects — z.array(z.object({...})).
• tags is a string array — z.array(z.string()).
• amount_cents is an integer; treat it as z.number().int() if your generator supports it.

If you also need a language-agnostic contract for non-TypeScript consumers, generate a JSON Schema first with a JSON Schema generator, then keep Zod as the TypeScript runtime layer. For the “what is JSON Schema even for?” background, the short guide JSON Schema validation explained is a useful companion.

Step 1 — Decide: types only, or types + runtime?

For an untyped third-party API, Zod + z.infer is usually the sweet spot: one source of truth for both layers.

Step 2 — Generate (or write) the Zod schema

Paste the JSON into a converter — for example JSON to Zod Schema — and enable format detection if available. You should get something close to:

import { z } from "zod";

export const InvoiceSchema = z.object({
  id: z.string(),
  amount_cents: z.number().int(),
  currency: z.string(),
  status: z.string(),
  hosted_url: z.string().url(),
});

export const AddressSchema = z.object({
  line1: z.string(),
  city: z.string(),
  country: z.string(),
  postal_code: z.string(),
});

export const CustomerSchema = z.object({
  id: z.string(),
  email: z.string().email(),
  created_at: z.string().datetime(),
  plan: z.string(),
  is_active: z.boolean(),
  address: AddressSchema,
  invoices: z.array(InvoiceSchema),
  tags: z.array(z.string()),
});

Zod 4 note: the top-level string formats z.email(), z.url() and z.iso.datetime() are now preferred. The method forms used here (z.string().email() and friends) still work but are deprecated, so use whichever matches the Zod version in your project.

Hand-tune after generation:

1. Tighten enums you know are closed: status: z.enum(["paid", "open", "void"]), plan: z.enum(["free", "pro", "enterprise"]).
2. Mark fields that are sometimes missing with .optional() or .nullable() — generators only see what you pasted.
3. Add .describe() or comments for teammates reviewing the PR.

Prefer types without runtime checks for a quick prototype? Use JSON to TypeScript for interfaces, then upgrade hot paths to Zod later. Many teams start with interfaces and promote boundary schemas once production traffic proves which fields lie.

Step 3 — Infer TypeScript types from the schema

Do not rewrite a parallel interface Customer. Derive it:

export type Invoice = z.infer<typeof InvoiceSchema>;
export type Address = z.infer<typeof AddressSchema>;
export type Customer = z.infer<typeof CustomerSchema>;

Now Customer["invoices"][number]["status"] stays in sync when you change the schema. That sync is a big reason many TypeScript teams moved from hand-maintained interface + Yup pairs to Zod.

Step 4 — Validate at the fetch boundary

async function fetchCustomer(id: string): Promise<Customer> {
  const res = await fetch(`https://api.example.com/customers/${id}`, {
    headers: { Authorization: `Bearer ${process.env.API_TOKEN}` },
  });

  if (!res.ok) {
    throw new Error(`HTTP ${res.status}`);
  }

  const json: unknown = await res.json();
  return CustomerSchema.parse(json); // throws ZodError on mismatch
}

Prefer not to throw? Use safeParse:

const result = CustomerSchema.safeParse(json);
if (!result.success) {
  console.error(result.error.flatten());
  throw new Error("Unexpected customer payload");
}
const customer: Customer = result.data;

Put this in one module (api/customers.ts). Callers get a typed Customer and never see raw any.

Step 5 — Wire it into forms and server routes (optional)

• React Hook Form: useForm({ resolver: zodResolver(CustomerSchema) }) via @hookform/resolvers.
• tRPC / Next.js Server Actions: CustomerSchema.parse(input) at the start of the procedure.
• Tests: assert fixtures with expect(() => CustomerSchema.parse(fixture)).not.toThrow().

The same schema that typed your client now guards the server.

A five-minute checklist

1. Capture one real JSON response (happy path + one edge case if you can).
2. Generate Zod (or write the outer z.object and nest inward).
3. Tighten enums, optionals, and URL/email/datetime refinements.
4. Export type X = z.infer<typeof XSchema>.
5. parse / safeParse in the single place that talks to the network.
6. (Optional) Keep a JSON Schema copy for OpenAPI or non-TS clients via a JSON Schema generator.

Common pitfalls

• Sample too small. If every invoice is "paid", a naive generator may not hint an enum — you still need domain knowledge.
• Numbers that arrive as strings. Some APIs send "2900". Prefer z.coerce.number().int() only when you intentionally accept coercion.
• Dates as strings vs Date. Zod's .datetime() validates ISO strings; use .transform((s) => new Date(s)) if you want Date objects downstream.
• Trusting as Customer. Type assertions silence the compiler without checking runtime shape — that is the bug Zod exists to catch.

Zod vs JSON Schema (quick orientation)

Teams often ask whether they still need JSON Schema if they adopt Zod. Short answer: they solve overlapping but different jobs.

• Zod lives in TypeScript. You validate at runtime and infer static types with z.infer. Ideal inside Node APIs, Next.js Server Actions, tRPC, and React forms.
• JSON Schema is language-agnostic JSON. Ideal when Python, Go, and TypeScript services must share one contract, or when an OpenAPI document is the source of truth.

A practical pattern: keep JSON Schema (or OpenAPI) as the cross-language contract, and generate or maintain a Zod schema for the TypeScript edge. Browser helpers for both sides — JSON Schema generator and JSON to Zod Schema — make the first draft cheap. For the conceptual background on drafts, $ref, and validation errors, see JSON Schema validation explained.

If your world is TypeScript-only for the next year, Zod alone is enough. Add JSON Schema when a second language or an external partner needs the same shape.

When browser converters help

Hand-writing Zod for a 200-field OpenAPI dump is miserable. Pasting a redacted sample into a client-side converter gets you 80% of the schema in seconds; you spend the remaining time on enums and business rules. FindUtils' developer converters (Zod, TypeScript, JSON Schema) are free, run in the browser, and are designed not to upload the JSON you paste on those pages — which matters when the sample is a staging customer record. Treat the output as a draft: always review before merging.

Wrap-up

Untyped JSON in, trusted Customer out: that is the whole job. Zod gives you runtime checks and z.infer types from one definition. Start from a real payload, generate or sketch the schema, tighten what the sample cannot teach you, and validate only at the boundary. Five minutes today saves a week of “why is email sometimes a number?” debugging later.

Want more free browser tools for JSON, APIs and everyday dev chores? They're all at findutils.com.

EmojiFavicons is a URL-based emoji favicon service from FindUtils. You point a normal <link rel="icon"> at an emoji slug, and the API returns an SVG (under 200 bytes for a plain emoji, per the docs). No signup, no API key, no build step. This article sticks to what the site documents and what the endpoints actually return when you request them.

The one-line install

Drop this in your HTML <head>:

<link rel="icon" href="https://emojifavicons.com/fire">

Replace fire with any emoji name. The homepage and docs state that EmojiFavicons covers 3,773 Unicode emojis, searchable by 11,550 keywords. Names and keywords are case-insensitive. You can also use a keyword (/hot resolves to fire), a URL-encoded emoji character, or a country code for flags (/US, /TR).

A live request to https://emojifavicons.com/fire returns image/svg+xml — a tiny SVG wrapping the 🔥 glyph. Safari is a special case: the how-it-works page explains that Safari has limited SVG favicon support, so the service auto-serves a PNG (Twemoji) for Safari User-Agents. You can force a format with ?svg or ?png.

If you later want a classic downloadable package instead of a live URL, the docs also expose GET /download/:emoji — for example https://emojifavicons.com/download/rocket returns a ZIP of standard sizes. That is the honest self-host path when you do not want a third-party dependency at runtime.

Customize with URL parameters

Every favicon endpoint accepts query parameters. A documented example that works today:

https://emojifavicons.com/fire?size=64&bg=1a1a1a&animate=pulse

That request returns image/svg+xml with a 64×64 viewBox, a dark background, and a CSS pulse animation. Useful parameters from the docs (not an exhaustive list):

Combine freely. Combos put 2–10 emojis side by side with a +:

<link rel="icon" href="https://emojifavicons.com/fire+rocket">

That endpoint returns a single SVG with 🔥 and 🚀 next to each other — useful for “shipping fast” vibes without designing an icon.

For a red unread count without JavaScript:

<link rel="icon" href="https://emojifavicons.com/bell?badge=3">

Framework installs (verified against the official guides)

The frameworks index lists 19 tutorials. Below are the four most common, copied from the site’s own guides. In every case, remove or rename any default favicon.ico / favicon.svg in public/ so the external URL wins.

Next.js (App Router)

From the Next.js guide:

// app/layout.tsx
import type { Metadata } from 'next';

export const metadata: Metadata = {
  icons: {
    icon: 'https://emojifavicons.com/rocket',
    apple: 'https://emojifavicons.com/apple-touch-icon/rocket',
  },
};

Pages Router: put the <link> tags in pages/_document.tsx inside <Head>. The guide notes that Vercel may cache an old favicon for up to 24 hours; append ?v=2 if you need a cache bust.

Astro

From the Astro guide, in your shared layout:

<head>
  <link rel="icon" type="image/svg+xml" href="https://emojifavicons.com/rocket" />
  <link rel="apple-touch-icon" href="https://emojifavicons.com/apple-touch-icon/rocket" />
</head>

Delete public/favicon.svg if the starter left one. Keep the tags in the persistent layout so View Transitions do not thrash the tab icon.

Vue 3 + Vite / Nuxt 3

From the Vue guide:

Vite: edit root index.html:

<link rel="icon" type="image/svg+xml" href="https://emojifavicons.com/rocket" />
<link rel="apple-touch-icon" href="https://emojifavicons.com/apple-touch-icon/rocket" />

Nuxt 3: in nuxt.config.ts:

export default defineNuxtConfig({
  app: {
    head: {
      link: [
        { rel: 'icon', type: 'image/svg+xml', href: 'https://emojifavicons.com/rocket' },
        { rel: 'apple-touch-icon', href: 'https://emojifavicons.com/apple-touch-icon/rocket' },
      ],
    },
  },
});

WordPress

From the WordPress guide, append to the child theme’s functions.php (or a site-wide snippets plugin):

add_action('wp_head', function () {
  echo '<link rel="icon" href="https://emojifavicons.com/rocket" type="image/svg+xml">';
  echo '<link rel="apple-touch-icon" href="https://emojifavicons.com/apple-touch-icon/rocket">';
});

Clear the Customizer Site Icon if WordPress still serves its own. Hard-refresh after saving.

Dynamic tab notifications with the JS SDK

Static badges are fine for marketing sites. Apps need the tab to react. Include the SDK once:

<script src="https://emojifavicons.com/sdk.js"></script>

https://emojifavicons.com/sdk.js returns application/javascript. The homepage and docs show the core surface; the script itself exposes:

EmojiFavicon.set('rocket');
EmojiFavicon.animate('fire', 'pulse');
EmojiFavicon.notify('bell', 3000); // flash, then restore
EmojiFavicon.restore();
EmojiFavicon.setBadge('bell', 5);
EmojiFavicon.clearBadge();

A practical pattern for “you have new messages while the tab is in the background”:

function onUnread(count) {
  if (count > 0) {
    EmojiFavicon.set('bell', { badge: Math.min(count, 99) });
  } else {
    EmojiFavicon.restore();
  }
}

notify(emoji, durationMs) temporarily swaps the icon, then restores the previous href — handy for a one-shot alert without managing state yourself.

Trade-offs (be honest)

Depends on a third-party URL. If emojifavicons.com is unreachable, the tab falls back to whatever the browser does without a favicon. The compare page itself lists this as a con of URL-based services, mitigated by CDN caching (docs: 7-day CDN cache, 24-hour browser cache) and by downloading the SVG/PNG/ICO (or the ZIP from /download/:emoji) and hosting it yourself.

Rate limits. The FAQ on the homepage says normal use has a 300 requests per minute limit, with no signup or credit card required. That is generous for a favicon, but it is still a shared free resource — do not hammer it from a CI loop that rebuilds thousands of pages per minute.

Not a full custom-image favicon studio. If you need a hand-drawn mark or a photo crop, generate and self-host assets instead. For that workflow, FindUtils’ favicon generator is a useful browser-side companion when you already have an image file. EmojiFavicons and the FindUtils generator solve different problems: emoji-as-URL vs. image-to-package.

Caching surprises. CDNs and browsers cache favicons aggressively. After a change, hard-refresh or bump a query string.

Quick health check

Before you ship, run your domain through the Favicon Health Checker. It scores icon / SVG / apple-touch / manifest coverage and offers copy-paste fixes. Pair that with the interactive generator if you want to preview size, shape, animation, and badge before pasting a URL into production.

Wrap-up

1. Add <link rel="icon" href="https://emojifavicons.com/fire"> (or your emoji of choice).

2. Optionally tune with ?size=, ?bg=, ?animate=, ?badge=, or a combo like /fire+rocket.

3. Follow the framework guide for Next.js, Astro, Vue, or WordPress.

4. For runtime swaps, load sdk.js and call set / animate / notify.

5. If you outgrow a third-party URL, download the ZIP and self-host.

Start at emojifavicons.com, browse the docs, and if you need more free browser-only utilities beyond favicons, FindUtils has a large catalog of client-side tools that never upload your files.

This guide walks through the triage in the order I'd actually do it. No terminal, no paid dashboards. Just evidence first, then fixes.

Step 0: Get one real message that went to spam

Don't start by guessing at DNS. Start with a message that actually failed. Ask the recipient to forward it as an attachment, or send a test to a Gmail or Outlook account you control and find it in Junk.

Then open the full headers ("Show original" in Gmail, "View message source" in Outlook). This raw text is the single most useful thing you'll look at today, because it contains the receiving server's own verdict.

Step 1: Read the headers before touching anything

Paste the raw headers into the Email Header Analyzer. It traces the routing path hop by hop, lists the IP address of each server along the way, and pulls out the SPF, DKIM and DMARC results.

Look for the Authentication-Results line. You'll typically see something like spf=pass, dkim=fail, dmarc=fail. That tells you where to dig:

• All three pass, still spam? The problem is probably reputation or content, not authentication. Skip ahead to Steps 5 and 6.
• SPF fails: the sending IP isn't authorized for your domain.
• DKIM fails or is missing: the message wasn't signed, or the signature doesn't match the published key.
• DMARC fails: neither SPF nor DKIM passed in alignment with the From domain (more on that below).

Also write down the IP of the first external hop, the server that actually handed the mail to the internet. You'll need it for the PTR and blocklist checks.

Step 2: Check SPF

What it is: SPF (Sender Policy Framework) is a TXT record on your domain that lists which servers are allowed to send mail for it. It looks like this:

v=spf1 include:_spf.google.com include:sendgrid.net -all

Run your domain through the Email Security Checker, which inspects SPF, DKIM, DMARC, MTA-STS, TLS-RPT and BIMI in one pass. If you want to see the raw records yourself, the DNS Lookup tool will show every TXT record on the domain.

Common SPF mistakes:

• Two SPF records. You're allowed exactly one TXT record starting with v=spf1. Adding a second one when you sign up for a new service is the most common way people break SPF. Merge them into one.
• Too many lookups. SPF has a limit of 10 DNS lookups (each include, a, mx and redirect counts, and nested includes count too). Go over and receivers return a permerror, which often behaves like a fail.
• Forgetting a sender. Your newsletter tool, your CRM, your helpdesk, the billing system: every service that sends as your domain needs to be covered, or needs to send from a subdomain with its own record.
• Using +all. This authorizes the entire internet. Never do it. Use ~all (soft fail) while testing and -all once you're confident.

Step 3: Check DKIM

What it is: DKIM adds a cryptographic signature to each message. The receiving server fetches your public key from DNS at selector._domainkey.yourdomain.com and verifies the signature.

The selector is in the DKIM-Signature header of your message (the s= value). Look up that exact name as a TXT record with DNS Lookup, or let the Email Security Checker find it.

Common DKIM mistakes:

• Signing is never turned on. Many providers require you to publish the key and click an "enable" or "authenticate" button. Publishing the record alone does nothing.
• Broken key formatting. Long keys are sometimes split into multiple strings, and some DNS panels add stray quotes or line breaks. A single missing character invalidates the key.
• Signing with the provider's domain. If the signature says d=mailprovider.net instead of d=yourdomain.com, DKIM may "pass" but won't align for DMARC.

Step 4: Check DMARC

What it is: DMARC is a TXT record at _dmarc.yourdomain.com that tells receivers what to do when a message fails authentication, and where to send reports:

v=DMARC1; p=none; rua=mailto:dmarc-reports@yourdomain.com

The key idea is alignment. DMARC passes only if SPF or DKIM passes and the domain they validated matches the domain in the visible From address. This is why a message can show spf=pass and still fail DMARC: SPF validated your email provider's bounce domain, not yours.

Common DMARC mistakes:

• No record at all. Major mailbox providers now expect bulk senders to have DMARC. Even p=none is better than nothing.
• Jumping straight to p=reject. If you haven't confirmed every legitimate sender aligns, you'll block your own mail. Start at none, read the reports, move to quarantine, then reject.
• Reports going nowhere. A rua address that no one reads means you're flying blind.

For a broader look at your domain's email-related DNS (MX, SPF, DKIM, DMARC together), the DNS Security Scanner gives a useful second opinion.

Step 5: Check reverse DNS (PTR)

What it is: A PTR record maps an IP address back to a hostname. Receiving servers check that your sending IP has one, and ideally that the hostname resolves back to the same IP (forward-confirmed reverse DNS).

Take the sending IP from Step 1 and run it through the Reverse DNS Lookup.

• If you use Google Workspace, Microsoft 365, or a major sending service, PTR is their job, and it's almost always fine.
• If you run your own mail server or send from a VPS, a missing or generic PTR (something like 123-45-67-89.static.isp.net) is a classic spam trigger. PTR records are set by whoever owns the IP block, so you'll usually configure it in your hosting provider's panel, not your DNS provider's.

Step 6: Check blocklists

If authentication passes and PTR looks right, reputation is the next suspect. Put the sending IP into the IP Blacklist Checker, which queries five DNS-based blocklists (Barracuda, SpamCop, UCEPROTECT, JustSpam and PSBL) in one go.

Be honest with yourself about the results:

• On a shared sending service? A listing on a shared IP is the provider's problem to solve. Open a ticket with them.
• On your own server? Find out why before requesting delisting. A compromised account, an open form that sends mail, or a purchased list will get you relisted within days.
• Clean everywhere and still in spam? Five lists aren't the whole picture. Big providers like Gmail and Microsoft keep their own internal reputation data, and content, engagement and sending volume all matter. Check their postmaster tools for your domain.

Step 7: Re-check after you change DNS

DNS changes aren't instant. After editing a record, use the DNS Propagation Checker to confirm resolvers around the world see the new value before you conclude a fix "didn't work." Then send a fresh test message and read the headers again. That's the only verdict that really counts.

The quick checklist

• Got the raw headers from a real message that went to spam
• Read Authentication-Results: which of SPF, DKIM, DMARC failed?
• Exactly one SPF record, under 10 lookups, covering every sender, ending in ~all or -all
• DKIM enabled at every sending service, signing with your domain (d=yourdomain.com)
• DMARC record exists at _dmarc, with a rua address someone actually reads
• SPF or DKIM aligns with the From domain
• Sending IP has a PTR that resolves back to the same IP
• Sending IP isn't on any blocklist (and you know why if it was)
• DNS changes have propagated before you re-test
• Sent a fresh test and confirmed the new headers say pass

A final word

Deliverability problems feel mysterious, but they rarely are. In most cases the headers tell you exactly what failed, and the fix is a single DNS record: a duplicate SPF entry, a DKIM key that never got enabled, a missing DMARC policy. Work through it in order, change one thing at a time, and verify with a real message after each change.

All the tools linked above are part of FindUtils, a collection of 463 free tools that run in your browser. No sign-up, no installs.