Invisible Unicode Is Easy to Flag—Blanket Blocking Can Break Valid Text

The safe approach is to scan decoded Unicode code points before security-sensitive parsing, then apply category-specific rules to a separate inspection view. Preserve the original input; do not make blanket deletion the default for every invisible, directional, or visually confusable character.

These categories are not interchangeable. Tag characters can conceal text from literal matching, joining controls can be required by writing systems or emoji sequences, bidirectional controls affect display order, and confusables are visible characters that resemble others. Treating them as one blocklist can corrupt valid international content.

Classify the input before changing it

Begin with a scalar-value check for the Tags block, U+E0000 through U+E007F. The Unicode 3.1 specification describes 95 special-use characters for spelling ASCII-derived tag values, plus LANGUAGE TAG and CANCEL TAG; it also says general language-tagging use is strongly discouraged outside supporting protocols. Because the block lies outside the Basic Multilingual Plane, UTF-16 represents its characters with surrogate pairs.

A range hit is easy to flag but is not proof of abuse. A Microsoft hunting signature initially flagged legitimate messages containing the subdivision flags of England, Scotland, and Wales because those emoji sequences use tag characters; the company’s analysis of tag-based phishing evasion therefore pairs pre-detection stripping or folding with exceptions for known legitimate sequences.

Iterate over Unicode scalar values rather than bytes or isolated UTF-16 code units. Record both the scalar position and the source offset required by your runtime so that findings remain useful without misidentifying half of a surrogate pair.

Build a detector that reports categories

The detector should describe what it found before policy decides whether to allow, transform, quarantine, or reject the value. Write hidden test characters as escaped code points so reviewers can see the relevant input.

1. Decode bytes strictly with the declared encoding. Reject or separately report malformed sequences instead of silently replacing them before inspection.
2. Retain the original value, or a retention-policy-compliant reference to it, without modifying it.
3. Walk through Unicode scalar values and record code point and position for each finding.
4. Classify any value from 0xE0000 through 0xE007F as TAG.
5. Independently classify joining and zero-width characters, bidirectional controls, and other default-ignorable characters covered by the field’s policy.
6. Create a derived security view and apply only the transformations defined for that field.
7. Send the derived view—not an uninspected copy—to keyword matching, URL parsing, authorization logic, tokenization, or model ingestion.
8. For identifiers, run script restrictions and confusable comparison separately from invisible-character detection.

A language-neutral interface can be expressed as inspect(raw) → {rawReference, findings, securityView, decision}. The decision needs the field type and sequence context: an isolated tag character inside a keyword is different from the same range appearing in a complete, supported subdivision-flag sequence.

Use a separate policy for each risk

• Tag characters: report every occurrence. Strip or fold them in the derived view used for content matching, while allowing complete documented sequences only where the product supports them. Strict identifiers can reject unexpected or malformed tag runs.
• Zero-width and joining characters: do not delete them globally. ZWJ and ZWNJ can affect shaping and orthography, and ZWJ participates in emoji sequences. General multilingual text normally needs a broader policy than usernames, host-like labels, or authorization tokens.
• Bidirectional controls: report the exact control and its position. Permit directional formatting only where the field expects it and the sequence is structurally valid; unexpected overrides or unmatched controls in security identifiers should ordinarily trigger rejection or quarantine rather than silent deletion.
• Confusables: compare identifiers instead of treating visual similarity as invisibility. The Unicode Security Mechanisms standard defines confusable skeletons and script-based checks, while warning that skeleton results are internal comparison forms and are unsuitable for display.

This separation avoids the destructive shortcut of converting every value to ASCII. A narrowly defined security identifier may justify a restricted repertoire; a personal name, address, or customer message generally requires rules that preserve its supported scripts.

Test detection and normalization order

Detection and Unicode normalization answer different questions. NFC can align canonically equivalent spellings, while compatibility normalization can change additional distinctions; neither should be treated as a universal invisible-character filter. Scan the decoded input first, normalize a copy according to the field contract, apply the explicit security transformation, and inspect the exact value delivered to the downstream parser.

Use visible escapes in fixtures:

• Plain multilingual text: “İstanbul 東京 مرحبا” should remain unchanged and produce no TAG finding.
• Inserted tag: “fun\u{E0020}ding” should produce a TAG finding before matching; a tag-removing inspection view should expose “funding.”
• Canonical equivalence: composed “café” and its canonically equivalent decomposed form should compare as the field requires without deleting unrelated letters or marks.
• Documented exception: a complete supported subdivision-flag sequence should be classified as allowed, while a detached tag character or incomplete sequence remains anomalous.
• Confusable identifier: Latin “paypal” and a similar mixed-script value containing Cyrillic “а” should reach the confusable check even though neither depends on an invisible character.
• Directional formatting: a permitted, balanced isolate sequence and an unmatched override should produce different decisions.

Assert the result at every boundary: raw scan, normalized copy, security view, and downstream parser input. Pin the Unicode data version used in production and rerun the fixture corpus when the runtime or security tables change.

Log evidence without replaying the payload

Make findings reviewable without letting a log viewer reinterpret hidden controls. Record the category, escaped code point such as U+E0020, scalar offset, field name, relevant script information, policy version, and final action. Escape every control in excerpts, cap excerpt length, and avoid retaining sensitive message bodies solely to explain a match.

Aggregate counts by category and decision, including use of documented exceptions. A rise in tag findings is useful telemetry, but it should remain one signal alongside authentication, sender reputation, URL analysis, and rate patterns.

The implementation boundary is precise: preserve legitimate input, classify hidden or confusable content, and derive a view tailored to the security decision. Flag broadly, transform narrowly, and reject only when the field’s documented contract forbids the detected character or sequence.

Also read:

• 3 Easy Steps to Improve Personal Online Security
• Radar Charts Show the Profile—But Exact Comparisons Can Break Down
• API Security’s Hidden Gap: Valid Logins Can Still Expose Business Data