HTML entities vs UTF-8, and when escaping stops XSS

HTML entities such as &, é and — date from a time when web pages often could not store characters outside ASCII. Today almost every page is UTF-8, so most entities are optional, but a few escapes are essential, and getting them wrong is the root of cross-site scripting (XSS). This article separates the two, shows how browsers really decode references (with some surprising rules), and marks the places where HTML escaping is not enough. Every example can be pasted into the HTML entity encoder.

Entities were an encoding workaround

A character reference names a character by its Unicode code point (é or é) or by a name from the HTML table (é). On a page saved as ISO-8859-1, "é" could be stored directly, but "—" or "€" could not, so authors wrote — and €. With UTF-8, declared by <meta charset="utf-8">, every Unicode character can be stored directly: é, —, €, ạ and 😀 need no entity at all.

So the first rule is simple: save files as UTF-8 and write characters as themselves. It keeps the source readable and the file smaller (é is 8 bytes, é is 2).

The escapes that are not optional

Some characters mean something to the HTML parser, and must be escaped when you want them as text:

Example: the text Tom & Jerry <3 becomes Tom & Jerry <3. Without the escapes, <3 begins what the parser takes for a tag and the rest of the line can vanish, and & Jerry happens to survive only because it is not a reference.

How browsers decode references

The HTML specification describes decoding precisely, including some old compatibility rules:

• Longest match. ∉ is ∉. &notit; is not a name, so the parser takes the longest name that matches, the legacy &not, and outputs ¬it;.
• Legacy names without a semicolon. 106 names from HTML 4 (the Latin-1 set plus &AMP, &COPY, &GT, &LT, &QUOT and &REG) are recognised even without ;. So &copy2024 in text shows ©2024. Names added later, such as &hellip or &euro, need the semicolon.
• Attribute exception. In an attribute value, a legacy name without a semicolon is left alone when the next character is = or a letter or digit. This rule exists so that links such as href="?a=1&copy=2" keep working; in body text the same characters would display ©=2. It is also why you should still write & between URL parameters in HTML.
• Numeric oddities. �, surrogates (�) and numbers above U+10FFFF become U+FFFD, the replacement character. € to Ÿ are mapped to the Windows-1252 characters they meant on old pages: – is an en dash, € the euro sign.

The tool's decoder follows these rules, and its "Attribute value" option shows the difference.

Escaping and XSS

Cross-site scripting happens when text from a user is inserted into a page and the browser treats part of it as markup or code. Escaping the characters above when you put text into HTML content or a quoted attribute value prevents that: <script>alert(1)</script> becomes visible text instead of a script.

It does not protect you in other contexts:

• Inside `<script>`: the HTML parser does not decode entities there, and JavaScript has its own string rules. Use JSON serialisation (and escape </).
• In URLs (href, src): javascript:alert(1) contains no special HTML characters at all. Validate the scheme (allow https: and mailto:), then percent-encode with the URL encoder.
• In event handler attributes (onclick) and `style`: the value is decoded, then run as JavaScript or CSS. Avoid putting user data there.
• Unquoted attributes: a space ends the value, so class=x onmouseover=... injects a new attribute. Always quote attributes.

In application code, rely on your template engine's automatic escaping (React, Vue, Jinja, Razor and others escape by default) and on a Content Security Policy as a second layer. Encode by hand only for one-off content, emails and debugging.

Common mistakes

• Double escaping. Escaping text that is already escaped turns & into &, which then shows as & on the page. Escape once, at output time.
• Escaping on input. Store the original text; escape when you insert it into HTML. The same text may go to JSON, CSV or an email next.
• Encoding emoji as two references. �� encodes the two UTF-16 halves of 😀 separately; browsers show two replacement characters. Use one reference for the code point: 😀.
• Trusting `&apos;` everywhere. It is valid HTML5, but not HTML 4 and not every older system knows it. ' is safe everywhere.
• Pasting from a word processor and getting   or invisible characters in names and URLs. Decoding shows what is actually there.

Try it

Paste a snippet into the HTML entity encoder, switch between "Page text" and "Attribute value" when decoding a URL with &copy=, and encode some text with an emoji to see a single reference per code point, along with the matching JavaScript and CSS escapes.