UnicodeDecodeError: 'utf-8' codec can't decode byte

What this error means

You asked Python to turn bytes into text using UTF-8, and it met a byte sequence that is not valid UTF-8. Decoding is a translation with strict rules: UTF-8 encodes every character as a specific pattern of one to four bytes, so a stray byte breaks the pattern and Python stops rather than guessing.

Traceback (most recent call last):
  File "load.py", line 2, in <module>
    text = f.read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Cause 1 — the file is not text at all

The clue is in the byte. 0xff at position 0, or 0xd8 0xff, is the magic number of a binary format — an image, a PDF, a zip, an Excel workbook. Reading it as text can never work, whatever the encoding:

with open("photo.jpg", "rb") as f:      # 'rb' — binary, no decoding
    data = f.read()

print(data[:8])                          # b'\xff\xd8\xff\xe0...' — a JPEG

# and if it is a spreadsheet, use a library rather than open()
import csv
with open("data.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.reader(f))

Cause 2 — the text is real, but not UTF-8

Windows-created files are often cp1252 (Western European), older exports are latin-1, and files from Windows consoles or PowerShell are frequently utf-16. Decode with the encoding the file actually uses:

for enc in ("utf-8", "utf-8-sig", "cp1252", "latin-1", "utf-16"):
    try:
        text = open("notes.txt", encoding=enc).read()
        print(f"{enc} worked, {len(text)} chars")
        break
    except (UnicodeDecodeError, UnicodeError) as e:
        print(f"{enc}: {e}")

utf-8-sig is worth trying early: it is UTF-8 with a byte-order mark, which Windows Notepad and Excel add by default. Decoding such a file as plain utf-8 leaves a stray \ufeff at the start, which then breaks json.loads and header matching.

Cause 3 — you never said which encoding, so Python guessed

open() without an encoding argument uses your platform default — locale.getpreferredencoding(). That is UTF-8 on Linux and macOS but often cp1252 on Windows, which is why the same script works on one machine and fails on another. Always pass encoding explicitly:

import locale, sys
print(sys.getdefaultencoding(), locale.getpreferredencoding())

with open("data.csv", encoding="utf-8") as f:   # never rely on the default
    text = f.read()

The errors= escape hatch — and when not to use it

text = open("log.txt", encoding="utf-8", errors="replace").read()   # bad bytes -> \ufffd
text = open("log.txt", encoding="utf-8", errors="ignore").read()    # bad bytes dropped

with open("log.txt", encoding="utf-8", errors="strict") as f:       # the default
    ...

# keep going but know exactly what happened
for lineno, line in enumerate(open("log.txt", encoding="utf-8", errors="replace"), 1):
    if "\ufffd" in line:
        print(f"line {lineno} had undecodable bytes")

errors="replace" is right for logs and best-effort display. It is wrong for data you will store, compare or re-encode — the replacement character silently corrupts your content. For real data, find the correct encoding instead.

How to debug it

1. Read the reported byte and position. 0xff or 0xd8 at position 0 means a binary file, not a mislabelled text file.
2. Inspect the first bytes: open(path, 'rb').read(16). \xef\xbb\xbf is a UTF-8 BOM, \xff\xfe is UTF-16, PK is a zip, %PDF is a PDF.
3. Try the encodings in the loop above. latin-1 never raises, so treat it as a last resort rather than a diagnosis.
4. Always pass encoding= to open(). Relying on the platform default is the single biggest cause of 'works on my machine'.
5. For unknown encodings in bulk, use the charset-normalizer or chardet package to detect before decoding.

Related messages worth knowing

• FileNotFoundError — the other half of file handling — the file is not there at all.
• SyntaxError: invalid character in identifier — the source-code cousin — wrong characters in your .py, not your data.
• ValueError: invalid literal for int() — the next error in line once the text decodes but a field is not a number.