Recover File Names and Lines from Mono Stack Traces

mono-symbolicate turns a Mono stack trace full of <filename unknown>:0 into one with real file names and line numbers.

The examples use mono-symbolicate from mono-devel 6.8.0.105+dfsg-3.6ubuntu2, the version installed on this machine. Allow about fifteen minutes.

• You need: a shell, a stack-trace text file, and the .exe or .dll files built with their managed debugging symbols.
• No sudo unless your input or symbol files are genuinely unreadable.
• Watch out: do not copy sensitive production traces into a shared temporary directory.

1. Confirm the installed command

Check which executable will run and record the package version. These are read-only commands:

$ command -v mono-symbolicate
/usr/bin/mono-symbolicate
$ dpkg-query -W -f='${Package} ${Version}\n' mono-devel
mono-devel 6.8.0.105+dfsg-3.6ubuntu2
$ mono-symbolicate --help
Usage: symbolicate [options] <msym dir> <input file>
       symbolicate [options] store-symbols <msym dir> [<dir>]+

Available options:
  -h, --help                 Show this help
  -q                         Quiet, warnings are not displayed
  -v                         Verbose, log debug messages

The help text calls the operation symbolicate, but the installed command name is mono-symbolicate. Its two modes are easy to confuse: one reads a stack trace, while store-symbols prepares a symbol directory from directories containing managed assemblies.

2. Keep the original trace and inspect its shape

Work on a copy, particularly when a trace may contain customer data, paths or arguments. The tool writes the converted text to standard output, so the input is never modified.

$ cp --preserve=all /path/to/stacktrace.txt /path/to/stacktrace.txt.original
$ sed -n '1,40p' /path/to/stacktrace.txt
at Example.Worker:Run () [0x00000] in <filename unknown>:0

That placeholder is the useful symptom: Mono has a managed frame but cannot show its source location from the information currently available. Keep the trace as text. Do not pass a directory where a file is expected, and do not expect this tool to repair native frames or reconstruct symbols that were never generated.

3. Build a separate symbol directory

Make an empty directory for the symbol store, then add the directories containing the matching managed assemblies:

$ mkdir -p /path/to/msym
$ mono-symbolicate store-symbols /path/to/msym /path/to/release/bin /path/to/release/plugins
$ find /path/to/msym -maxdepth 2 -type f -print

The command accepts one or more source directories after store-symbols. Supply directories, not individual assembly paths, and make sure the assemblies match the build that produced the trace. A same-named .dll from another build can be worse than no symbol at all, because it can lead you to trust an incorrect file and line.

This step writes into /path/to/msym. Treat that directory as disposable derived data. To start again, stop the command first, move the old directory aside, and create a new one rather than mixing releases:

$ mv /path/to/msym /path/to/msym-old
$ mkdir /path/to/msym

That move is reversible until you deliberately remove the old directory, and no elevated privilege is needed when the paths belong to you.

4. Symbolicate to standard output

Pass the symbol directory first and the trace file second. Save the output to a new file so a failed or incomplete run cannot overwrite the original:

$ mono-symbolicate /path/to/msym /path/to/stacktrace.txt > /path/to/stacktrace-symbolicated.txt
$ test -s /path/to/stacktrace-symbolicated.txt && sed -n '1,40p' /path/to/stacktrace-symbolicated.txt
at Example.Worker:Run () [0x00000] in /srv/build/Worker.cs:42

The exact frame text depends on the assembly, source paths and symbols. What matters is that a matching frame can now show a file and line instead of <filename unknown>:0. Frames without a match stay unresolved, and a successful process exit is not a promise that every frame was symbolicated.

Compare the two files rather than deleting the source trace:

$ diff -u /path/to/stacktrace.txt /path/to/stacktrace-symbolicated.txt

5. Diagnose unresolved frames

First check the basics without changing anything: the input is readable, the symbol directory exists, and its contents came from the same build:

$ test -r /path/to/stacktrace.txt && echo 'trace readable'
trace readable
$ test -d /path/to/msym && find /path/to/msym -type f | head
$ ls -l /path/to/release/bin

If a file is missing, fix the path rather than adding sudo. On the installed version, a missing input file produces a non-zero status and a .NET file-not-found exception: that is an input problem, not evidence the symbols are invalid.

Use -v when you need debug messages, or -q when a script must suppress warnings:

$ mono-symbolicate -v /path/to/msym /path/to/stacktrace.txt > /path/to/stacktrace-symbolicated.txt
$ mono-symbolicate -q /path/to/msym /path/to/stacktrace.txt > /path/to/stacktrace-symbolicated.txt

Do not use -q while investigating a failed match: it hides warnings that may explain what the tool could not load. Do not treat -v output as part of the symbolicated trace either. The trace itself is written to standard output, so keep diagnostics separate if you capture both streams.

6. Automate only after a manual match

Once the manual result is correct, a script can process a batch of traces. Use a temporary destination and replace the final report only after the command succeeds:

input=/path/to/stacktrace.txt
output=/path/to/stacktrace-symbolicated.txt
temporary="${output}.new"

if mono-symbolicate /path/to/msym "$input" > "$temporary"; then
    mv -- "$temporary" "$output"
else
    status=$?
    printf 'symbolication failed with status %s\n' "$status" >&2
    rm -f -- "$temporary"
    exit "$status"
fi