Architecture

This is the comprehensive technical reference for Discord Ferry's internals. It covers every module, data model, async pattern, and design decision in the codebase. If you want to contribute code, understand a bug, or build a new integration, start here.

For a quick introduction to what Ferry does and how to use it, see the Getting Started guides instead.

Overview

Discord Ferry is a Python 3.11+ migration tool that moves a Discord server to Stoat (formerly Revolt). It reads DiscordChatExporter (DCE) JSON exports, transforms them into Stoat API calls, and sends everything, messages, channels, roles, emoji, attachments, permissions, to the target Stoat instance.

Tech stack: aiohttp (HTTP client), NiceGUI (web GUI), Click (CLI), Rich (formatted terminal output), ijson (streaming JSON), pytest + pytest-asyncio (tests), ruff (lint/format), mypy strict (types), PyInstaller (binary packaging).

Terms used in this reference

The Python package registry is the Python Package Index (PyPI).
The tool-download stage is called EXPORT. The export-checking stage is called VALIDATE. CONNECT means the API connection stage.
SERVER means server creation.
ROLES means role creation. Category creation is called CATEGORIES.
CHANNELS means channel creation. Emoji copying is called EMOJI. Avatar copying is called AVATARS.
MESSAGES means message copying.
REACTIONS means reaction copying.
PINS means pin copying.
REPORT means report writing.
TextChannel means a text server-channel record.
VoiceChannel means a voice record. NSFW means not safe for work.
POST means the resource-creation web method.
PATCH means the partial-update method.
MigrationEvent means a progress record.
A web-service interface is known as REST. TUI means a terminal user interface.
A cache can discard its least recently used entry, known as LRU. A cache expiry period is called a time to live.
TTL stands for time to live.
The Python web framework is called FastAPI. Estimated time of arrival is abbreviated ETA.
ZIP means a compressed archive. .NET means Microsoft's software runtime. SDK stands for software development kit.
Discord's all-access permission is called ADMINISTRATOR.

Core design principle: one engine, two shells. The migration engine (core/engine.py) contains all logic. It never imports from the GUI or CLI. The GUI and CLI are thin wrappers that configure the engine, subscribe to its event stream, and render progress in their own way.

CI / Build Tooling

All Ferry workflows run on the Node 24 GitHub Actions runtime as of v2.1.2, ahead of GitHub's June 2026 Node 20 retirement. Action pins follow the lowest-Node-24-major principle (for example, actions/checkout v6 and actions/upload-artifact v6) to minimise behavior delta versus the previous Node 20 versions. The pypa/gh-action-pypi-publish step is Docker-based and unaffected by Node runtime deprecation. Every remote action is pinned to a full commit SHA with a trailing # vX.Y.Z comment (#1003), so the major named in the comment is the one in use. Dependabot's github-actions ecosystem proposes the SHA bumps, and tests/test_packaging.py fails on an unpinned reference.

The auto-tag.yml → release.yml cascade, tag pushes from auto-tag.yml automatically firing release.yml to build binaries and publish to PyPI, is enabled (as of v2.1.3) by a GitHub App (Ferry Auto-Tag Bot, owned by nordscope-fi, installed on this repo only with Contents: Read and write permission). The App's credentials (AUTO_TAG_APP_CLIENT_ID variable + AUTO_TAG_APP_PRIVATE_KEY secret) live in the auto-tag GitHub Environment. At runtime, auto-tag.yml mints a ~1-hour installation token via actions/create-github-app-token@v3 and passes it to actions/checkout@v6's token: input, the resulting tag push originates from the App identity (not GITHUB_TOKEN), so it triggers downstream workflows. If the App is deleted or the credentials revoked, auto-tag.yml stops starting the release workflow. The manual git push --delete origin v<tag> plus re-push protocol remains the fallback.

Project Layout

src/discord_ferry/
├── __init__.py              # Package version (__version__)
├── config.py                # FerryConfig dataclass, all runtime settings
├── state.py                 # MigrationState dataclass, ID maps, counters, resume checkpoints
├── errors.py                # Custom exception hierarchy (FerryError base)
├── reporter.py              # Post-migration report generator
├── review.py                # Pre-creation review summary builder
├── blueprint.py             # Server blueprint export/import
├── gui.py                   # NiceGUI web interface (4-screen workflow)
├── cli.py                   # Click CLI (migrate, validate, build, export-blueprint, rollback, stats, check, repair, retry, probe, tls-check)
│
├── core/
│   ├── engine.py            # Migration orchestrator, runs all 13 phases
│   ├── events.py            # MigrationEvent dataclass and EventCallback type
│   └── security.py          # SecureTokenStore, token masking and sanitization at output boundaries
│
├── parser/
│   ├── models.py            # 12 DCE dataclasses (DCEExport, DCEMessage, etc.)
│   ├── dce_parser.py        # JSON parsing, validation, streaming
│   └── transforms.py        # Content transforms (mentions, emoji, embeds, stickers)
│
├── migrator/
│   ├── api.py               # Stoat REST API wrapper with retry + rate limit handling
│   ├── connect.py           # Phase 2: CONNECT, test credentials, discover Autumn URL
│   ├── structure.py         # Phases 3–6: SERVER, ROLES, CATEGORIES, CHANNELS
│   ├── emoji.py             # Phase 7: EMOJI, extract, upload, register
│   ├── avatars.py           # Phase 7.5: AVATARS, pre-flight avatar download + Autumn upload
│   ├── messages.py          # Phase 8: MESSAGES, 9-step per-message pipeline
│   ├── reactions.py         # Phase 9: REACTIONS, apply queued reactions
│   └── pins.py              # Phase 10: PINS, re-pin messages
│
├── uploader/
│   └── autumn.py            # Autumn (Stoat file storage) upload client with caching
│
├── discord/
│   ├── __init__.py          # fetch_and_translate_guild_metadata() orchestration
│   ├── client.py            # Async HTTP client for Discord REST API v10
│   ├── models.py            # DiscordRole, DiscordChannel, PermissionOverwrite
│   ├── permissions.py       # Discord → Stoat permission bit translation
│   └── metadata.py          # DiscordMetadata persistence (discord_metadata.json)
│
├── exporter/
│   ├── __init__.py          # Public API
│   ├── manager.py           # DCE binary download and version management
│   └── runner.py            # Async subprocess execution and progress parsing
│
└── templates/
    ├── gaming.json          # Preset: gaming server template
    ├── community.json       # Preset: community server template
    └── education.json       # Preset: education server template

Module Dependency Graph

gui.py ──────┐
             ├──→ core/engine.py ──→ core/events.py
cli.py ──────┘         │
                       ├──→ config.py, state.py, errors.py
                       │
                       ├──→ exporter/ (DCE binary + subprocess)
                       ├──→ parser/ (JSON parsing + transforms)
                       ├──→ discord/ (Discord API + permission translation)
                       ├──→ review.py (pre-creation summary)
                       │
                       ├──→ migrator/connect.py
                       ├──→ migrator/structure.py ──→ migrator/api.py
                       ├──→ migrator/emoji.py ─────→ migrator/api.py, uploader/
                       ├──→ migrator/avatars.py ───→ uploader/
                       ├──→ migrator/messages.py ──→ migrator/api.py, uploader/, parser/transforms
                       ├──→ migrator/reactions.py ─→ migrator/api.py
                       ├──→ migrator/pins.py ──────→ migrator/api.py
                       │
                       └──→ reporter.py (post-migration report)

Key constraint: engine.py never imports from gui.py or cli.py. The GUI and CLI only import from core/, config.py, and state.py.

Data Models

FerryConfig (`config.py`)

All runtime configuration in a single dataclass. Created by the GUI or CLI, passed to the engine. Never persisted to disk.

@dataclass
class FerryConfig:
    # Required credentials
    export_dir: Path  # DCE JSON exports directory
    stoat_url: str  # Stoat API base URL
    token: str  # Stoat user token (repr=False, hidden in logs)

    # Optional input
    server_id: str | None  # Attach to existing Stoat server (else create new)
    server_name: str | None  # Override server display name
    dry_run: bool  # Preview without API calls
    skip_messages: bool  # Structure-only migration
    skip_emoji: bool  # Skip emoji upload
    skip_reactions: bool  # Skip reaction application
    skip_threads: bool  # Skip thread/forum flattening
    message_rate_limit: float  # Seconds per message (default 1.0)
    upload_delay: float  # Seconds between Autumn uploads (default 0.5)
    output_dir: Path  # Where state.json and report go
    resume: bool  # Load existing state and continue
    verbose: bool  # Extra logging for CLI
    max_channels: int  # Stoat channel limit (default 200)
    max_emoji: int  # Stoat emoji limit (default 100)
    checkpoint_interval: (
        int  # State save frequency during messages (default 50, min 1, throttled max 1/5s)
    )
    skip_avatars: bool  # Skip the avatar pre-flight phase (default False)
    reaction_mode: str  # "text" (default), "native" (Phase 9 API), or "skip"
    min_thread_messages: int  # Filter threads below this count (0 = include all)
    validate_after: bool  # Run post-migration validation phase (default False)
    max_concurrent_requests: int  # asyncio.Semaphore bound for API requests (default 5, min 1)
    max_concurrent_channels: int  # Parallel channel workers in MESSAGES (default 3)
    thread_strategy: str  # "flatten" (default), "merge", or "archive"
    incremental: bool  # Delta migration (default False; mutually exclusive with resume)
    force: bool  # Override DCE freshness warning (>30 days old) and soft errors
    create_invite: bool  # Create an invite to the migrated server on completion (default True)
    invite_channel_id: (
        str | None
    )  # Discord channel ID to base the invite on (default: first eligible text channel)
    verify_uploads: bool  # Post-upload file size verification for Autumn uploads (default False)
    cleanup_orphans: bool  # Report unreferenced Autumn uploads after migration (report-only)
    force_unlock: bool  # Override a stale [FERRY_LOCK:...] marker on the target server
    skip_dce_verify: bool  # Skip SHA-256 verification of DCE binary downloads
    token_store: (
        SecureTokenStore | None
    )  # Injected credential store for masking at output boundaries

    # Discord API (orchestrated mode only)
    discord_token: str | None  # Discord user token (repr=False)
    discord_server_id: str | None  # Source Discord guild ID

    # Runtime control (set by engine, not by user)
    skip_export: bool  # Don't run DCE subprocess
    pause_event: asyncio.Event | None  # GUI: block until user resumes
    cancel_event: asyncio.Event | None  # User stopped migration
    session: aiohttp.ClientSession | None  # Shared HTTP session (set by engine)

MigrationState (`state.py`)

Tracks everything the engine produces across phases. Serialised to state.json after each phase for crash recovery and resume.

@dataclass
class MigrationState:
    # ID mappings (Discord ID → Stoat ID)
    role_map: dict[str, str]
    channel_map: dict[str, str]
    category_map: dict[str, str]
    message_map: dict[str, str]  # For reply reference resolution, saved to message_map.json
    emoji_map: dict[str, str]

    # Upload caches (avoid re-uploading identical files)
    avatar_cache: dict[str, str]  # User ID → Autumn avatar file ID
    upload_cache: dict[str, str]  # "<tag>:<local file path>" → Autumn file ID

    # Author context (for mention remapping in transforms)
    author_names: dict[str, str]  # Discord user ID → display name

    # Name mirrors (used by ferry check to detect renames on the server)
    category_names: dict[str, str]  # Discord category ID → name Ferry created it under
    created_channel_names: dict[str, str]  # Stoat channel ID → name Ferry created it under
    created_role_names: dict[str, str]  # Stoat role ID → name Ferry created it under
    channel_categories: dict[str, str]  # Stoat channel ID → Stoat category ID it belongs to

    # Deferred operations (collected during MESSAGES, applied in later phases)
    pending_pins: list[tuple[str, str]]  # (stoat_channel_id, stoat_message_id)
    pending_reactions: list[
        dict[str, object]
    ]  # {channel_id, message_id, emoji, and optional count}

    # Dead-letter queue (messages that failed after retries)
    failed_messages: list[
        FailedMessage
    ]  # discord_msg_id, stoat_channel_id, error, retry_count, content_preview

    # Autumn upload tracking (for orphan detection and resume)
    autumn_uploads: dict[str, str]  # Autumn file_id → source_id
    referenced_autumn_ids: set[
        str
    ]  # Autumn IDs confirmed used in sent messages (serialised as list)

    # Post-migration validation results
    validation_results: dict[
        str, object
    ]  # Channel/role count comparisons from VALIDATE MIGRATION phase

    # Logs (structured dicts with phase, type, and message)
    errors: list[dict[str, str]]
    warnings: list[dict[str, str]]

    # Server discovery
    stoat_server_id: str
    autumn_url: str  # Discovered during CONNECT phase
    invite_code: str  # Server invite code created after migration (empty when disabled)
    invite_url: str  # Full https://rvlt.gg/... URL matching invite_code

    # Forum/thread tracking (keyed by forum-category key, not by forum channel ID)
    forum_index_message_ids: dict[str, str]  # forum_cat_key → index message Stoat ID
    forum_channel_members: dict[
        str, list[str]
    ]  # forum_cat_key → Discord channel IDs of the forum's members
    forum_category_names: dict[str, str]  # forum_cat_key → category name on Stoat

    # Resume tracking
    current_phase: str  # Phase name for resume skip logic
    completed_channel_ids: set[str]  # Discord channel IDs fully processed in MESSAGES phase
    channel_message_offsets: dict[
        str, str
    ]  # Discord channel ID → last checkpointed Discord message ID
    channel_high_water: dict[
        str, str
    ]  # Discord channel ID → highest Discord message ID successfully sent (durable for merge/incremental)
    thread_strategy: (
        str  # The strategy the migration ran under (recorded from 2.17.0 for ferry check)
    )

    # Counters
    attachments_uploaded: int
    attachments_skipped: int
    reactions_applied: int
    reactions_capped: int  # Reactions dropped because Stoat's 20-per-message cap was hit
    reactions_dropped: (
        int  # Reactions dropped for any other reason (invalid emoji, unreachable message)
    )
    reaction_message_counts: dict[
        str, int
    ]  # Stoat channel ID → messages that carried a text-mode reaction summary
    pins_applied: int
    channel_message_counts: dict[str, int]  # Discord channel ID → messages sent count
    source_messages_total: (
        int  # Total messages seen in the source export (denominator for fidelity)
    )
    prior_messages_total: int  # Total messages in prior (resumed) state
    embeds_total: int
    embeds_dropped: int  # Embeds dropped when validate_sum would exceed the 2000-char body ceiling
    native_fidelity_counts: dict[
        str, int
    ]  # Per-strategy native reaction sends, for fidelity scoring
    replies_linked: int
    replies_total: int

    # Roles finalization
    roles_finalized: bool  # ROLES phase completed the attributes/rank pass (guards resume)

    # Timing
    started_at: str  # ISO 8601
    completed_at: str  # ISO 8601

    # Rollback tracking (populated by ferry rollback)
    rollback_progress: dict[str, object]  # rolled_back_ids, failures, per-category counters

    # Flags
    is_dry_run: bool  # Reject resume from dry-run state
    export_completed: bool  # Smart export phase skipping
    migration_lock_marker: (
        str  # The [FERRY_LOCK:ts:host] marker Ferry wrote (for force-unlock diagnostics)
    )

Persistence: save_state() writes atomically (temp file + rename). load_state() reads and validates. Both use _state_to_dict() / _dict_to_state() for JSON round-tripping.

DCE Parser Models (`parser/models.py`)

Twelve dataclasses representing DiscordChatExporter JSON structure:

Dataclass	Key Fields
`DCEGuild`	`id`, `name`, `icon_url`
`DCEChannel`	`id`, `type`, `name`, `category_id`, `category`, `topic`
`DCEAuthor`	`id`, `name`, `discriminator`, `nickname`, `color`, `is_bot`, `avatar_url`, `roles`
`DCERole`	`id`, `name`, `color`, `position`
`DCEAttachment`	`id`, `url`, `file_name`, `file_size_bytes`
`DCEEmoji`	`id`, `name`, `is_animated`, `image_url`
`DCEReaction`	`emoji` (DCEEmoji), `count`
`DCEReference`	`message_id`, `channel_id`, `guild_id`, `type`
`DCEForwardedMessage`	`timestamp`, `timestamp_edited`, `content`, `attachments`, `embeds`, `stickers`
`DCEInteraction`	`id`, `name`, `user`
`DCEMessage`	`id`, `type`, `timestamp`, `timestamp_edited`, `content`, `author`, `is_pinned`, `attachments`, `embeds`, `stickers`, `reactions`, `mentions`, `inline_emojis`, `interaction`, `reference`, `forwarded_message`
`DCEExport`	`guild`, `channel`, `messages`, `message_count`, `exported_at`, `is_thread`, `parent_channel_name`, `json_path`

Discord Metadata Models (`discord/models.py`, `discord/metadata.py`)

PermissionOverwrite(id, type, allow, deny)  # type: 0=role, 1=member
DiscordRole(id, name, permissions, position, color, hoist, managed)
DiscordChannel(id, name, type, nsfw, permission_overwrites)

PermissionPair(allow, deny)  # In Stoat bit space (translated)
RoleOverride(discord_role_id, allow, deny)  # Per-role channel override
ChannelMeta(nsfw, default_override, role_overrides)

DiscordMetadata(  # Persisted to discord_metadata.json
    guild_id,
    fetched_at,
    server_default_permissions,  # @everyone server-wide permissions
    role_permissions,  # {role_id: PermissionPair}
    channel_metadata,  # {channel_id: ChannelMeta}
)

Blueprint Models (`blueprint.py`)

BlueprintRole(name, colour, permissions, rank)
BlueprintChannel(name, type, nsfw, user_limit=0)  # type: "Text" or "Voice"
BlueprintCategory(name, channels)
ServerBlueprint(name, description, roles, categories, uncategorized_channels)

Blueprints use names, not IDs, making them portable across Stoat instances.

Voice channels retain a positive user_limit in blueprint JSON. Missing or zero means unlimited; zero is omitted on export. Invalid caps are rejected before server creation.

Migration Pipeline

Phase Execution

The engine runs 13 phases in strict order (0 through 12, with an additional 7.5 for AVATARS). Each phase depends on ID mappings and state produced by earlier phases.

EXPORT → VALIDATE → CONNECT → SERVER → ROLES → CATEGORIES → CHANNELS
                                                                 ↓
  VALIDATE MIGRATION ← REPORT ← PINS ← REACTIONS ← MESSAGES ← EMOJI → AVATARS

#	Phase	Module	Reads	Writes	API Calls
0	EXPORT	`exporter/`	Discord API	DCE JSON files	Discord API
1	VALIDATE	`parser/`	DCE JSON files	warnings, author_names	None
2	CONNECT	`migrator/connect.py`	config	autumn_url, stoat_server_id	`GET /` + `GET /users/@me`
3	SERVER	`migrator/structure.py`	exports[0].guild	stoat_server_id	`POST /servers`, Autumn upload
4	ROLES	`migrator/structure.py`	exports, discord_metadata	role_map	`POST /servers/:id/roles`, permission PUTs
5	CATEGORIES	`migrator/structure.py`	exports	category_map	`PATCH /servers/:id`
6	CHANNELS	`migrator/structure.py`	exports, discord_metadata	channel_map	`POST /servers/:id/channels`, permission PUTs
7	EMOJI	`migrator/emoji.py`	message content + reactions	emoji_map	Autumn upload + `PUT /servers/:id/emoji/:id`
7.5	AVATARS	`migrator/avatars.py`	all exports (unique authors)	avatar_cache	Autumn upload
8	MESSAGES	`migrator/messages.py`	all messages	message_map, pending_pins/reactions	`POST /channels/:id/messages`, Autumn uploads
9	REACTIONS	`migrator/reactions.py`	pending_reactions	reactions_applied	`PUT /channels/:id/messages/:id/reactions`
10	PINS	`migrator/pins.py`	pending_pins	pins_applied	`PUT /channels/:id/messages/:id/pin`
11	REPORT	`reporter.py`	state	migration_report.json	None
12	VALIDATE MIGRATION	inline in `engine.py`	state, config	validation_results	`GET /servers/:id`

Phase Skip Logic

Phases can be skipped in three ways:

Config flags: skip_messages, skip_emoji, skip_reactions, skip_threads, skip_export, skip_avatars; reaction_mode="skip" for REACTIONS; validate_after=False (default) for VALIDATE MIGRATION
Resume: If state.current_phase index > phase index, the phase was already completed
Mode: EXPORT phase is skipped entirely in offline mode (no Discord token)

Review Gate

Between VALIDATE and SERVER, the engine inserts a review step:

review.py builds a ReviewSummary from parsed exports and Discord metadata
Engine emits a MigrationEvent with status="confirm" and the summary as detail
GUI: Blocks on pause_event until user clicks Proceed or Cancel
CLI: Prints a Rich table and continues (non-interactive)

Per-Phase Detail

EXPORT (Phase 0): Runs inline in run_migration(). Downloads the DCE binary if not cached, validates the Discord token, then runs DCE as a subprocess. Progress is parsed from DCE's stdout and emitted as events. Produces DCE JSON files in config.export_dir.

VALIDATE (Phase 1): Runs inline. Calls parse_export_directory(metadata_only=True) which reads channel metadata but skips loading messages into memory. Then calls validate_export() which streams messages to check for issues (rendered markdown, missing media, duplicate IDs). Collects author_names in a single pass to avoid re-scanning later.

CONNECT (Phase 2): GET / on the Stoat API to discover the Autumn upload URL from the features.autumn.url field. GET /users/@me to validate the token. If config.server_id is set, verifies the server is accessible (best-effort, non-fatal on failure).

SERVER (Phase 3): Creates a new server via POST /servers (or verifies the existing one). Uploads the guild icon to Autumn and applies it. Versions before 2.41.6 also set the server default permissions to FERRY_MIN_PERMISSIONS (1,022,361,624); that write was removed because Stoat seeds every member's effective permissions from the server default (#957). The Ferry account's bits come from owner grants or ownership instead, and ferry repair on 2.41.7 or later converges servers migrated by older versions.

ROLES (Phase 4): Iterates all exports to collect unique role IDs (skipping @everyone where role_id == guild_id). The create call carries name only. A second attributes pass applies colour, hoist, and icon, keyed by the created role's ID. A resume with role_map populated retries those fields without recreating the role. state.roles_finalized protects the attributes pass when its last write failed. Permission bits are translated and written via api_set_role_permissions().

Hierarchy is applied last, once for the whole server, by reading the server back and sending the complete ordered role list to PATCH /servers/:id/roles/ranks. Index 0 of that list is the top of the hierarchy, so Ferry sorts by Discord position descending. Only roles Ferry created are moved: every other role keeps its position, so migrating into an existing server with --server-id leaves that server's own hierarchy alone. The step is skipped when it cannot change anything, and a failure degrades to a role_ordering_failed warning, or role_ordering_not_permitted when Ferry lacks the rank or permission to reorder that server. The per-role PATCH does not carry a rank: the Stoat backend accepts that field and discards it.

CATEGORIES (Phase 5): Collects unique category names from exports. Creates each category via the two-step process: create a channel-like object, then PATCH the server's categories array. Populates state.category_map.

CHANNELS (Phase 6): Per export, creates a TextChannel or VoiceChannel. Sets NSFW flag from Discord metadata. Assigns to category via the two-step PATCH. Flattens threads into standalone text channels. Groups forum/media threads into dedicated categories named after the parent forum. Applies channel-level permission overrides (@everyone as default, per-role as overrides). Populates state.channel_map.

EMOJI (Phase 7): Scans all messages (via streaming if metadata_only) for custom emoji in content (<:name:ID>) and reactions. Deduplicates by ID. Downloads each emoji image, uploads to Autumn with tag emojis, creates on server. 2.0s delay between creates (shares the 5/10s /servers rate bucket). Populates state.emoji_map.

AVATARS (Phase 7.5): Pre-flight avatar download and upload. run_avatars() in migrator/avatars.py scans all exports for unique authors and downloads each author's avatar. It uploads the local file or remote URL to Autumn with the avatars tag and populates state.avatar_cache (user ID → Autumn file ID). This front-loads all avatar uploads before the MESSAGES phase begins, so message sends never block on avatar I/O. Skippable via skip_avatars config flag. Added in v1.5.0.

MESSAGES (Phase 8): The largest phase. In v2.0.0 this phase uses a parallel channel architecture:

Channels are processed concurrently via asyncio.gather bounded by asyncio.Semaphore(config.max_concurrent_channels) (default 3)
Each channel worker accumulates results in a ChannelResult dataclass containing: warnings, errors, failed_messages, message_map_updates, pending_pins, pending_reactions, and attachment counters
Results from each worker are merged into the main MigrationState after the channel completes (serialized under a save_lock: asyncio.Lock)
Thread merge mode (thread_strategy="merge") processes thread channels sequentially after all parent channels finish, so parent message IDs are available for reply linking

Per channel, messages are streamed oldest-first. Per message (9-step pipeline):

Check skip types → skip system messages (join, boost, etc.)
If ChannelPinnedMessage → extract reference, add to pending_pins
Add interaction context and transform content (spoilers → underline → mentions → emoji → timestamps)
Upload attachments and local sticker images to Autumn (max 5 per message)
Flatten embeds (extract media, convert to Stoat format)
Preserve DCE fallback content and embeds for system notifications such as PollResult
Append the reaction summary when text reaction mode is selected
Build masquerade (original author name + avatar + colour)
Send via api_send_message with Idempotency-Key header ferry-{discord_msg_id}

Collects pending_reactions for Phase 9. Saves state every checkpoint_interval messages (default 50, time-throttled to at most once per 5 seconds via save_lock). Completed channel IDs are recorded in state.completed_channel_ids.

REACTIONS (Phase 9): The behaviour depends on config.reaction_mode.

native (Phase 9 as historically described): iterates state.pending_reactions and calls api_add_reaction for each. Enforces the 20-reactions-per-message Stoat limit. Fire-and-forget error handling (failures logged as warnings, do not stop migration).
text (default from v2.7.0): no API calls in Phase 9. Instead, the MESSAGES phase calls _build_reaction_text() per message to append a reaction summary to the message body, e.g. Reactions: :thumbsup: 3, :heart: 1. Custom emoji migrated in Phase 7 render as <:stoat_id:> (the Stoat inline-emoji syntax); custom emoji that were not migrated fall back to [:name:]; unicode emoji pass through unchanged. Native-mode also appends an [Original counts: ...] annotation to preserve fidelity when Stoat's 20-cap truncates.
skip: no reaction handling of any kind.

PINS (Phase 10): Iterates state.pending_pins tuples. Calls api_pin_message for each. Fire-and-forget error handling.

REPORT (Phase 11): Runs inline. Calls generate_report() which writes migration_report.json containing summary counts, ID maps, timing, warnings, errors, and a dynamic post-migration checklist.

VALIDATE MIGRATION (Phase 12, optional): Runs inline in engine.py only when validate_after=True. Calls run_check() from migrator/verify.py, which is the same engine ferry check uses on its own, every recorded channel, role, category and emoji is fetched by ID, and each channel's most recent messages are compared against the tail Ferry recorded. The result is a CheckReport with per-entity ok / warn / fail / unverifiable verdicts, serialised into state.validation_results and echoed to the run log. Does not modify anything on the server. Added in v1.6.0; switched from a bare count comparison to the shared run_check() engine in v2.16 (the standalone ferry check release).

Event System

MigrationEvent

@dataclass
class MigrationEvent:
    phase: str  # "export", "validate", "messages", etc.
    status: str  # See table below
    message: str  # Human-readable status text
    current: int = 0  # Progress: items processed so far
    total: int = 0  # Progress: total items in this phase
    channel_name: str = ""  # Currently active channel (if applicable)
    detail: dict | None  # Extra context (review summary, error info)

Status Values

Status	Meaning	Emitted by
`started`	Phase has begun	Engine, at phase entry
`progress`	Work in progress	Phase implementations, repeatedly
`completed`	Phase finished successfully	Engine, at phase exit
`error`	Fatal error occurred	Phase implementations
`warning`	Non-fatal issue	Phase implementations
`notice`	Configuration notice the user must see before the run starts. Printed unconditionally, not counted as a warning.	Preflight checks
`skipped`	Phase was skipped (config flag or resume)	Engine
`confirm`	Awaiting user confirmation (review gate)	Engine, once

# Engine signature
async def run_migration(
    config: FerryConfig,
    on_event: EventCallback,          # GUI or CLI provides this
    phase_overrides: dict | None,     # For testing: inject mock phases
) -> MigrationState

GUI passes a callback that updates NiceGUI UI elements (progress bars, status chips, log entries). Uses ui.timer to drive async updates in NiceGUI's event loop.

CLI passes a _ProgressTracker instance that renders Rich progress bars, status tables, and prints warnings/errors to the console via Rich.Live.

Neither shell knows about the other. Adding a new UI (REST API, TUI, etc.) only requires writing a new callback, no engine changes needed.

Stoat API Layer (`migrator/api.py`)

Function Inventory

Function	HTTP	Purpose
`api_create_server`	`POST /servers/create`	Create new Stoat server, returns its id
`api_fetch_server`	`GET /servers/:id`	Verify server exists
`api_edit_server`	`PATCH /servers/:id`	Update server settings
`api_create_role`	`POST /servers/:id/roles`	Create role
`api_edit_role`	`PATCH /servers/:id/roles/:id`	Update role (colour, hoist, icon)
`api_edit_role_ranks`	`PATCH /servers/:id/roles/ranks`	Set the whole role hierarchy, index 0 highest
`api_set_role_permissions`	`PUT /servers/:id/permissions/:id`	Set role permission bits
`api_set_server_default_permissions`	`PUT /servers/:id/permissions/default`	Set @everyone server permissions
`api_create_channel`	`POST /servers/:id/channels`	Create channel
`api_upsert_categories`	`PATCH /servers/:id`	Set full categories array on server
`api_set_channel_default_permissions`	`PUT /channels/:id/permissions/default`	Set @everyone channel override
`api_set_channel_role_permissions`	`PUT /channels/:id/permissions/:role_id`	Set per-role channel override
`api_send_message`	`POST /channels/:id/messages`	Send message with masquerade
`api_edit_message`	`PATCH /channels/:id/messages/:id`	Edit a sent message (content, embeds)
`api_add_reaction`	`PUT /channels/:id/messages/:id/reactions/:emoji`	Add reaction
`api_create_emoji`	`PUT /custom/emoji/:id`	Register emoji on server
`api_pin_message`	`PUT /channels/:id/messages/:id/pin`	Pin a message

String Sanitization (`migrator/sanitize.py`)

The Stoat API enforces a 32-character maximum on all name fields. Ferry sanitizes at call sites (not inside API wrappers) using two helpers:

Helper	Applied to	Rules
`truncate_name(name, max_length=32)`	Channel names, role names, category titles, masquerade display names	Truncate to 32 chars
`sanitize_emoji_name(name)`	Custom emoji names	Lowercase, replace non-`[a-z0-9_]` with `_`, strip edges, truncate to 32, fallback to `"emoji"` if empty

Channel name collisions after truncation are handled by make_unique_channel_name() which appends -1, -2 suffixes (eating into the 32-char budget as needed).

Rate Limit Buckets

Stoat uses fixed 10-second windows (not sliding).

Bucket	Limit	Shared across
`/servers`	5 per 10s	Server create, channel create, role create, emoji create, category edit
Messages	10 per 10s	`POST /channels/:id/messages` only
Catch-all	20 per 10s	Everything else, including Autumn uploads

The /servers bucket is shared

Creating a channel, a role, and an emoji in quick succession all draw from the same 5-per-10-second budget. Ferry paces structure creation phases to stay within this limit.

Response headers: Stoat sets X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset-After and X-RateLimit-Bucket on every response, and exposes them through CORS. X-RateLimit-Reset-After is in milliseconds, unlike Discord's identically named header. See Stoat API Notes.

429 response body: { "retry_after": 4200 }, Ferry sleeps for this duration (milliseconds) and retries. In addition, Ferry's adaptive 429-frequency optimizer tracks 429 frequency in a rolling 60-second window and auto-adjusts a delay multiplier (1.5× increase on burst, 0.75× decay when clear).

Retry Logic

_api_request() retries up to 3 times on:

429 Too Many Requests: Sleep for retry_after milliseconds from response body
502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout: Exponential backoff

All other HTTP errors raise MigrationError immediately.

British Spelling

The Stoat API uses British English. Fields with American spelling can be accepted without taking effect.

Always use	Never use
`colour`	`color`
`ManageCustomisation`	`ManageCustomization`

This applies to masquerade payloads, embed objects, role objects, and permission names.

Permission Bits

Stoat has no single all-access permission named ADMINISTRATOR. Every capability must be granted individually. Stoat defines 34 permission bits; the complete list lives in one place, in Stoat API Notes → Permission Bits.

This page does not copy that table. An older copy held 13 bits after Stoat's enum had grown to 34. That gap allowed Ferry to ship voice channels without voice permissions. One table, one owner.

Ferry minimum permissions (bits 3, 4, 20–23, 26–29):

FERRY_MIN_PERMISSIONS = (
    8
    | 16
    | 1_048_576
    | 2_097_152
    | 4_194_304
    | 8_388_608
    | 67_108_864
    | 134_217_728
    | 268_435_456
    | 536_870_912
)  # == 1_022_361_624

Category Management Pattern

Categories in Stoat live on the Server object, not on channels. There is no category_id parameter on channel creation. The process is:

Create all channels via POST /servers/:id/channels
Build the full categories array locally (each category has a client-generated ID, title, and channel list)
PATCH the server with {"categories": [...]} in a single call via api_upsert_categories

channel = await api_create_channel(session, stoat_url, token, server_id, name="general")
categories = [
    {"id": uuid4().hex[:26], "title": "Text Channels", "channels": [channel["_id"]]},
]
await api_upsert_categories(session, stoat_url, token, server_id, categories)

Category IDs are generated client-side (uuid4().hex[:26]). The PATCH replaces the entire categories array on the server.

Parser and Transforms

DCE JSON Parsing (`parser/dce_parser.py`)

Full parse (parse_export_directory(export_dir)): Reads all *.json files in the directory, parses guild, channel, and message data into typed dataclasses.

Metadata-only parse (parse_export_directory(export_dir, metadata_only=True)): Streams through the file and retains direct scalar fields from guild and channel, plus messageCount and exportedAt. Message values are read as parser events and discarded instead of being collected into an array. Each DCEExport therefore has an empty messages list, with json_path and message_count set so messages can be streamed later. This is the default mode, and its memory use does not grow with the message count.

Streaming (stream_messages(json_path)): Uses ijson.items() to iterate messages one at a time with O(1) memory per message. Used by VALIDATE (for scanning), EMOJI (for extraction), and MESSAGES (for import) when running in metadata-only mode.

Directory parse (parse_export_directory(export_dir)): Refuses duplicate channel IDs before the validation scan. Its error names every file carrying each repeated ID so the operator can remove the older export.

Validation (validate_export(exports, export_dir, author_names)): Single-pass scan that checks for rendered markdown detection (missing --markdown false), missing media files, and limit violations. Populates author_names dict to avoid a second scan.

Thread Inference

DCE 2.48 writes a thread's parent channel ID to channel.categoryId and its parent name to channel.category. Ferry uses the parsed channel.type as the authority on whether an export is a thread. A known non-thread type remains a regular channel even when its guild or channel name contains -.

For a thread with a parent ID, Ferry takes the parent name from channel.category. It falls back to the parent segment in {Guild} - {Parent} - {Thread} [{id}].json only when the JSON has no parent ID. The DCEExport dataclass stores the resulting is_thread and parent_channel_name values.

Content Transforms (`parser/transforms.py`)

All transforms are code-block-aware: they split content on ```...``` and `...` blocks, apply the transformation only to non-code regions, then reassemble. This prevents mangling code snippets in messages.

Transform pipeline (applied in order during MESSAGES phase):

Transform	Input	Output	Notes
`convert_spoilers`	`\\|\\|text\\|\\|`	`!!text!!`	Discord → Stoat spoiler syntax
`strip_underline`	`__text__`	`text`	Stoat does not support underline
`remap_mentions`	`<@ID>`, `<#ID>`, `<@&ID>`	Display name or Stoat ID	Uses channel_map, role_map, author_names
`remap_emoji`	`<:name:ID>`	`:stoat_id:` or `[:name:]`	Uses emoji_map, fallback for unmapped
`format_original_timestamp`	ISO 8601	`[2024-01-15 14:30 UTC]`	Prepended to message body
`flatten_embed`	Discord embed dict	Stoat SendableEmbed + media path	Author, fields, footer → description
`handle_stickers`	Sticker list	Text reference or upload path	Image if local, `[Sticker: name]` fallback

DCE 2.48 writes inlineEmojis metadata on messages and embeds. The parser adds those image paths to the emoji asset candidates without changing message or reaction counts. An interaction adds the command name and invoking user's display name to message content. Its ID and other user profile fields are parsed but have no matching Stoat object to send.

DCE 2.48 does not write active poll objects. A PollResult notification carries ordinary fallback content and embeds, so it follows the standard content and embed paths. Ferry does not call a native Stoat poll endpoint.

Embed Flattening

Discord embeds have a richer structure than Stoat embeds. flatten_embed() converts:

embed.author.name → first line of description
embed.fields → appended to description as **Name**: Value
embed.footer.text → last line of description
embed.thumbnail.url or embed.image.url → returns local media path for Autumn upload

The result is a Stoat-compatible embed dict plus an optional local file path for the media.

Description budget (from v2.19.5, PR #349). Stoat rejects a message whose combined content + embed body exceeds a 2000-character validate_sum ceiling. The messages pipeline enforces that ceiling before sending: at step 7b it walks the embed list in order, tracks the running sum, and drops any embed whose description would push the total past 2000. Each dropped embed appends a compact note to the message content ([Embed dropped: <title or 'untitled'>]) and is counted in state.embeds_dropped, which the fidelity score reads. Trailing embeds are the ones lost, earlier embeds always fit, so an embed-heavy message that arrives with the first two present and the tail replaced by drop notes is expected, not a bug.

File Upload, Autumn (`uploader/autumn.py`)

Autumn is Stoat's media storage service. It accepts file uploads via multipart form POST and returns a file ID that can be referenced in messages, avatars, and emoji.

Critical constraint: Autumn cannot fetch URLs. You must download the file locally first, then upload it as multipart form data.

Upload Flow

Local file → validate size → POST multipart to {autumn_url}/{tag} → file ID

Size Limits by Tag

Tag	Max Size	Used for
`attachments`	20 MB	Message attachments
`avatars`	4 MB	User/masquerade avatars
`icons`	2.5 MB	Server icon, role icon
`banners`	6 MB	Server banner
`emojis`	500 KB	Custom emoji

Upload Cache

upload_with_cache() maintains an in-memory cache keyed on local file path. If the same file appears multiple times (e.g. an author who appears in thousands of messages), the avatar is uploaded once and the Autumn file ID is reused for all subsequent messages.

A conservative 0.5s sleep is inserted between uploads to avoid bursting the catch-all rate limit bucket.

Discord Integration (`discord/`)

The discord/ package is only active when both discord_token and discord_server_id are provided (1-Click mode). In offline mode, it is skipped entirely and permissions are not migrated.

What It Fetches

Guild roles (GET /guilds/{id}/roles): All roles with permission bitfields, position, colour, managed flag
Guild channels (GET /guilds/{id}/channels): All channels with NSFW flag and per-channel permission overwrites

Permissions come as strings

Discord's REST API returns permission bitfields as strings (e.g. "2048"), not integers. The client calls int() when parsing.

Permission Translation (`discord/permissions.py`)

translate_permissions(discord_bits: int) -> int converts a Discord permission bitfield to the equivalent Stoat bitfield.

Discord Permission	Discord Bit	Stoat Permission	Stoat Bit
CREATE_INSTANT_INVITE	0	InviteOthers	25
KICK_MEMBERS	1	KickMembers	6
BAN_MEMBERS	2	BanMembers	7
MANAGE_CHANNELS	4	ManageChannel	0
MANAGE_GUILD	5	ManageServer	1
ADD_REACTIONS	6	React	29
VIEW_AUDIT_LOG	7	ViewAuditLogs	40
STREAM	9	Video	32
VIEW_CHANNEL	10	ViewChannel	20
SEND_MESSAGES	11	SendMessage	22
MANAGE_MESSAGES	13	ManageMessages	23
EMBED_LINKS	14	SendEmbeds	26
ATTACH_FILES	15	UploadFiles	27
READ_MESSAGE_HISTORY	16	ReadMessageHistory	21
MENTION_EVERYONE	17	MentionEveryone + MentionRoles	37, 38
CONNECT	20	Connect + Listen	30, 36
SPEAK	21	Speak	31
MUTE_MEMBERS	22	MuteMembers	33
DEAFEN_MEMBERS	23	DeafenMembers	34
MOVE_MEMBERS	24	MoveMembers	35
CHANGE_NICKNAME	26	ChangeNickname	10
MANAGE_NICKNAMES	27	ManageNicknames	11
MANAGE_ROLES	28	ManagePermissions + ManageRole + AssignRoles	2, 3, 9
MANAGE_WEBHOOKS	29	ManageWebhooks	24
MANAGE_EMOJIS	30	ManageCustomisation	4
MODERATE_MEMBERS	40	TimeoutMembers	8
BYPASS_SLOWMODE	52	BypassSlowmode	39

Special cases:

ADMINISTRATOR (Discord bit 3): On a server role or the @everyone server default, expands to ALL_STOAT_PERMISSIONS, derived from STOAT_PERMISSION_BITS, every one of the 34 bits Stoat defines, and nothing from its reserved 41–52 free area. Inside a channel overwrite it grants nothing, because Discord checks it only on server-level role permissions. Ferry drops it there and translates the overwrite's other bits
CONNECT maps to two bits: Stoat gates joining a voice channel on Connect but gates hearing anyone on Listen, and Discord's single permission covers both
PIN_MESSAGES (51) is deliberately unmapped, Stoat has no pin-only bit, and its nearest equivalent (ManageMessages) also permits deletion
Unmapped Discord bits: Dropped and recorded as having no Stoat equivalent
Managed/bot roles: Permissions skipped (role still created for mention remapping)

@everyone Handling

Discord's @everyone role has id == guild_id. In channel permission overwrites, this must be extracted as the channel's default_override and applied via Stoat's separate PUT /channels/:id/permissions/default endpoint. If it goes into role_overrides, it is dropped because guild_id is never in role_map.

Metadata Persistence

DiscordMetadata is saved to discord_metadata.json alongside state.json. This ensures permission data survives resume, the Discord API does not need to be re-queried.

Message Pipeline (`migrator/messages.py`)

The MESSAGES phase is the most complex. It processes messages oldest-first, per channel, with full content transformation and author attribution.

Per-Message Processing (9 Steps)

1. Type check         → skip system messages (join, boost, pin notification, etc.)
2. Pin detection      → if ChannelPinnedMessage, extract reference for pending_pins
3. Content transforms → interaction context, spoilers, underline, mentions, emoji, timestamps
4. Attachment upload   → upload attachments and local sticker images (max 5 per message)
5. Embed flattening    → convert Discord embed to Stoat format, upload media
6. System fallback     → preserve DCE content and embeds, including `PollResult` notifications
7. Reaction summary    → append text when text reaction mode is selected
8. Masquerade build    → original author name + avatar (lazy upload) + colour
9. Send               → api_send_message with Idempotency-Key for deduplication

Author Attribution (Masquerade)

Every message is sent with a masquerade payload that makes it appear to come from the original Discord author:

masquerade = {
    "name": "Alice",  # Original display name
    "avatar": "autumn-file-id-for-avatar",  # Uploaded once, cached
    "colour": "#5865F2",  # British spelling required
}

Masquerade permission (bit 28) is required for name and avatar
ManageRole permission (bit 3) is additionally required for colour

Avatar upload is lazy: the first message from an author triggers an Autumn upload; all subsequent messages reuse the cached file ID from state.avatar_cache.

Idempotency-Key Deduplication

Every message send includes an Idempotency-Key HTTP header set to ferry-{discord_msg_id}.

It does not make the MESSAGES phase safe to re-run, and an earlier revision of this page said it did. Stoat's store is a 1000-entry, in-memory, process-local LRU with no TTL that is emptied on restart, and a repeated key returns HTTP 409, not the original message. Treat it as a guard against an immediate double-send inside one run and nothing more, see Stoat API Notes → Message Deduplication for the source-verified contract and for which client-side marker actually covers each strategy.

Reply Reference Resolution

When type == "Reply", the reference.messageId is looked up in state.message_map to get the corresponding Stoat message ID. If the referenced message predates the export or was otherwise not migrated, the reply is sent as a regular message. Ferry logs a warning.

Message Type Handling

Type	Action
`"Default"`	Import normally
`"Reply"`	Import with reply reference
`"ChannelPinnedMessage"`	Import, schedule re-pinning
`"ThreadStarterMessage"`	Import as first message in flattened thread
`"RecipientAdd"`, `"RecipientRemove"`	Skip
`"ChannelNameChange"`, `"ChannelIconChange"`	Skip
`"GuildMemberJoin"`	Skip (system noise)
`"UserPremiumGuildSubscription"`	Skip (boost notification)
`"ThreadCreated"`	Skip (Ferry injects its own thread header)
`"Call"`	Skip

Unknown types are logged as warnings and skipped.

Thread Headers

When processing a flattened thread channel, Ferry injects a system message at the start: [Thread migrated from #parent-channel] (or [Forum post migrated from #parent-forum] for forum/media threads).

State and Resume

state.json

Written to config.output_dir/state.json after each phase completes. Contains the full MigrationState as JSON.

Phase-Level Resume

On --resume, the engine loads state.json and compares state.current_phase against the phase order list. Any phase with an index less than the saved phase is skipped (already completed).

Message-Level Resume

The MESSAGES phase has finer granularity (v2.0.0+):

state.completed_channel_ids: set of Discord channel IDs whose messages were fully sent
state.channel_message_offsets: maps a partially-processed channel ID to the last Discord message ID checkpointed, the phase resumes within that channel from that offset. Note "checkpointed", not "successfully sent": the counter advances on a failed send too, so a failure can sit below the offset. Such a message stays in state.failed_messages and is re-attempted by --incremental (not by --resume, which is a pure continuation)
Under --thread-strategy merge, the same pair of markers is kept per thread, keyed by the thread's own channel ID (batch 6, #107). That path writes no message_map. The markers prevent a re-run from duplicating the thread in its parent.

v1→v2 automatic migration: On --resume with a v1 state file, Ferry detects the old last_completed_channel / last_completed_message fields, converts them to the new set/dict format, and saves a backup of the original state file before proceeding.

Dry-Run Rejection

state.is_dry_run is set to True during dry-run mode. If a user attempts --resume on a dry-run state file, the engine raises StateError, you cannot resume a dry run into a real migration.

Atomic Saves

save_state() writes to a temporary file first, then renames. This prevents corrupt state files if the process is killed mid-write.

Async Patterns

Single Shared Session

The engine creates one aiohttp.ClientSession before any phases run and stores it in config.session. All phases reuse this session for connection pooling. The session is closed in a finally block after all phases complete.

Pause and Cancel

Two asyncio.Event objects on FerryConfig control flow:

pause_event: Created by the GUI and starts unset. The engine waits on it at the review gate. The GUI sets it when the user clicks Proceed. In CLI mode, this is None (no blocking).
cancel_event: Created by both GUI and CLI. Starts unset. The engine checks cancel_event.is_set() between phases. If set, it saves state and raises MigrationError. The message rate limiter also checks cancel during its sleep.

Rate Limiting Strategy

Three layers:

HTTP-level retry: _api_request() handles 429 responses by sleeping for the server-specified retry_after duration, then retrying (up to 3 times)
Adaptive 429-frequency optimizer: A rolling 60-second window tracks 429 frequency. On a 429 burst the delay multiplier increases by 1.5×; when the window is clear it decays by 0.75×. This compensates for the absence of X-RateLimit-* response headers.
Phase-level pacing: Structure creation phases (ROLES, CATEGORIES, CHANNELS, EMOJI) add explicit asyncio.sleep() between API calls to stay within the 5/10s /servers bucket
User-configurable delay: config.message_rate_limit (default 1.0s) adds a sleep between each message send as a safety margin above the 10/10s message bucket

Error Handling

Exception Hierarchy

FerryError (base)
├── ValidationError            # Export validation failed (red status)
├── StoatConnectionError       # API unreachable or token invalid
├── AutumnUploadError          # File upload to Autumn failed
├── MigrationError             # Generic phase failure
├── StateError                 # state.json read/write problem
└── ExportError (→ MigrationError)
    ├── DCENotFoundError       # DCE binary not available
    ├── DotNetMissingError     # Legacy import-compatible type; current export path does not raise it
    └── DiscordAuthError       # Discord token invalid or expired

Phase-Level Error Handling

Each phase catches exceptions internally and:

Appends a structured dict to state.errors: {"phase": "...", "type": "...", "error": "..."}
Saves state to disk (crash recovery)
Re-raises the exception to the engine

The engine catches FerryError at the top level, saves final state, and returns the MigrationState to the shell for display.

Warnings vs Errors

Warnings (state.warnings): Non-fatal. Migration continues. Examples: failed attachment upload, animated emoji (animation lost), missing sticker image.
Errors (state.errors): Fatal. Migration stops. Examples: API unreachable, token expired, state file corrupt.

Both use structured dicts with phase, type, and message fields for filtering and reporting.

Presentation Layers

GUI (`gui.py`), NiceGUI

Architecture: NiceGUI runs an embedded FastAPI server with Vue.js frontend. Ferry launches it in native mode (pywebview window) if available, otherwise opens the default browser to http://localhost:8765.

4-Screen Workflow:

Setup: Credential inputs, mode toggle (1-Click vs Offline), advanced options (rate limit, skip flags, dry run, server ID). All fields persist to app.storage.user.
Export (1-Click only): DCE download progress, per-channel export progress, cached export detection with Use Cached / Re-export choice.
Validate: Export summary table, warnings list, ETA estimate, green/amber/red status.
Migrate: Phase indicator chips, per-channel progress bar, running totals (messages, attachments, errors), scrolling log, pause/resume button, cancel button with confirmation.

Event subscription: Passes a callback to run_migration() that updates NiceGUI elements. Uses ui.timer for async event loop integration.

Storage: app.storage.user writes to .nicegui/storage-user.json. Sensitive data (Discord tokens) is cleared in a finally block.

CLI (`cli.py`), Click + Rich

Commands:

Command	Purpose
`ferry migrate`	Run full migration (orchestrated or offline)
`ferry validate`	Pre-check exports without migrating
`ferry build`	Create server from template or blueprint
`ferry export-blueprint`	Convert DCE export to reusable blueprint JSON
`ferry rollback`	Reverse a recorded migration, preserving the audit trail in state.json
`ferry stats`	Print a Rich-table summary of a completed run (read-only, zero network calls)
`ferry check`	Ask the live Stoat server whether every entity Ferry recorded is still there
`ferry repair`	Re-send messages and re-create structural entities that `ferry check` reported as failed
`ferry retry`	Re-send the messages parked in the dead-letter queue after a failed send
`ferry probe`	Diagnose a live Stoat instance (upload limits, voice, webhooks, rate-limit shape)
`ferry tls-check`	Report the CA bundle and proxy resolution Ferry is using for outbound HTTPS

Event subscription: _ProgressTracker class wraps Rich's Live display with a progress bar, status table, and warning/error output. Subscribes to the same on_event callback as the GUI.

Environment variables: STOAT_URL, STOAT_TOKEN, DISCORD_TOKEN, DISCORD_SERVER_ID. Loaded from .env via python-dotenv.

Adding a New UI

To add a new interface (e.g. REST API, TUI, Electron wrapper):

Create a new entry point that constructs a FerryConfig
Write an on_event callback that handles MigrationEvent instances
Call await run_migration(config, on_event)

No engine changes required. The event system is the only integration point.

Exporter Module (`exporter/`)

Binary Management (`manager.py`)

DCE (DiscordChatExporter) is an external export tool. Ferry downloads and caches its self-contained command-line archive automatically.

Version: Pinned to DCE_VERSION = "2.48"
Cache location: ~/.discord-ferry/bin/dce/{version}/
Source-install platform detection: Selects one of the nine upstream downloader targets: win-x64, win-arm64, win-x86, linux-musl-x64, linux-x64, linux-arm64, linux-arm, osx-x64, or osx-arm64
Packaged Ferry applications: Published as Windows x64, macOS x64, and macOS Arm64 builds. This application release list is separate from the nine DCE downloader targets available to source and Python-package installations.
Download: Fetches ZIP from GitHub Releases, extracts, validates
Runtime: All DCE 2.48 command-line archives used by Ferry are self-contained. No separate .NET installation is required. detect_dotnet() remains as a compatibility stub and does not start a process.
Retry: download_dce() retries once on network error before raising DCENotFoundError

Subprocess Execution (`runner.py`)

Runs DCE as an async subprocess with progress parsing:

async def run_dce_export(config, on_event) -> None

Launches DiscordChatExporter.Cli exportguild with the required flags
Parses DCE's stdout for per-channel progress updates
Emits MigrationEvent for each channel exported
Raises ExportError on non-zero exit code

Blueprint System (`blueprint.py`, `templates/`)

Blueprints

A ServerBlueprint captures server structure (roles, categories, channels) without messages or IDs. Useful for:

Exporting a migration's structure as a reusable template
Applying the same server layout to multiple Stoat instances

# Export blueprint from DCE exports
ferry export-blueprint --from ~/exports/my-server/ --output my-server.json

# Include cached voice caps without a login
ferry export-blueprint --from ./exports --metadata-dir ./migration-output --output server.json

# Create server from blueprint
ferry build --blueprint my-server.json --stoat-url ...

Both export interfaces accept an optional folder containing discord_metadata.json. The cache must match the single exported server. A selected missing, malformed or mismatched cache stops export without replacing the output. No folder retains the existing offline behavior.

Preset Templates

Three built-in templates in templates/:

Template	Roles	Categories	Channels
`gaming`	Admin, Moderator, Member	General, Voice, Gaming	general, announcements, game-chat, voice
`community`	Admin, Moderator, Helper, Member	Welcome, General, Voice	welcome, rules, general, help, voice
`education`	Instructor, TA, Student	Announcements, Coursework, Discussion	syllabus, assignments, q-and-a, office-hours

ferry build --template gaming --stoat-url ...

Design Decisions

Why One Engine, Not Per-Phase Executables?

Phases share state (ID maps, caches, pending lists). Running them as separate processes would require serialising and deserialising state between each step, adding complexity without benefit. A single async engine with phase functions is simpler and faster.

Why Streaming Parser?

DCE exports for large servers can be hundreds of megabytes. Loading all messages into memory at once would crash on modest hardware. The streaming parser (ijson.items) processes messages one at a time with O(1) memory. The trade-off is slightly more complex code paths (phases must call stream_messages() when metadata_only=True).

Why Masquerade + Idempotency-Key?

Stoat has no bulk message import API. Every message must be sent individually via the Ferry account. Masquerade makes each message display the original Discord author's name and avatar, preserving conversation readability. The Idempotency-Key header (ferry-{discord_msg_id}) guards against an immediate double-send within a single run. It does not protect a resume: Stoat's cache holds 1000 entries in memory, is cleared on restart, and returns HTTP 409 rather than the original message. Resume safety comes from Ferry's own markers, state.message_map and completed_channel_ids under flatten, channel_high_water and channel_message_offsets under merge.

Why Separate discord_metadata.json?

Discord permission data comes from the live Discord API, not from DCE exports. Storing it in state.json would mix "what we discovered from Discord" with "what we created on Stoat." A separate file keeps concerns clean and survives state file corruption independently.

Why Separate Category Management?

This is a Stoat API constraint, not a design choice. Categories in Stoat are a property of the server object (an array of {id, title, channels[]}), not a property of channels. There is no category_id parameter on channel creation. Ferry creates all channels first, then sends a single PATCH /servers/{id} with the full categories array.

Why No ADMINISTRATOR Permission?

Stoat does not have one. There is no equivalent to Discord's bit 3 that grants all permissions. Ferry must grant each permission individually. When translating Discord roles, ADMINISTRATOR is expanded to all individual Stoat permission bits. Channel overwrites are the exception: Discord ignores the bit there, so Ferry does too.

Why Not Use stoat.py SDK?

The project originally depended on stoat.py but replaced it with a custom aiohttp wrapper (migrator/api.py) in v0.9.0. The custom wrapper provides exact control over retry logic, rate limit handling, and British spelling conventions without depending on SDK release cycles.

Limits Reference

Resource	Stoat Default	Self-Hosted Configurable
Channels per server	200	`server_channels` in Revolt.overrides.toml
Roles per server	200	Not configurable
Custom emoji per server	100	`server_emoji`
Message length	2,000 chars	`message_length`
Attachments per message	5	Not configurable
Embeds per message	5	Not configurable
Reactions per message	20	Not configurable
Upload request body size	20 MB	`body_limit_size`

Ferry's VALIDATE phase warns when source data is likely to exceed these limits. Pass --max-channels N and --max-emoji N to match your self-hosted configuration.

Message Splitting (`migrator/messages.py`)

Messages that exceed the 2,000-character Stoat limit are split into multiple sends by _split_message():

Splits on sentence/word boundaries where possible
Each part carries a [continued K/N] marker appended to the content (budget-aware: the marker length is subtracted from the available character budget before generation)
All parts share the same masquerade and are sent in order with a small inter-part delay
The Stoat message ID of the first part is recorded in message_map for reply linking
Delivered parts commit to state before a later part's failure (PR #381, from v2.19.6). If parts 1 and 2 send and part 3 fails, part 3 enters state.failed_messages and the run keeps going; parts 1 and 2 are not re-sent on the next retry or incremental run. Earlier versions treated the whole split as one transaction and rolled the delivered parts back on failure, which produced duplicate messages on retry without reporting them.

Thread Strategies (`migrator/messages.py`, `migrator/structure.py`)

The thread_strategy config option controls how forum and thread channels are handled:

Strategy	Behaviour
`flatten`	Each thread becomes its own standalone text channel (default pre-v2)
`merge`	Thread messages are appended into the parent channel after it is processed; threads are NOT created as separate channels. Processed after all parent channels complete.
`archive`	Thread messages are written only to local markdown files at `{output_dir}/threads/{parent_channel_name}/{thread_name}.md`; no API calls are made and nothing is posted to Stoat

min_thread_messages (default 0 = include all) filters out threads with fewer messages than the threshold, useful for suppressing near-empty auto-created threads.

Filename sanitisation (from v2.19.12, PR #443). The archive strategy derives its local parent folder and markdown filename from free-form Discord parent-channel and thread names. sanitize_filename() in migrator/messages.py strips characters that are illegal on Windows (< > : " \ | ? *). It collapses forward slashes so a thread called wip/2026 does not create a subdirectory, and rewrites Win32 reserved device names (CON, NUL, etc.). During one run, threads whose sanitised names collide within a parent folder receive numeric suffixes such as _2 and _3.

Migration Lock (`core/engine.py`, `core/security.py`)

Ferry places an advisory lock in the target Stoat server's description field using the marker [FERRY_LOCK:{timestamp}:{hostname}]. This prevents two Ferry instances from migrating into the same server simultaneously.

Lock is set at the start of the SERVER phase and cleared on completion or error
Locks older than 24 hours are considered stale
--force-unlock removes a stale lock without starting a migration

Using the server description (not a dedicated channel) avoids consuming one of the 200-channel limit.

Fidelity Scoring (`reporter.py`)

The post-migration report includes a fidelity score: a 0–100 weighted score across five categories:

Category	Weight	What is measured
Messages	40%	Messages sent vs. total, failed message rate
Attachments	20%	Attachments uploaded vs. total
Embeds	15%	Embeds sent vs. dropped
Reactions	15%	Reactions applied (native mode only)
Replies	10%	Reply links resolved vs. total replies

The score is written to migration_report.json under fidelity_score.

Incremental / Delta Migration (`core/engine.py`)

--incremental mode re-uses an existing state file's ID maps to migrate only new content:

channel_map, role_map, category_map, emoji_map, message_map, and avatar_cache are carried forward from the prior state
Only channels whose Discord Snowflake ID is absent from completed_channel_ids are processed
Within a partially-migrated channel, channel_message_offsets marks the resume point
--resume and --incremental are mutually exclusive (enforced at CLI startup)

SecureTokenStore (`core/security.py`)

SecureTokenStore centralises token masking and sanitization at all output boundaries:

Tokens are stored once and never appear in log lines, event messages, or error strings
sanitize(text) replaces any stored token substring with [REDACTED] before the text reaches a shell, logger, or state file
Used by the engine, CLI, and GUI before emitting any MigrationEvent.message that could contain a request URL or header value

Per-key full-mask policy (from v2.19.9, PR #436). register() and register_secret() take a keyword-only fully_mask flag. For opaque high-entropy tokens (Stoat, Discord) the default ****{last4} tail is fine, leaking the last four characters of a 40-character random string tells an attacker nothing. For short, human-chosen secrets that tail exposes a large fraction of the value, so _strip_userinfo() in core/http.py registers both proxy_password and the derived base64 proxy_authorization header value with fully_mask=True, and masked() returns bare **** for those keys. A new redact_url() helper strips user:pass@ userinfo from a URL before it reaches any MigrationError or StoatConnectionError text, closing the eight sites where a --stoat-url https://user:pass@host was previously interpolated verbatim into error messages.

Testing

The test suite is exercised in CI on every push. For a current count run find tests -name 'test_*.py' | wc -l for the file count and uv run pytest --collect-only -q | tail -1 for the test count, a hard-coded number here rots after the first PR that adds a test. Key patterns:

Phase tests: Mock aiohttp.ClientSession with aioresponses, inject via config.session
Parser tests: Use fixture JSON files in tests/fixtures/
Transform tests: Pure function tests with edge cases (code blocks, nested formatting)
Engine tests: Inject mock phase functions via phase_overrides dict
CLI tests: Click's CliRunner with mocked engine
GUI tests: Test helper functions (ETA calculation, formatting); integration tests for cancel/pause behaviour

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=discord_ferry

# Run specific module
uv run pytest tests/test_messages.py -v