ewan's projects — docs

projects

Malachite

March 2, 2026

# malachite# atproto# lastfm# spotify# tools

Note: The standalone malachite repository has been archived. Development continues in the @ewanc26/pkgs monorepo — CLI at packages/malachite/, web frontend at packages/malachite-web/.

Malachite is a tool for importing your Last.fm and Spotify listening history to the AT Protocol network as fm.teal.alpha.feed.play records. It's designed to be safe, resumable, and smart about rate limits — so you don't accidentally hammer your PDS.

The name is a deliberate nod to the teal lexicon it publishes to: malachite is a greenish-blue copper mineral associated with preservation and transformation, sitting squarely in that teal/green colour range.

Usage Options

Malachite comes in two forms:

Web interface — the easiest way to get started. Visit malachite.croft.click, authenticate via ATProto OAuth (recommended) or an app password, upload your export files, and import. Everything runs locally in your browser — no data is sent to any server other than your own PDS.

CLI — a Node.js command-line tool for local use. Useful if you want full control over batch settings, need to automate imports, or simply prefer the terminal. Requires cloning the repository and building from source.

Web Interface

No installation required. Open malachite.croft.click and follow the wizard:

  1. Choose a mode — Last.fm, Spotify, combined, sync, or deduplicate
  2. Sign in — via ATProto OAuth (recommended; redirects to your PDS and back, your credentials are never shared with Malachite) or an app password
  3. Upload your export — CSV or JSON, parsed entirely in the browser (skipped in deduplicate mode)
  4. Options — optionally enable dry run, reverse chronological order, or skip the Teal duplicate check
  5. Run — records are published directly to your PDS with automatic rate-limit handling

The web app is built with SvelteKit.

CLI

Prerequisites

  • Node.js v18 or later
  • pnpm (recommended) — or npm / yarn

Install and Build

# Clone the monorepo
git clone https://github.com/ewanc26/pkgs.git
cd pkgs

# Install all workspace dependencies
pnpm install

# Build the malachite CLI
pnpm --filter @ewanc26/malachite build

Quick Start

# Run in interactive mode (recommended for first-time use)
pnpm --filter @ewanc26/malachite start

# Or with command line arguments
pnpm --filter @ewanc26/malachite start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y

Interactive mode walks you through everything: choosing a mode, entering credentials, picking files, and setting optional flags.

Common Invocations

All commands run from the pkgs root:

# Import from Last.fm CSV
pnpm --filter @ewanc26/malachite start -- -i lastfm.csv -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y

# Import from Spotify JSON export
pnpm --filter @ewanc26/malachite start -- -i spotify-export/ -m spotify -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y

# Merge both sources
pnpm --filter @ewanc26/malachite start -- -i lastfm.csv --spotify-input spotify-export/ -m combined -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y

# Sync (skip already-imported records)
pnpm --filter @ewanc26/malachite start -- -i lastfm.csv -m sync -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx -y

# Remove duplicates from your Teal feed
pnpm --filter @ewanc26/malachite start -- -m deduplicate -h alice.bsky.social -p xxxx-xxxx-xxxx-xxxx

# Preview without publishing
pnpm --filter @ewanc26/malachite start -- -i lastfm.csv --dry-run

Import Modes

Mode Flag Description
Last.fm -m lastfm (default) Import a Last.fm CSV export
Spotify -m spotify Import Spotify Extended Streaming History JSON
Combined -m combined Merge both sources with deduplication
Sync -m sync Skip records that already exist on Teal
Deduplicate -m deduplicate Remove duplicate records already on Teal

Command Line Options

Required

Option Short Description
--input <path> -i Path to Last.fm CSV or Spotify JSON file/directory
--handle <handle> -h Your ATProto handle or DID
--password <pass> -p Your ATProto app password (not your main password)

Common Options

Option Short Description
--mode <mode> -m Import mode (see table above)
--spotify-input <path> Spotify export path (for combined mode)
--reverse -r Process newest tracks first
--yes -y Skip confirmation prompts
--dry-run Preview records without publishing
--verbose -v Debug-level logging
--quiet -q Warnings and errors only
--dev Verbose + file logging + smaller batches
--pds <url> Skip identity resolution and use a known PDS URL directly

Getting Your Data

Last.fm: Export your scrobbles from lastfm.ghan.nl/export as a CSV.

Spotify: Go to Spotify Privacy Settings, request your "Extended streaming history" (takes up to 30 days), then use either a single Streaming_History_Audio_*.json file or the whole extracted directory. Malachite automatically filters out podcasts and non-music content.

Duplicate Prevention

Malachite has two layers of protection against duplicates:

Input deduplication — before publishing anything, it removes entries within your source file that share the same track name, artist, and timestamp.

Teal comparison via CAR export — it downloads your entire repo as a single CARv1 file using com.atproto.sync.getRepo (the sync namespace, not the AppView), parses it locally, and skips anything already imported. This costs zero AppView write-quota points. It runs automatically for every mode; credentials are required even for dry runs.

Rate Limiting

Bluesky's AppView enforces rate limits on PDS instances. Exceeding 10K records per day can rate-limit your entire PDS — affecting all users on it, not just your account.

Malachite handles this automatically:

  • Monitors your rate limit quota in real-time from response headers
  • Dynamically adjusts batch size between 1 and 200 records
  • Maintains a 15% headroom buffer so the quota is never fully exhausted
  • Hard daily cap of 7,500 records (75% safety margin)
  • Pauses 24 hours between days for large imports
  • Scales immediately back to maximum speed after a quota reset

File Storage

All CLI data is stored in ~/.malachite/:

~/.malachite/
├── cache/            # Cached Teal records (24-hour TTL)
├── state/            # Import state for resume support
├── logs/             # Logs when --dev is active
└── credentials.json  # AES-256-GCM encrypted credentials (optional)

Credentials are saved automatically after every successful login — no separate prompt. They are encrypted using a key derived from your hostname and username, making them machine-specific. Clear them with pnpm start --clear-credentials.

Record Format

Each scrobble is published as an fm.teal.alpha.feed.play record. Required fields are trackName, artists, playedTime, submissionClientAgent, and musicServiceBaseDomain. Last.fm imports also include MusicBrainz IDs when available.

Example Last.fm record:

{
  "$type": "fm.teal.alpha.feed.play",
  "trackName": "Paint My Masterpiece",
  "artists": [{ "artistName": "Cjbeards", "artistMbId": "c8d4f4bf-..." }],
  "releaseName": "Masquerade",
  "playedTime": "2025-11-13T23:49:36Z",
  "originUrl": "https://www.last.fm/music/Cjbeards/_/Paint+My+Masterpiece",
  "submissionClientAgent": "malachite/v0.10.0",
  "musicServiceBaseDomain": "last.fm"
}

Development

All commands run from the pkgs root:

pnpm --filter @ewanc26/malachite run type-check   # Type checking
pnpm --filter @ewanc26/malachite run build        # Build
pnpm --filter @ewanc26/malachite run dev          # Rebuild and run
pnpm --filter @ewanc26/malachite run test         # Run tests
pnpm --filter @ewanc26/malachite run clean        # Clean build artifacts

# Web frontend
pnpm --filter @ewanc26/malachite-web dev          # Dev server at http://127.0.0.1:5173
pnpm --filter @ewanc26/malachite-web build        # Build for deployment

Lexicon

Malachite publishes to the fm.teal.alpha lexicon. The schema definitions live in /lexicons/fm.teal.alpha/ in the repository.

License

AGPL-3.0-only.

← all docs