nex_shift/docs/MOBILE.md

21 KiB
Raw Permalink Blame History

Shift for iOS and Android

Planning document. It answers one question: which parts of Shift's desktop design survive translation to a phone, and how the two ends share ground. Decisions are recorded as decisions, with the reason, so a later reader knows what was considered and rejected.

What Shift actually is

Strip the desktop chrome and Shift is four things:

  1. A gesture that opens a capability menu attached to a file, not to a window.
  2. A catalogue of what each file type can become, matched to what is really installed to do the work.
  3. Local processing — convert, compress, redact, OCR, extract — with no upload.
  4. A hand-off path: the result lands next to the source, in a history library, or on another machine on the LAN.

All four translate. The desktop UI does not: a 760 px radial ring around a cursor has no mobile equivalent, and neither does "hold Shift while dragging in Finder".

The one feature worth copying

Share-drag is the feature. Everything else on desktop is either table stakes (convert, crop, compress) or platform-specific (menu bar, global hotkeys). The thing users remember is that a capability menu appears while you are already moving a file, so the action is part of the gesture instead of a place you navigate to.

iOS and Android both have exactly one gesture with that shape, and it is the one users already perform every day:

Desktop iOS Android
Shift-drag out of the Finder row, release over a wedge Share sheet, including the drag-and-drop share from Files/Photos/Safari where the picker slides out from under the finger Share sheet, same shape, from any app
Radial ring of wedges The share sheet's own app list, plus a Share Extension whose UI we own Share sheet targets, plus an Action/Share Activity we own
Ring order (desktop reads settings.json; no editor UI in 0.1.11) Share sheet "Edit Actions" ordering, mirrored in a mobile Settings surface Same, mirrored

So the mobile pitch is not "Shift in your pocket", it is: release a share over a Shift action. Concretely: long-press an image in Files → Share → Shift → HEIC → PNG, then redact, and the file comes back where it went out from. Same mental model, native gesture, no invented interaction.

Two more mobile entry points carry the same idea cheaply:

  • Quick Action / home-screen shortcuts (long-press the app icon): "Capture + convert", "Scan text", "Compress". The iOS Action Button and Android's app shortcuts give the "one gesture away" promise better than any menu.
  • Share Extension from other apps is the only sanctioned way in on both platforms; neither lets a third party overlay an arbitrary app's window the way the macOS drag bridge does. Do not plan a mobile equivalent of the global drag bridge — it does not exist, and an app that behaves like a system overlay gets rejected or uninstalled within a day.

Ecosystem surfaces

"Hook into the OS ecosystem" on mobile means a short list of sanctioned doors. Each one is an entry point with a different activation rule, and each costs one well-scoped UI. Nothing here needs an account, a server, or a background service that fights the OS.

Surface iOS Android What it buys
Files in, other apps out Share Extension (com.apple.share-services, activation rule on images/PDF/office types) ACTION_SEND / ACTION_SEND_MULTIPLE intent filter The core loop: any app's Share button reaches Shift
Files as a place File Provider extension so Shift's history library appears in the Files app sidebar DocumentsProvider exposing the same library Shift's outputs are reachable and openable from outside, not trapped in a sandbox
Open a type CFBundleDocumentTypes + UTType declarations (public.image, com.adobe.pdf, org.openxmlformats...) ACTION_VIEW filters per MIME type "Open with Shift" from mail, chat, and browser downloads
Write a type back ACTION_CREATE_DOCUMENT equivalent: iOS document-based launch + "Save on iPhone" ACTION_CREATE_DOCUMENT / ACTION_GET_CONTENT response Shift can be the saving app, so it can hand a converted file back to whatever asked
One-tap entry Home-screen Quick Actions, Action Button, App Intents (iOS 16+) so Shortcuts can call "Convert", "Redact", "Scan text" Launcher long-press shortcuts, Quick Settings tile, Assistant/app_functions.xml The "one gesture away" promise without a global hotkey, and other apps can compose Shift
Hand-off out NSItemProvider with a fileURL, plus AirDrop, Save to Photos, Copy ClipData FILE_URI through a FileProvider grant, temp-read permission The result returns where it came from, which is the desktop behaviour that matters
Widgets WidgetKit: one widget for "last capture → redact/share", one for history Glance/App widget + notification with a Copy/Send action Value without opening the app
Cross-machine NSLocalNetworkUsageDescription + _shift._tcp Bonjour browse/register ACCESS_NETWORK_STATE, CHANGE_WIFI_MULTICAST_STATE lock while discovering, NSD The LAN Send path, described in the next section

Two rules keep this from sprawling. First, every surface ships with a job list, not a window: the share extension opens, does the work, and returns to the caller; the widget taps deep-link into the library. Second, no surface may require a persistent background service. iOS will kill one and Android's battery optimisers will throttle it, and a Shift that silently stopped receiving is worse than one that never claimed to.

Making the two ends actually meet

Discovery is the part that breaks between ecosystems, so it gets explicit answers:

  • Same-LAN discovery. Desktop already beacons UDP multicast to 239.255.77.88:47888 with broadcast and unicast fallbacks. Phones keep that as a fallback but lead with Bonjour/mDNS on iOS and NSD on Android, because both give a wake-up that survives doze and neither needs a multicast lock held hostage. The desktop registers _shift._tcp alongside the existing beacon, so a phone finds it by name instead of guessing.
  • First contact is a pairing code, not a prompt. A phone joining a network with five desktops needs to pick one. The desktop shows a short code and the phone types it (or scans a QR encoding host, port, and code); the desktop then pins that phone's identity the way SSH does. This removes the "who is Desktop-2.local?" problem and gives the consent card something to vouch for.
  • Background receiving is asymmetric, and stays that way. iOS gives a socket a few seconds of life, so a phone is a first-class sender and a wake-to-receive peer: a push or a tap opens the app, which then pulls the queued job. Desktop-to-phone therefore behaves like a pull: the desktop queues, the phone collects. Saying so beats shipping a receiver that drops files at random.
  • No cloud relay, still. The pairing code is what makes a user-run hub workable if a remote case ever appears — one desktop as the hub, the phone dialling it — without inventing a hosted service, accounts, or a retention policy.

Capability map

Legend: Native = OS API, do not reimplement. Port = existing Shift logic moves with little change. Substitute = the desktop mechanism is unavailable, use the listed replacement. Skip = no mobile answer that fits the product.

Desktop capability Mobile Mechanism
Radial ring Substitute Share-sheet extension with its own ring-style picker; the wedge catalogue is shared code, the geometry is a phone-sized arc
Ring catalogue + custom order Port src/lib/radialOrder.ts is pure TypeScript with no Tauri import; reuse it verbatim, persist orders in app group / SharedPreferences
Format catalogue (conversionFormats) Port Pure TS, same table; prune per platform at build time where a codec is absent
Image convert (PNG/JPG/WebP/TIFF/BMP/GIF) Port Keep the Rust image crate — it builds for aarch64-apple-ios and aarch64-linux-android; or substitute ImageIO/ImageDecoder
HEIC decode/encode Native ImageIO on iOS, HEIF support in Android's decoder (API 28+); desktop's sips/ImageMagick has no mobile role
Image pixel tools (rotate, flip, B&W, 50%, crop, compress) Port Same math. On desktop these now decode HEIC/AVIF through open_for_pixels, which shells out to sips/ImageMagick; that function is the one thing in the path to replace, and both mobile platforms decode those formats natively, so the mobile core needs an injected decoder rather than a subprocess
Video/audio convert, compress, 720p, mute, extract audio, mono, frame Native AVFoundation on iOS, Media3/MediaCodec on Android; mobile-ffmpeg-style bundled FFmpeg works but costs 15–25 MB and licence review for GPL builds — decide per platform, not globally
Video GIF Native + port Frame sampling natively, palette-quantise in Rust (desktop already caps GIF at 15 s / 12 fps / 480 px / 8 MB — keep those caps, phones care more)
PDF text, split, merge, rotate Port lopdf is pure Rust; or native PDFKit/PdfRenderer for page images
PDF → images Native PDFKit (iOS), PdfRenderer (Android) — replaces PDFKit-sidecar/Poppler
PDF → DOCX Substitute Native page render + the existing pure-Rust DOCX packager; the macOS AppKit renderer has no mobile twin
PPTX / XLSX / CSV → Markdown Port calamine + quick-xml, pure Rust, no platform dependency. Highest value-to-effort item on the list
Document → PDF Substitute iOS: UIPrintRenderer/UIMarkupTextPrintFormatter; Android: PdfRenderer/WebView print. Neither matches the AppKit renderer's fidelity; say so in the UI
Annotate: rectangle, arrow, text, highlight, number Port The canvas model is React/TS; a mobile build reuses it, though native PencilKit is better for finger/stylus
Annotate: redact, pixelate, blur, stamps Port Same drawing primitives; redaction stays "burned in", which is the whole point
Cutout (corner-sample keying) Port Works, but iOS VNGeneratePersonSegmentationRequest / Android segmenter are dramatically better — substitute where available
OCR EN + ZH Native Drop Tesseract WASM on mobile. Vision (VNRecognizeTextRequest, .enUS+.zh-Hans) and ML Kit Text Recognition v2 are faster, smaller and better on-device engines. Keep Tesseract only if one shared engine matters more than quality
Screen capture Native iOS ReplayKit (broadcast or one-shot, user-initiated); Android MediaProjection, plus the built-in screenshot + MediaStore observer
Screen recording Native ReplayKit / MediaProjection; frame-rate honesty (capabilities()) carries straight over — mobile has the same "backend ignores this" problem
Result card (Copy / Pin / Send / Reveal / Edit) Substitute No floating always-on-top windows. iOS: the Share-sheet extension's own completion UI + Live Activity where useful; Android: a floating service (easy) and a Quick Settings tile
History library Port history.rs logic (capacity, pinning, missing files) is portable; storage becomes the app group container / scoped storage. Search + pin behave identically
Drag a row out Native NSItemProvider with fileURL on iOS, ClipData FILE_URI + FileProvider on Android — same intent as text/uri-list with a file:// entry
Pin to screen Substitute Android: floating overlay window. iOS: no API — pinned images become a Widget or nothing
Menu bar / tray Substitute iOS: Menu Bar Extra is not a thing; share sheet + Shortcuts actions. Android: Quick Settings tile, launcher icon long-press
Global hotkeys Skip Do not request Accessibility for a keyboard shortcut. Offer App Intents (iOS 16+) / Quick Settings tile + Assistant shortcuts
Launch at login Skip Meaningless on mobile
LAN Send (SHSF beacons + TCP) Port UDP multicast + direct TCP work on both platforms; needs NSLocalNetworkUsageDescription + Bonjour on iOS (the privacy prompt is unavoidable) and ACCESS_NETWORK_STATE/multicast lock on Android. Background reception is unreliable on iOS, so treat mobile as a first-class sender and a best-effort receiver
Update channel on git Skip Store review is the only permitted update path for executable code. Keep the SHA256 discipline for content (models, catalogues), not for the binary
Runtime components panel Substitute Nothing is installable. Show "detected codecs" read-only, or drop it; do not fake a desktop panel
i18n en/zh Port The two message tables are data; move them as-is

Architecture for two ecosystems

The claim that the core is portable was checked rather than assumed, against this repository as it stands:

  • cargo metadata --no-deps reports one workspace package, shift. Nothing is separable yet, so step zero is a real refactor, not a packaging detail.
  • src-tauri/src/pdf.rs and src-tauri/src/office.rs contain zero references to tauri:: or AppHandle. Their functions already take &Path and return Result<PathBuf, String>, which is exactly the shape a C header wraps. They move essentially unchanged.
  • src-tauri/src/files.rs is 2,761 lines with 11 #[tauri::command] wrappers, and 28 of its lines mention tauri:: or AppHandle. The wrappers are thin — the workers beneath them already take &Path and an output_dir: Option<&Path> — but the settings lookups inside the command layer (files_output_dir, auto_reveal) have to become parameters, and the history::record call each command makes after a write becomes a callback.
  • The dependency closure of image, lopdf, printpdf, calamine, quick-xml, zip, sha2, ed25519-dalek, base64, serde and serde_json is 161 crates, and not one of them needs a C compiler: the only system-flavoured members are libc, windows-sys, adler2 and miniz_oxide, all of which build for iOS and Android. There is no AV1 or HEVC encoder in the tree, which is why HEIC and video stay native on mobile.

So the core is genuinely the leverage: image, lopdf, printpdf, calamine, quick-xml, zip, sha2 cross-compile to aarch64-apple-ios, aarch64-apple-ios-sim and aarch64-linux-android. Put the conversion pipeline, the wedge catalogue, the PDF and Office readers, and the SHSF protocol in a Rust static library and let both platforms call it; that is where most of the value is, and it is the part with no platform API in it.

shared/
├─ shift-core (Rust staticlib + C header)
│   ├─ convert / compress / pixel tools / pdf / office  ← from src-tauri/src/files.rs, pdf.rs, office.rs
│   ├─ catalogue            ← from src/lib/radialOrder.ts + conversionFormats.ts (generated or mirrored)
│   └─ shsf                 ← from src-tauri/src/share.rs
├─ ios/       Swift UI + Share Extension + App Intents, links shift-core
└─ android/   Kotlin UI + Share Activity + Tile, JNI to shift-core

Step zero, before any UI: split the workspace into shift-core (the pure functions above, plus a decoder trait standing in for open_for_pixels), a shift-ffi staticlib over it, and the existing Tauri app consuming shift-core. The desktop then exercises the same code the phones ship, which is the only thing that keeps three platforms from drifting. That split is verifiable on desktop alone — cargo test --lib keeps passing while the pure functions move out from under the command layer.

Two consequences worth deciding early:

  • The wedge catalogue should stop being TypeScript-only. It is currently the source of truth for the desktop ring. Either generate the mobile lists from it, or move it into shift-core and have the desktop read it. Do not maintain three hand-written copies.
  • Tauri does not run on iOS/Android usefully here. Tauri v2 can target mobile, but the product is a share-sheet extension, which is a small native UI, not a webview app. Plan a native shell around the Rust core; keep the desktop's React UI on desktop.

What mobile must send to desktop

Cross-ecosystem hand-off is the feature that justifies a mobile app at all. The current SHSF frame is: four magic bytes SHSF, a big-endian u32 header length, a JSON header (fileName, size, peerName, transferId, batchId, batchIndex, batchTotal), the raw file bytes, then a one-byte accept decision. Notably there is no version field — compatibility today comes from #[serde(default)] letting a newer reader accept an older header. That is enough for two desktop revisions and not enough once a phone app is out in the wild and cannot be force-updated.

Before the first mobile release, freeze the protocol:

"SHSF" | version:u8 = 1 | headerLen:u32 | header:JSON | payload | decision:u8

Plus, in the header: platform, appVersion, kind ("file" | "job"), and for jobs action + args. Everything else the mobile app needs is a job it can describe:

  1. Phone → desktop, direct. LAN transfer, receiver stores to its output folder, both histories update. This is Send with a phone as a peer. Small change: the peer beacon needs a kind so a phone does not look like a desktop, and the consent card needs to say "iPhone sent this".
  2. Phone → desktop as a job. A phone encodes 4K video or a 900-page PDF poorly. Send the file plus a requested action, the desktop runs it in the existing job pipeline (perform_file_action already reports progress and writes conflict-safe names), and the result comes back. The desktop queue already exists; the phone needs a second, smaller queue that only shows status.
  3. Desktop → phone. Symmetric; a phone receiving a file is one URLSession/OkHttp upload to the desktop's existing receive port.

Deliberately not part of the plan: a cloud relay. Shift's whole claim is that nothing leaves the machine, and a relay would need authentication, retention limits, and a privacy policy for a feature that works fine over the LAN. If a remote case ever demands it, the right shape is user-run (one desktop is the hub, the phone dials it with a pairing code), not a hosted service.

The privacy posture carries over unchanged: no telemetry, no analytics SDK, no crash reporter that uploads file content. On iOS that also means the local-network permission string and a Bonjour service type in Info.plist — write them before the first submission, because the denial is silent and the feature looks broken.

Build order

Each step ships something usable on its own; none depends on a feature Shift lacks today.

  1. Extract shift-core on desktop. Move the pure conversion, PDF and Office code out from under the Tauri command layer and add the staticlib target. Ships nothing new to users, and it is the step that can be finished and verified without an Apple developer account or an Android SDK: cargo test --lib keeps passing the whole way, and the tests that follow the functions prove the boundary is real.
  2. iOS share-sheet converter for images. shift-core + the ported catalogue + a ring-style picker. No networking, no capture, no accounts. Proves the gesture.
  3. Android share-sheet converter. Same core, same catalogue; the second platform is the test that the abstraction is real rather than iOS-shaped.
  4. PDF and Office text extraction. Pure-Rust ports, no platform APIs, high perceived value ("turn this deck into Markdown on my phone").
  5. Native OCR (Vision / ML Kit) + annotation. Redact first — it is the mobile-native killer use (screenshot → redact → share, all on device).
  6. LAN peer + phone→desktop jobs. Add the version byte and the kind field to SHSF here, in the same release as the first phone that speaks it.
  7. Capture on mobile, last. ReplayKit/MediaProjection is real work, mobile screenshots are already excellent, and the desktop capture loop is the least portable thing in the product. Do it when users ask for it rather than for parity.

Open decisions

Question Leaning Why
Tauri mobile or native shell Native shell + Rust core Share extensions are small native UIs; a webview adds size and latency to the one interaction that must feel instant
One FFmpeg binary per platform iOS: no, Android: measure AVFoundation covers most mobile media needs at zero licence risk; Android's codecs are as capable but the surface is fragmented
Shared OCR engine or native Native Vision and ML Kit beat Tesseract WASM on speed and Chinese accuracy; a shared engine buys nothing users can see
Catalogue home shift-core Desktop can read from it; phones and desktop cannot drift apart
Priced app to fund it Undecided The desktop app is MIT and free; a paid mobile app is defensible but changes the project's centre of gravity