nex_shift/docs/MOBILE.md

266 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

# Shift for iOS and Android
Planning document. It answers one question: which parts of Shift's desktop design survive
translation to a phone, and how the two ends share ground. Decisions are recorded as
decisions, with the reason, so a later reader knows what was considered and rejected.
## What Shift actually is
Strip the desktop chrome and Shift is four things:
1. A **gesture** that opens a capability menu attached to a file, not to a window.
2. A **catalogue** of what each file type can become, matched to what is really installed
to do the work.
3. **Local processing** — convert, compress, redact, OCR, extract — with no upload.
4. A **hand-off** path: the result lands next to the source, in a history library, or on
another machine on the LAN.
All four translate. The desktop UI does not: a 760 px radial ring around a cursor has no
mobile equivalent, and neither does "hold Shift while dragging in Finder".
## The one feature worth copying
**Share-drag is the feature.** Everything else on desktop is either table stakes
(convert, crop, compress) or platform-specific (menu bar, global hotkeys). The thing users
remember is that a capability menu appears *while you are already moving a file*, so the
action is part of the gesture instead of a place you navigate to.
iOS and Android both have exactly one gesture with that shape, and it is the one users
already perform every day:
| Desktop | iOS | Android |
| --- | --- | --- |
| Shift-drag out of the Finder row, release over a wedge | **Share sheet**, including the drag-and-drop share from Files/Photos/Safari where the picker slides out from under the finger | **Share sheet**, same shape, from any app |
| Radial ring of wedges | The share sheet's own app list, plus a Share Extension whose UI we own | Share sheet targets, plus an Action/Share Activity we own |
| Ring order (desktop reads `settings.json`; no editor UI in 0.1.11) | Share sheet "Edit Actions" ordering, mirrored in a mobile Settings surface | Same, mirrored |
So the mobile pitch is not "Shift in your pocket", it is: **release a share over a Shift
action**. Concretely: long-press an image in Files → Share → Shift → *HEIC → PNG, then
redact*, and the file comes back where it went out from. Same mental model, native
gesture, no invented interaction.
Two more mobile entry points carry the same idea cheaply:
- **Quick Action / home-screen shortcuts** (long-press the app icon): "Capture + convert",
"Scan text", "Compress". The iOS Action Button and Android's app shortcuts give the
"one gesture away" promise better than any menu.
- **Share Extension from other apps** is the only sanctioned way in on both platforms;
neither lets a third party overlay an arbitrary app's window the way the macOS drag
bridge does. Do not plan a mobile equivalent of the global drag bridge — it does not
exist, and an app that behaves like a system overlay gets rejected or uninstalled
within a day.
## Ecosystem surfaces
"Hook into the OS ecosystem" on mobile means a short list of sanctioned doors. Each one is
an entry point with a different activation rule, and each costs one well-scoped UI. Nothing
here needs an account, a server, or a background service that fights the OS.
| Surface | iOS | Android | What it buys |
| --- | --- | --- | --- |
| Files in, other apps out | Share Extension (`com.apple.share-services`, activation rule on images/PDF/office types) | `ACTION_SEND` / `ACTION_SEND_MULTIPLE` intent filter | The core loop: any app's Share button reaches Shift |
| Files as a place | File Provider extension so Shift's history library appears in the Files app sidebar | `DocumentsProvider` exposing the same library | Shift's outputs are reachable *and* openable from outside, not trapped in a sandbox |
| Open a type | `CFBundleDocumentTypes` + UTType declarations (`public.image`, `com.adobe.pdf`, `org.openxmlformats...`) | `ACTION_VIEW` filters per MIME type | "Open with Shift" from mail, chat, and browser downloads |
| Write a type back | `ACTION_CREATE_DOCUMENT` equivalent: iOS document-based launch + "Save on iPhone" | `ACTION_CREATE_DOCUMENT` / `ACTION_GET_CONTENT` response | Shift can be the *saving* app, so it can hand a converted file back to whatever asked |
| One-tap entry | Home-screen Quick Actions, Action Button, App Intents (iOS 16+) so Shortcuts can call "Convert", "Redact", "Scan text" | Launcher long-press shortcuts, Quick Settings tile, Assistant/`app_functions.xml` | The "one gesture away" promise without a global hotkey, and other apps can compose Shift |
| Hand-off out | `NSItemProvider` with a `fileURL`, plus AirDrop, Save to Photos, Copy | `ClipData` `FILE_URI` through a `FileProvider` grant, temp-read permission | The result returns where it came from, which is the desktop behaviour that matters |
| Widgets | WidgetKit: one widget for "last capture → redact/share", one for history | Glance/App widget + notification with a Copy/Send action | Value without opening the app |
| Cross-machine | `NSLocalNetworkUsageDescription` + `_shift._tcp` Bonjour browse/register | `ACCESS_NETWORK_STATE`, `CHANGE_WIFI_MULTICAST_STATE` lock while discovering, NSD | The LAN Send path, described in the next section |
Two rules keep this from sprawling. First, **every surface ships with a job list, not a
window**: the share extension opens, does the work, and returns to the caller; the widget
taps deep-link into the library. Second, **no surface may require a persistent background
service**. iOS will kill one and Android's battery optimisers will throttle it, and a Shift
that silently stopped receiving is worse than one that never claimed to.
### Making the two ends actually meet
Discovery is the part that breaks between ecosystems, so it gets explicit answers:
- **Same-LAN discovery.** Desktop already beacons UDP multicast to `239.255.77.88:47888`
with broadcast and unicast fallbacks. Phones keep that as a fallback but lead with
Bonjour/mDNS on iOS and NSD on Android, because both give a wake-up that survives
doze and neither needs a multicast lock held hostage. The desktop registers `_shift._tcp`
alongside the existing beacon, so a phone finds it by name instead of guessing.
- **First contact is a pairing code, not a prompt.** A phone joining a network with five
desktops needs to pick one. The desktop shows a short code and the phone types it (or
scans a QR encoding host, port, and code); the desktop then pins that phone's identity
the way SSH does. This removes the "who is `Desktop-2.local`?" problem and gives the
consent card something to vouch for.
- **Background receiving is asymmetric, and stays that way.** iOS gives a socket a few
seconds of life, so a phone is a first-class sender and a *wake-to-receive* peer: a push
or a tap opens the app, which then pulls the queued job. Desktop-to-phone therefore
behaves like a pull: the desktop queues, the phone collects. Saying so beats shipping a
receiver that drops files at random.
- **No cloud relay, still.** The pairing code is what makes a *user-run* hub workable if a
remote case ever appears — one desktop as the hub, the phone dialling it — without
inventing a hosted service, accounts, or a retention policy.
## Capability map
Legend: **Native** = OS API, do not reimplement. **Port** = existing Shift logic moves
with little change. **Substitute** = the desktop mechanism is unavailable, use the listed
replacement. **Skip** = no mobile answer that fits the product.
| Desktop capability | Mobile | Mechanism |
| --- | --- | --- |
| Radial ring | Substitute | Share-sheet extension with its own ring-style picker; the wedge catalogue is shared code, the geometry is a phone-sized arc |
| Ring catalogue + custom order | Port | `src/lib/radialOrder.ts` is pure TypeScript with no Tauri import; reuse it verbatim, persist orders in app group / `SharedPreferences` |
| Format catalogue (`conversionFormats`) | Port | Pure TS, same table; prune per platform at build time where a codec is absent |
| Image convert (PNG/JPG/WebP/TIFF/BMP/GIF) | Port | Keep the Rust `image` crate — it builds for `aarch64-apple-ios` and `aarch64-linux-android`; or substitute `ImageIO`/`ImageDecoder` |
| HEIC decode/encode | Native | `ImageIO` on iOS, HEIF support in Android's decoder (API 28+); desktop's `sips`/ImageMagick has no mobile role |
| Image pixel tools (rotate, flip, B&W, 50%, crop, compress) | Port | Same math. On desktop these now decode HEIC/AVIF through `open_for_pixels`, which shells out to `sips`/ImageMagick; that function is the one thing in the path to replace, and both mobile platforms decode those formats natively, so the mobile core needs an injected decoder rather than a subprocess |
| Video/audio convert, compress, 720p, mute, extract audio, mono, frame | Native | `AVFoundation` on iOS, `Media3`/`MediaCodec` on Android; `mobile-ffmpeg`-style bundled FFmpeg works but costs 15–25 MB and licence review for GPL builds — decide per platform, not globally |
| Video GIF | Native + port | Frame sampling natively, palette-quantise in Rust (desktop already caps GIF at 15 s / 12 fps / 480 px / 8 MB — keep those caps, phones care more) |
| PDF text, split, merge, rotate | Port | `lopdf` is pure Rust; or native `PDFKit`/`PdfRenderer` for page images |
| PDF → images | Native | `PDFKit` (iOS), `PdfRenderer` (Android) — replaces PDFKit-sidecar/Poppler |
| PDF → DOCX | Substitute | Native page render + the existing pure-Rust DOCX packager; the macOS AppKit renderer has no mobile twin |
| PPTX / XLSX / CSV → Markdown | Port | `calamine` + `quick-xml`, pure Rust, no platform dependency. Highest value-to-effort item on the list |
| Document → PDF | Substitute | iOS: `UIPrintRenderer`/`UIMarkupTextPrintFormatter`; Android: `PdfRenderer`/WebView print. Neither matches the AppKit renderer's fidelity; say so in the UI |
| Annotate: rectangle, arrow, text, highlight, number | Port | The canvas model is React/TS; a mobile build reuses it, though native `PencilKit` is better for finger/stylus |
| Annotate: redact, pixelate, blur, stamps | Port | Same drawing primitives; redaction stays "burned in", which is the whole point |
| Cutout (corner-sample keying) | Port | Works, but iOS `VNGeneratePersonSegmentationRequest` / Android segmenter are dramatically better — substitute where available |
| OCR EN + ZH | Native | **Drop Tesseract WASM on mobile.** `Vision` (`VNRecognizeTextRequest`, `.enUS`+`.zh-Hans`) and ML Kit Text Recognition v2 are faster, smaller and better on-device engines. Keep Tesseract only if one shared engine matters more than quality |
| Screen capture | Native | iOS `ReplayKit` (broadcast or one-shot, user-initiated); Android MediaProjection, plus the built-in screenshot + `MediaStore` observer |
| Screen recording | Native | `ReplayKit` / MediaProjection; frame-rate honesty (`capabilities()`) carries straight over — mobile has the same "backend ignores this" problem |
| Result card (Copy / Pin / Send / Reveal / Edit) | Substitute | No floating always-on-top windows. iOS: the Share-sheet extension's own completion UI + Live Activity where useful; Android: a floating service (easy) and a Quick Settings tile |
| History library | Port | `history.rs` logic (capacity, pinning, missing files) is portable; storage becomes the app group container / scoped storage. Search + pin behave identically |
| Drag a row out | Native | `NSItemProvider` with `fileURL` on iOS, `ClipData` `FILE_URI` + `FileProvider` on Android — same intent as `text/uri-list` with a `file://` entry |
| Pin to screen | Substitute | Android: floating overlay window. iOS: no API — pinned images become a Widget or nothing |
| Menu bar / tray | Substitute | iOS: Menu Bar Extra is not a thing; share sheet + Shortcuts actions. Android: Quick Settings tile, launcher icon long-press |
| Global hotkeys | Skip | Do not request Accessibility for a keyboard shortcut. Offer App Intents (iOS 16+) / Quick Settings tile + Assistant shortcuts |
| Launch at login | Skip | Meaningless on mobile |
| LAN Send (SHSF beacons + TCP) | Port | UDP multicast + direct TCP work on both platforms; needs `NSLocalNetworkUsageDescription` + Bonjour on iOS (the privacy prompt is unavoidable) and `ACCESS_NETWORK_STATE`/multicast lock on Android. Background reception is unreliable on iOS, so treat mobile as a first-class *sender* and a best-effort receiver |
| Update channel on git | Skip | Store review is the only permitted update path for executable code. Keep the SHA256 discipline for *content* (models, catalogues), not for the binary |
| Runtime components panel | Substitute | Nothing is installable. Show "detected codecs" read-only, or drop it; do not fake a desktop panel |
| i18n en/zh | Port | The two message tables are data; move them as-is |
## Architecture for two ecosystems
The claim that the core is portable was checked rather than assumed, against this
repository as it stands:
- `cargo metadata --no-deps` reports one workspace package, `shift`. Nothing is separable
yet, so step zero is a real refactor, not a packaging detail.
- `src-tauri/src/pdf.rs` and `src-tauri/src/office.rs` contain **zero** references to
`tauri::` or `AppHandle`. Their functions already take `&Path` and return
`Result<PathBuf, String>`, which is exactly the shape a C header wraps. They move
essentially unchanged.
- `src-tauri/src/files.rs` is 2,761 lines with 11 `#[tauri::command]` wrappers, and 28 of
its lines mention `tauri::` or `AppHandle`. The wrappers are thin — the workers beneath
them already take `&Path` and an `output_dir: Option<&Path>` — but the settings lookups
inside the command layer (`files_output_dir`, `auto_reveal`) have to become parameters,
and the `history::record` call each command makes after a write becomes a callback.
- The dependency closure of `image`, `lopdf`, `printpdf`, `calamine`, `quick-xml`, `zip`,
`sha2`, `ed25519-dalek`, `base64`, `serde` and `serde_json` is 161 crates, and not one of
them needs a C compiler: the only system-flavoured members are `libc`, `windows-sys`,
`adler2` and `miniz_oxide`, all of which build for iOS and Android. There is no AV1 or
HEVC encoder in the tree, which is why HEIC and video stay native on mobile.
So the core is genuinely the leverage: `image`, `lopdf`, `printpdf`, `calamine`,
`quick-xml`, `zip`, `sha2` cross-compile to `aarch64-apple-ios`, `aarch64-apple-ios-sim`
and `aarch64-linux-android`. Put the conversion pipeline, the wedge catalogue, the PDF and
Office readers, and the SHSF protocol in a Rust static library and let both platforms call
it; that is where most of the value is, and it is the part with no platform API in it.
```text
shared/
├─ shift-core (Rust staticlib + C header)
│ ├─ convert / compress / pixel tools / pdf / office ← from src-tauri/src/files.rs, pdf.rs, office.rs
│ ├─ catalogue ← from src/lib/radialOrder.ts + conversionFormats.ts (generated or mirrored)
│ └─ shsf ← from src-tauri/src/share.rs
├─ ios/ Swift UI + Share Extension + App Intents, links shift-core
└─ android/ Kotlin UI + Share Activity + Tile, JNI to shift-core
```
Step zero, before any UI: split the workspace into `shift-core` (the pure functions above,
plus a decoder trait standing in for `open_for_pixels`), a `shift-ffi` staticlib over it,
and the existing Tauri app consuming `shift-core`. The desktop then exercises the same code
the phones ship, which is the only thing that keeps three platforms from drifting. That
split is verifiable on desktop alone — `cargo test --lib` keeps passing while the pure
functions move out from under the command layer.
Two consequences worth deciding early:
- **The wedge catalogue should stop being TypeScript-only.** It is currently the source of
truth for the desktop ring. Either generate the mobile lists from it, or move it into
`shift-core` and have the desktop read it. Do not maintain three hand-written copies.
- **Tauri does not run on iOS/Android usefully here.** Tauri v2 can target mobile, but the
product is a share-sheet extension, which is a small native UI, not a webview app. Plan
a native shell around the Rust core; keep the desktop's React UI on desktop.
## What mobile must send to desktop
Cross-ecosystem hand-off is the feature that justifies a mobile app at all. The current
`SHSF` frame is: four magic bytes `SHSF`, a big-endian `u32` header length, a JSON header
(`fileName`, `size`, `peerName`, `transferId`, `batchId`, `batchIndex`, `batchTotal`), the
raw file bytes, then a one-byte accept decision. Notably **there is no version field** —
compatibility today comes from `#[serde(default)]` letting a newer reader accept an older
header. That is enough for two desktop revisions and not enough once a phone app is out in
the wild and cannot be force-updated.
Before the first mobile release, freeze the protocol:
```text
"SHSF" | version:u8 = 1 | headerLen:u32 | header:JSON | payload | decision:u8
```
Plus, in the header: `platform`, `appVersion`, `kind` ("file" | "job"), and for jobs
`action` + `args`. Everything else the mobile app needs is a job it can describe:
1. **Phone → desktop, direct.** LAN transfer, receiver stores to its output folder, both
histories update. This is Send with a phone as a peer. Small change: the peer beacon
needs a `kind` so a phone does not look like a desktop, and the consent card needs to
say "iPhone sent this".
2. **Phone → desktop as a job.** A phone encodes 4K video or a 900-page PDF poorly. Send
the file plus a requested action, the desktop runs it in the existing job pipeline
(`perform_file_action` already reports progress and writes conflict-safe names), and the
result comes back. The desktop queue already exists; the phone needs a second, smaller
queue that only shows status.
3. **Desktop → phone.** Symmetric; a phone receiving a file is one `URLSession`/`OkHttp`
upload to the desktop's existing receive port.
Deliberately **not** part of the plan: a cloud relay. Shift's whole claim is that nothing
leaves the machine, and a relay would need authentication, retention limits, and a privacy
policy for a feature that works fine over the LAN. If a remote case ever demands it, the
right shape is user-run (one desktop is the hub, the phone dials it with a pairing code),
not a hosted service.
The privacy posture carries over unchanged: no telemetry, no analytics SDK, no crash
reporter that uploads file content. On iOS that also means the local-network permission
string and a Bonjour service type in `Info.plist` — write them before the first submission,
because the denial is silent and the feature looks broken.
## Build order
Each step ships something usable on its own; none depends on a feature Shift lacks today.
0. **Extract `shift-core` on desktop.** Move the pure conversion, PDF and Office code out
from under the Tauri command layer and add the `staticlib` target. Ships nothing new to
users, and it is the step that can be finished and verified without an Apple developer
account or an Android SDK: `cargo test --lib` keeps passing the whole way, and the tests
that follow the functions prove the boundary is real.
1. **iOS share-sheet converter for images.** `shift-core` + the ported catalogue + a
ring-style picker. No networking, no capture, no accounts. Proves the gesture.
2. **Android share-sheet converter.** Same core, same catalogue; the second platform is
the test that the abstraction is real rather than iOS-shaped.
3. **PDF and Office text extraction.** Pure-Rust ports, no platform APIs, high perceived
value ("turn this deck into Markdown on my phone").
4. **Native OCR (Vision / ML Kit) + annotation.** Redact first — it is the mobile-native
killer use (screenshot → redact → share, all on device).
5. **LAN peer + phone→desktop jobs.** Add the version byte and the `kind` field to SHSF
here, in the same release as the first phone that speaks it.
6. **Capture on mobile, last.** ReplayKit/MediaProjection is real work, mobile screenshots
are already excellent, and the desktop capture loop is the least portable thing in the
product. Do it when users ask for it rather than for parity.
## Open decisions
| Question | Leaning | Why |
| --- | --- | --- |
| Tauri mobile or native shell | Native shell + Rust core | Share extensions are small native UIs; a webview adds size and latency to the one interaction that must feel instant |
| One FFmpeg binary per platform | iOS: no, Android: measure | `AVFoundation` covers most mobile media needs at zero licence risk; Android's codecs are as capable but the surface is fragmented |
| Shared OCR engine or native | Native | Vision and ML Kit beat Tesseract WASM on speed and Chinese accuracy; a shared engine buys nothing users can see |
| Catalogue home | `shift-core` | Desktop can read from it; phones and desktop cannot drift apart |
| Priced app to fund it | Undecided | The desktop app is MIT and free; a paid mobile app is defensible but changes the project's centre of gravity |