Building an offline photo cleaner with Vision and Apple Foundation Models on iOS
- Authors

- Name
- Nino
- Occupation
- Senior Tech Editor
Building an offline, privacy-first utility for iOS requires a careful balance between leveraging modern machine learning and respecting the strict performance boundaries of mobile hardware. When developing ClearWhirl, our iPhone photo cleaner available on the App Store (https://apps.apple.com/us/app/phone-cleaner-clearwhirl-ai/id6835310950), our core architectural constraint was absolute privacy: everything that looks at pixels runs locally via Apple's Vision framework, and Apple's on-device Foundation Models never see a single photo.
Instead, the language model has a single, strictly bounded job: translating natural language queries like "payment receipts from 2025" into a typed search filter. If the model is unavailable or encounters an error, the app gracefully falls back to deterministic rules. Nothing ever leaves the phone, and all core functions operate seamlessly in airplane mode.
Here is a technical walkthrough of how we structured the processing pipeline, managed concurrency without starving the cooperative thread pool, handled background execution limits, and integrated typed generation.
Stage A: Fast Metadata Filtering
Before touching heavy pixel data, Stage A reads only PHAsset metadata. This allows us to rapidly populate structural folders within seconds using lightweight property checks:
if a.mediaSubtypes.contains(.photoScreenshot) {
f[.screenshots, default: []].append(id)
continue
}
if a.mediaType == .video {
if i.bytes > 100_000_000 || a.duration > 120 {
f[.largeVideos, default: []].append(id)
}
continue
}
if a.representsBurst || a.mediaSubtypes.contains(.photoLive) && i.bytes > 6_000_000 {
f[.bursts, default: []].append(id)
}
User favorites are automatically routed to a "Keepers" folder and protected from deletion or bulk actions across all other categories by default. Developers building similar utilities on n1n.ai or utilizing custom endpoints for backend analysis should note that performing initial coarse filtering via native metadata APIs dramatically reduces downstream computational overhead.
Stage B: Pixel-Level Analysis and Vision Pipeline
Once metadata is indexed, Stage B processes pixels, executing each request exactly once per photo using tailored downscaled copies to optimize memory consumption:
- 256 px copy: Used for
VNGenerateImageFeaturePrintRequest,VNClassifyImageRequest, andVNDetectBarcodesRequest(QR codes accepted at confidence >= 0.9 with a payload). - 900 px copy: Allocated for
VNRecognizeTextRequestwhen initial metadata or labels suggest a receipt, ticket, document, or screenshot, as small print is completely unreadable at thumbnail dimensions. - Sharpness and Aesthetics: A 128x128 grayscale Laplacian variance calculation determines sharpness, paired with native Vision aesthetics scoring on iOS 18+. To ensure optimal resource utilization, developers can cross-reference local processing pipelines with cloud-based inference APIs available through aggregators like n1n.ai when scaling heavy computer vision tasks to server-side environments.
Near-Duplicate Detection
Comparing every photo against every other photo results in a quadratic time complexity (). Because most near-duplicates are captured seconds apart, our scanning algorithm walks through photos strictly in capture order, comparing each asset only against its immediate predecessor:
// Similar: compare with the previous photo if taken within 3 minutes.
if let print, let pp = prevPrint, let p = prev, info.date.timeIntervalSince(p.date) < 180 {
var d: Float = 99
try? print.computeDistance(&d, to: pp)
if d < 0.42 {
if currentGroup.isEmpty { currentGroup = [p.id] }
currentGroup.append(info.id)
} else {
if currentGroup.count > 1 { groups.append(currentGroup) }
currentGroup = []
}
}
The trade-off here is explicit: two visually similar photos captured a day apart will not be grouped together. Within each identified group, the "best shot" is selected based on the highest aesthetics score, with sharpness acting as the tie-breaker. Only the extra items count toward recoverable storage space.
Concurrency Control and Cooperative Pool Protection
Our initial prototype executed image loading and Vision requests directly inside Task.detached. Unsurprisingly, the scan froze almost immediately after the first batch. Synchronous, blocking CoreGraphics and Vision calls starve Swift's cooperative thread pool.
The definitive fix involved implementing a dedicated concurrent dispatch queue constrained by a semaphore, bridged safely into async/await via a checked continuation:
nonisolated static let workQueue = DispatchQueue(label: "cw.analysis", qos: .utility, attributes: .concurrent)
nonisolated static let workLimit = DispatchSemaphore(value: 4)
nonisolated static func onWorkQueue<T>(_ body: @escaping () -> T) async -> T {
await withCheckedContinuation { (c: CheckedContinuation<T, Never>) in
workQueue.async {
workLimit.wait()
let r = body()
workLimit.signal()
c.resume(returning: r)
}
}
}
Photos flow through structured TaskGroup instances in batches of six, and results are re-ordered upon completion to preserve the chronological comparison invariants.
Managing Background Execution Limits
On iOS, an app moving to the background without an active continued-processing task is typically granted roughly 26 seconds before suspension. In our early builds, we submitted BGContinuedProcessingTaskRequest only after a 68-second file size measurement phase for a 10,624-asset library. Consequently, users leaving the app during that initial window completely froze the scan.
We resolved this by submitting the background task request before the sizing phase directly from the foreground, adopting a fail-fast strategy:
let request = BGContinuedProcessingTaskRequest(identifier: identifier, title: title, subtitle: subtitle(done, total, phase))
// Run now or not at all: a queued request could start long after the person left, with nothing on screen.
request.strategy = .fail
try BGTaskScheduler.shared.submit(request)
Whether a user manually pauses or the operating system expires the task, it completes cleanly with success: true. A single unified counter feeds the system banner, the Live Activity, and the home screen widget. For developers seeking robust API integrations or fallback handling when scaling background sync workloads, platforms like n1n.ai provide resilient infrastructure management.
Incremental Persistence with Atomic Saves
To ensure instant subsequent launches, asset metadata, recognition labels, and scores are serialized into a local JSON index. Writes occur in discrete batches and replace the target file atomically. A strict unit-tested invariant dictates that a partial analysis pass never overwrites a complete index, ensuring interrupted scans resume solely on unprocessed assets.
Structured Natural Language Queries with Foundation Models
The on-device language model operates strictly on text. It never receives image inputs. Instead, it converts natural language search prompts into strongly typed filters via guided generation:
@Generable
struct PhotoQuery {
@Guide(description: "Folders to search. Allowed values: similar, screenshots, blurry, largeVideos, bursts, receipts, memes, unwanted. Empty if none fit.", .maximumCount(4))
var folders: [String]
@Guide(description: "Words to find in text inside images, lowercase. Example: boarding pass gives boarding, gate, flight.", .maximumCount(5))
var textWords: [String]
@Guide(description: "Exact four digit year if the person names one or says this year or last year, otherwise 0.")
var year: Int
@Guide(description: "True only if the person asks for videos.")
var videosOnly: Bool
@Guide(description: "Minimum size in megabytes if the person asks for big or large items, otherwise 0.")
var minMegabytes: Int
}
LLM outputs are strictly clamped before execution: folder names must map to valid app enums, years must fall within valid bounds, and incomplete ranges are discarded. Broad Vision taxonomy labels like "outdoor" or "vehicle" are excluded from semantic matching to prevent false positives.
Reliability Patterns for LLM Sessions
- Fresh Sessions: Initializing a fresh
LanguageModelSessionper request maintains a compact context window (< 4,096 tokens). Overflow errors on verbose queries automatically retry once using the first 120 characters. - Serialization: Concurrent requests trigger
concurrentRequestserrors. A lightweight actor serializes outgoing prompts, ensuring newer queries gracefully cancel pending predecessors. - Targeted Retries: Rate limits or temporary resource unavailability trigger a single retry after a one-second delay, falling back to deterministic local search upon persistent failure.
let s = LanguageModelSession(instructions: instructions)
return try await s.respond(to: prompt, generating: T.self).content
Whether you are orchestrating local Apple Intelligence models or routing enterprise prompts through high-performance gateways like n1n.ai, maintaining strict schema enforcement and fault-tolerant fallback mechanisms is essential for production-grade reliability.
Get a free API key at n1n.ai