
OCR Coordinate Workaround for Missing API
Core Challenge
On macOS, there is no direct way to trigger a „Share Menu Item“ via AppleScript. Neither AppleScript nor Automator provide a native command to, for example, share a website from Safari directly to the Notes app. The provided script solves this problem with a creative workaround: It locates the desired menu item („Notes“) in Safari’s Share menu by taking a screenshot and using OCR (Optical Character Recognition) to determine the position of the menu entry. The script then simulates a mouse click at the detected position to open the sharing dialog and send the website to the Notes app.
How the Script Works
- Opens Safari and loads a sample website.
- Opens the Share menu and takes a screenshot of the Safari window.
- A Swift script using Apple’s Vision framework runs OCR on the screenshot to find the position of the desired menu item („Notes“).
- The script simulates a mouse click at the recognized position to open the Share dialog.
- Optionally, the script can automate further interactions with the Share dialog (e.g., entering a title, closing the dialog).
Why this workaround?
AppleScript can only address menu items directly if they are classic menus. The Share menu, however, is dynamic and system-integrated; there is no official AppleScript command to trigger a specific share target (like „Notes“). Therefore, the script takes a screenshot and uses OCR to find the menu item’s position—a workaround that can be adapted for other share targets as well.
Key Components
- Vision Framework OCR: Text recognition in menu screenshots
- Coordinate Mapping: swift
let x = Int(bbox.origin.x * imageSize.width / screenScale)
let y = Int((1 - bbox.origin.y) * imageSize.height / screenScale)
- UI Automation: Synthetic mouse clicks and keyboard events
Technical Implementation
shareToNotes()
-- Activate Safari & open Share menu
tell application "Safari" to activate
tell application "System Events" to click menu item "Share"
-- OCR coordinate calculation
set {x, y} to getCoordinates()
-- Simulate UI interactions
click at {x, y}
delay 0.5
keystroke "Note Title" & return
end shareToNotes
Requirements
- macOS with Swift and the Vision framework installed (default since macOS 10.15).
- Accessibility permissions for AppleScript and System Events must be enabled.
Download
The complete AppleScript including the Swift component will be available here.
Additional Notes:
- Script variables can be adjusted for different languages and menu entries.
- Menu names may need to be changed for different system languages or Safari versions.
- Script speed can be tuned via the
delay
values.
Note
Since the macOS Share menu cannot be accessed directly via AppleScript, this script uses a screenshot and OCR-based approach to locate and execute the desired menu item. This workaround is universally customizable and can be used for different applications and share targets.
Schreibe einen Kommentar