OCR Coordinate Workaround for Missing API

Core Challenge

On macOS, there is no direct way to trigger a „Share Menu Item“ via AppleScript. Neither AppleScript nor Automator provide a native command to, for example, share a website from Safari directly to the Notes app. The provided script solves this problem with a creative workaround: It locates the desired menu item („Notes“) in Safari’s Share menu by taking a screenshot and using OCR (Optical Character Recognition) to determine the position of the menu entry. The script then simulates a mouse click at the detected position to open the sharing dialog and send the website to the Notes app.

How the Script Works

• Opens Safari and loads a sample website.
• Opens the Share menu and takes a screenshot of the Safari window.
• A Swift script using Apple’s Vision framework runs OCR on the screenshot to find the position of the desired menu item („Notes“).
• The script simulates a mouse click at the recognized position to open the Share dialog.
• Optionally, the script can automate further interactions with the Share dialog (e.g., entering a title, closing the dialog).

Why this workaround?

AppleScript can only address menu items directly if they are classic menus. The Share menu, however, is dynamic and system-integrated; there is no official AppleScript command to trigger a specific share target (like „Notes“). Therefore, the script takes a screenshot and uses OCR to find the menu item’s position—a workaround that can be adapted for other share targets as well.

Key Components

• Vision Framework OCR: Text recognition in menu screenshots
• Coordinate Mapping: swift
  let x = Int(bbox.origin.x * imageSize.width / screenScale)
let y = Int((1 - bbox.origin.y) * imageSize.height / screenScale)
• UI Automation: Synthetic mouse clicks and keyboard events

Technical Implementation

shareToNotes()
  -- Activate Safari & open Share menu
  tell application "Safari" to activate
  tell application "System Events" to click menu item "Share"
  
  -- OCR coordinate calculation
  set {x, y} to getCoordinates()
  
  -- Simulate UI interactions
  click at {x, y}
  delay 0.5
  keystroke "Note Title" & return
end shareToNotes

Download Full Script

Requirements

• macOS with Swift and the Vision framework installed (default since macOS 10.15).
• Accessibility permissions for AppleScript and System Events must be enabled.

Download

The complete AppleScript including the Swift component will be available here.

Additional Notes:

• Script variables can be adjusted for different languages and menu entries.
• Menu names may need to be changed for different system languages or Safari versions.
• Script speed can be tuned via the delay values.

Note
Since the macOS Share menu cannot be accessed directly via AppleScript, this script uses a screenshot and OCR-based approach to locate and execute the desired menu item. This workaround is universally customizable and can be used for different applications and share targets.