Frame.io API

Frame.io API Documentation

Welcome to Frame.io's official documentation portal.In here, you'll find everything you need to:

Get up and running with Frame.io's public API 🤙
Leverage powerful features like Webhooks and Custom Actions 🚀
Build your own OAuth2.0 app 🚢

Get Started    Guides

Work with Annotations

Annotations describe any drawing data supplied with a Comment on Frame.io. This guide describes how to interpret annotation data relative to what a user sees on the screen.

Overview

Annotations in Frame.io are represented as a serialized JSON blob of drawing objects, complete with colors, coordinates, and sizes. In the blob, all quotation marks apart from the outer bounding pair are escaped with forward slashes (e.g. \"key\":\"value\").

There are three basic shapes -- arrow, rect, and line -- as well as free-hand drawing, which is outside the scope of this guide, but follows the same basic principles of serialized coordinate data.

The coordinate plane

All shapes are placed on a coordinate plane with an upper-left origin, and a scale of 0 to 1 on each axis.

The basic coordinate plane for annotations.

Basic shape data

All shapes are placed on the above plane with a combination of coordinates and style data. Each shape has some common attributes, and then each has a slightly different way of addressing the coordinate plane.

All attributes for distance/width/height use the same 0 to 1 scale as the main coordinate plane itself, but are generally marked from the shape's point of origin.

In other words, if a video is being displayed in a 640x480 viewport, x=1 will equal an absolute 640px in that context. This enables annotations to scale seamlessly while maintaining their relative position as viewing screen, size, and context shift.

Note: the origin of a rect is always the upper-left corner of the shape, regardless of drawing origin.

Common attributes

AttributeDescription
toolarrow, rect, or line. States which shape is being made.
colorSix-digit hex color for the shape.
sizePixel size for the shape's line. For arrows, this refers only to the start of the line, as end width is determined programmatically.

Arrow

AttributeDescription
xx-coordinate of the arrow's origin.
yy-coordinate of the arrow's origin.
wDifference in arrow's y-axis position from start to end, relative to the arrow's origin.
hDifference in arrow's x-axis position from start to end, relative to the arrow's origin.

Rect

AttributeDescription
xx-coordinate of the upper-left corner of the rectangle.
yy-coordinate of the upper-left corner of the rectangle.
wWidth of the rectangle, relative to upper-left rectangle origin.
hHeight of the rectangle, relative to upper-left rectangle origin.
ixThe drawing x-origin of the rectangle. (Not required).
iyThe drawing y-origin of the rectangle. (Not required).
radiusRadius of the rectangle's corner roundedness. Must be positive. Values above 150 are visually indistinguishable.

Line

AttributeDescription
x1x-coordinate of the line's origin.
y1y-coordinate of the line's origin.
x2x-coordinate of the line's terminus. Absolute value on the image plane.
y2y-coordinate of the line's terminus. Absolute value on the image plane.

Simple example: the coordinate plane

"annotation": "[{\"tool\":\"line\",\"color\":\"#E74A3C\",\"size\":3,\"x1\":0.0028530670470756064,\"y1\":0.012680297987002695,\"x2\":0.9964336661911555,\"y2\":0.9903312727849104},{\"tool\":\"line\",\"color\":\"#34A3DB\",\"size\":3,\"x1\":0.004992867332382311,\"y1\":0.98779521318751,\"x2\":0.9942938659058488,\"y2\":0.012680297987002695}]"
"annotation" : [
  {
    "tool":"line",
    "color":"#E74A3C",
    "size":3,
    "x1":0.0028530670470756064,
    "y1":0.012680297987002695,
    "x2":0.9964336661911555,
    "y2":0.9903312727849104
  },
  {
    "tool":"line",
    "color":"#34A3DB",
    "size":3,
    "x1":0.004992867332382311,
    "y1":0.98779521318751,
    "x2":0.9942938659058488,
    "y2":0.012680297987002695
  }
]

Breakdown

SerializedDeserialized
"tool":"line", "color":"#34A3DB", "size":3, "x1":0.004992867332382311, "y1":0.98779521318751, "x2":0.9942938659058488, "y2":0.012680297987002695json { "tool":"line", "color":"#34A3DB", "size":3, "x1":0.004992867332382311, "y1":0.98779521318751, "x2":0.9942938659058488, "y2":0.012680297987002695 }
SerializedDeserialized
"tool":"line", "color":"#E74A3C", "size":3, "x1":0.0028530670470756064, "y1":0.012680297987002695, "x2":0.9964336661911555, "y2":0.9903312727849104json { "tool":"line", "color":"#E74A3C", "size":3, "x1":0.0028530670470756064, "y1":0.012680297987002695, "x2":0.9964336661911555, "y2":0.9903312727849104 }

Complex example: lots of boxes

"annotation": "[{\"tool\":\"rect\",\"color\":\"#34A3DB\",\"size\":3,\"x\":0.008426966292134831,\"y\":0.02663337494798169,\"w\":0.0702247191011236,\"h\":0.10986267166042447,\"ix\":0.008426966292134831,\"iy\":0.02663337494798169,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#1ABCA1\",\"size\":3,\"x\":0.8932584269662921,\"y\":0.0449438202247191,\"w\":0.08895131086142327,\"h\":0.35122763212650854,\"ix\":0.8932584269662921,\"iy\":0.0449438202247191,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E67422\",\"size\":3,\"x\":0.023408239700374533,\"y\":0.7873491468997087,\"w\":0.1254681647940075,\"h\":0.2047440699126093,\"ix\":0.023408239700374533,\"iy\":0.7873491468997087,\"radius\":8},{\"tool\":\"rect\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.6919475655430711,\"y\":0.5409904286308781,\"w\":0.29868913857677903,\"h\":0.4427798585101955,\"ix\":0.6919475655430711,\"iy\":0.5409904286308781,\"radius\":8},{\"tool\":\"arrow\",\"color\":\"#E74A3C\",\"size\":3,\"x\":0.25374531835205993,\"y\":0.23137744486059092,\"w\":0.24250936329588013,\"h\":0}]"
"annotation" : [
  {
    "tool":"rect",
    "color":"#34A3DB",
    "size":3,
    "x":0.008426966292134831,
    "y":0.02663337494798169,
    "w":0.0702247191011236,
    "h":0.10986267166042447,
    "ix":0.008426966292134831,
    "iy":0.02663337494798169,
    "radius":8
  },
  {
    "tool":"rect",
    "color":"#1ABCA1",
    "size":3,
    "x":0.8932584269662921,
    "y":0.0449438202247191,
    "w":0.08895131086142327,
    "h":0.35122763212650854,
    "ix":0.8932584269662921,
    "iy":0.0449438202247191,
    "radius":8
  },
  {
    "tool":"rect",
    "color":"#E67422",
    "size":3,
    "x":0.023408239700374533,
    "y":0.7873491468997087,
    "w":0.1254681647940075,
    "h":0.2047440699126093,
    "ix":0.023408239700374533,
    "iy":0.7873491468997087,
    "radius":8
  },
  {
    "tool":"rect",
    "color":"#E74A3C",
    "size":3,
    "x":0.6919475655430711,
    "y":0.5409904286308781,
    "w":0.29868913857677903,
    "h":0.4427798585101955,
    "ix":0.6919475655430711,
    "iy":0.5409904286308781,
    "radius":8
  },
  {
    "tool":"arrow",
    "color":"#E74A3C",
    "size":3,
    "x":0.25374531835205993,
    "y":0.23137744486059092,
    "w":0.24250936329588013,
    "h":0
  }
]

Breakdown

With all that in mind, it's helpful to lay out each element in the above image to develop a clearer picture of the annotation coordinate system. Note that the size and radius for each of the rectangles in this example are taking the Frame.io UI defaults.

Blue rect

SerializedDeserialized
"tool":"rect", "color":"#34A3DB", "size":3, "x":0.008426966292134831, "y":0.02663337494798169, "w":0.0702247191011236, "h":0.10986267166042447, "ix":0.008426966292134831, "iy":0.02663337494798169, "radius":8json { "tool":"rect", "color":"#34A3DB", "size":3, "x":0.008426966292134831, "y":0.02663337494798169, "w":0.0702247191011236, "h":0.10986267166042447, "ix":0.008426966292134831, "iy":0.02663337494798169, "radius":8 }

Green rect

SerializedDeserialized
"tool":"rect", "color":"#1ABCA1", "size":3, "x":0.8932584269662921, "y":0.0449438202247191, "w":0.08895131086142327, "h":0.35122763212650854, "ix":0.8932584269662921, "iy":0.0449438202247191, "radius":8json { "tool":"rect", "color":"#1ABCA1", "size":3, "x":0.8932584269662921, "y":0.0449438202247191, "w":0.08895131086142327, "h":0.35122763212650854, "ix":0.8932584269662921, "iy":0.0449438202247191, "radius":8 }

Red rect

SerializedDeserialized
"tool":"rect", "color":"#E74A3C", "size":3, "x":0.6919475655430711, "y":0.5409904286308781, "w":0.29868913857677903, "h":0.4427798585101955, "ix":0.6919475655430711, "iy":0.5409904286308781, "radius":8json { "tool":"rect", "color":"#E74A3C", "size":3, "x":0.6919475655430711, "y":0.5409904286308781, "w":0.29868913857677903, "h":0.4427798585101955, "ix":0.6919475655430711, "iy":0.5409904286308781, "radius":8 }

Orange rect

SerializedDeserialized
"tool":"rect", "color":"#E67422", "size":3, "x":0.023408239700374533, "y":0.7873491468997087, "w":0.1254681647940075, "h":0.2047440699126093, "ix":0.023408239700374533, "iy":0.7873491468997087, "radius":8json { "tool":"rect", "color":"#E67422", "size":3, "x":0.023408239700374533, "y":0.7873491468997087, "w":0.1254681647940075, "h":0.2047440699126093, "ix":0.023408239700374533, "iy":0.7873491468997087, "radius":8 }

Red arrow

SerializedDeserialized
"tool":"arrow", "color":"#E74A3C", "size":3, "x":0.25374531835205993, "y":0.23137744486059092, "w":0.24250936329588013, "h":0json { "tool":"arrow", "color":"#E74A3C", "size":3, "x":0.25374531835205993, "y":0.23137744486059092, "w":0.24250936329588013, "h":0 }

Updated about a year ago

Work with Annotations


Annotations describe any drawing data supplied with a Comment on Frame.io. This guide describes how to interpret annotation data relative to what a user sees on the screen.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.