Add HTML-in-Canvas APIs - #11588
Conversation
Handwavy things that need fleshing out are marked with 👋
Kaiido
left a comment
Kaiido
left a comment
There was a problem hiding this comment.
Glad to see this being worked on, thanks.
Not quite sure how much discussion should be held at this stage. So to note, this doesn't seem to fully match the latest state of https://github.com/WICG/html-in-canvas. e.g. the rename to drawHTMLElement. The layoutsubtree attribute is also missing along with the implications to the existing fallback contents.
Still, thanks for making this move.
foolip
changed the title
|
@Kaiido thank you for the review! I've fleshed things out more, renaming to There's still some handwaving going on of course, in particular what causes the subtree to be laid out but not painted. |
foolip
changed the title
|
I've fleshed this out some more now, in particular the hit testing. |
There was a problem hiding this comment.
One common complain with the use of dictionaries in the Canvas2D API is that this makes GC kick in very often during animations which has a non-negligible performance cost.
This API shape makes a big use of such dictionaries with one for the wrapper CanvasElementHitTestRegion and then a nested one for the CanvasHitTestRect, and I guess there will be scenarios where multiple of these will need to be updated at every frame. Since the values are copied over from the passed objects to new internal objects, it's unclear if even a careful author, who would try to reuse the same objects, could avoid GC at all here.
On the other hand, I really like how this API shape enables future additions like using a Path2D, or even a bitmap mask, instead of a CanvasHitTestRect. (btw can we bikeshed on rect for that purpose?)
It's not my area of expertise, but would an actual exposed interface allow for non copy from JS, so that authors can just update the regions instead of setting new ones?
Well, the premise is faulty because you can never have an Element on a worker thread. But I think this is a moot point; what I really meant is that there is no difference between Element and ElementImage as it pertains to the issue of drawing into a different canvas from the one containing the Element. |
|
@szager-chromium I tried to contact you by email to avoid flooding this PR, but I must have found the wrong address.
First, in my last example, like in the one before, For an So in code that would give canvas.addEventListener("paint", (evt) => {
// First event firing
// Here we have a snapshot created for the element, let’s call it snapshot_1
const img = canvas.captureElementImage(element);
element.style.color = red; // trigger a new paint
canvas.addEventListener("paint", (evt) => {
// Second event firing
// Here we have another snapshot created for the element, snapshot_2
ctx.drawElementImage(img, x, y); // uses the old snapshot_1, because it’s the one held by the `ElementImage`
ctx.drawElementImage(element, x, y); // uses the new snapshot_2, available in the live canvas map of snapshots
}, { once: true });
}, { once: true });You can’t have canvas.drawElementImage(img, x, y); // uses snapshot_1
// the map on the main thread is updated, but in the worker we're sync
canvas.drawElementImage(img, x, y); // uses snapshot_2This shouldn’t happen. Now, this means the “necessary information” from the canvas also needs to be accessible for as long as the So, if that information is held even for the same-canvas policy, why can’t it be used by another canvas? Why would it work between canvasA and canvasA-after-a-resize, but not between canvasA and canvasB? |
When drawing an element from canvasA into canvasB, it is difficult to correctly update the DOM position (transform) of the element so that it matches the on-screen location, which is needed for web platform features like hit testing, accessibility, etc. |
But you face the same issues with an But if you believe it's a no-go, then there should probably be further limitation to |
This can never happen, for all kinds of reasons that have nothing to do with HTML-in-canvas. There is no concurrent access of this kind anywhere in JavaScript, with the exception of SharedArrayBuffer. But again -- and I'm having a hard time making myself understood on this point -- the "liveness" of
Among the canvas information required to determine the correct raster scale for As mentioned before, allowing an In my opinion, for all these reasons and for other reasons that I can't even think of right now, your stated reasons for wanting this behavior do not outweight the complexity and potential for confusion. |
I might be wrong, but I'm pretty sure it's still something we defend against when writing cross-context specs.
And that information is gone as soon as the snapshot held by the
You can also send an
Once again, no. This prevents doing complex compositing in a Worker while maintaining the visible canvas clean with the last generated frame for the time the processing is done. It forces you to display an half-baked frame.
My main point isn't that this should work just for the sake of it, but rather that this limitation doesn't solve anything for an |
I sense that we have perhaps reached the limit of what can be resolved with this exchange. If you feel strongly on this point, I'd suggest you file a separate issue to bring it to the attention of the standards bodies. I don't think this PR should be blocked on resolving it. |
I think I disagree on this. But, even with this limitation, the issue you identified still does exist in some other parts of the API as is shown in this fiddle (for the green text). This needs to be resolved too. We can't leave the API with a known broken path in it. From what I gather from your past comments, I am under the impression that it's mainly the magical 3 argument version of |
This is a known issue with the feature as specified: if you change the canvas grid and draw an outdated In my opinion this is not a significant shortcoming. As a point of comparison: the canvas is already implicitly Specifying |
|
(Drive by review; I haven't read the full patch or comment history) Hey folks! I was working on some paint-timing spec cleanup (context: https://github.com/wicg/soft-navigations) and @shaseley pointed out that the "mark paint timing" steps are ambiguously expecting to start pre-paint but end post-paint (i.e., include paint time). I think your PR nicely clarifies the update-the-rendering steps, but looks like it moves "mark paint time" to come after paint, and would AFAIK change the intent of the See: w3c/paint-timing#126 I think the best solution would be to split "mark paint timing" into steps that come before paint, and steps that come after. An alternative solution might be to mark the pre-paint timestamp in update-the-rendering and then pass that timestamp to "mark paint timing", similar to how we mark first, then pass |
|
@foolip , to address @mmocny 's point, we can go ahead and land this "breaking" change that introduces a paint step, and then ask @mmocny to follow up with a change to split paint timing into a pre-paint and post-paint steps, and w3c/paint-timing#126 can track this. |
|
@progers with the structure I'm proposing in #12554 there won't be a definition of "paint" in HTML. I hope that means we can leave this to w3c/paint-timing#126? |
| <span>continue</span>.</p></li> | ||
|
|
||
| <li> | ||
| <p>Let <var>snapshot</var> be a new <span |
There was a problem hiding this comment.
I don't think this works for nested canvases. By creating the snapshots in here, before firing any event, the snapshot for the outer canvas would have the content of the inner canvas before it has handled its own onpaint event.
So, if we start empty, inner canvas would receive its snapshot with its content, draw it, and then outer will gets its snapshot that was created before inner's event fired, and thus with the inner canvas still empty.
Current Canary doesn't seem to do that and exposes inner's new content from outer's snapshot (though they weirdly fire 2 events on outer): https://jsfiddle.net/39am5of2/
And it seems the way they do it is by updating outer's snapshot synchronously during inner's onpaint event: https://jsfiddle.net/waqepx1b/
| <p>For each descendant element <var>element</var> of <var>canvas</var> within the | ||
| <span>flat tree</span> that has the <code data-x="attr-drawable">drawable</code> attribute | ||
| specified, in reverse <span>tree order</span>:</p> |
There was a problem hiding this comment.
This would generate one snapshot per canvas nesting level, allowing drawing descendants of a canvas directly on its ancestor.
Current Canary doesn't allow that and throws claiming "The element can only be drawn into its nearest ancestor ".
10 participants
(See WHATWG Working Mode: Changes for more details.)
/browsers.html ( diff )
/canvas.html ( diff )
/dnd.html ( diff )
/dom.html ( diff )
/index.html ( diff )
/indices.html ( diff )
/infrastructure.html ( diff )
/interaction.html ( diff )
/popover.html ( diff )
/references.html ( diff )
/rendering.html ( diff )
/webappapis.html ( diff )