Add speculation rules prefetch

[domenic]

Other spec PRs that should be merged shortly after this one:

• CSP integration pull request: Integrate with speculation rules w3c/webappsec-csp#776
• Fetch integration pull request: Add the "speculationrules" destination fetch#1841
• WICG/nav-speculation pull request to rebase on top of this: Rebase on top of HTML WICG/nav-speculation#398

---

This ports the relevant text from https://wicg.github.io/nav-speculation/speculation-rules.html, with minor improvements and additions. It does not port the actual navigational prefetching algorithms, which are left in https://wicg.github.io/nav-speculation/prefetch.html for now.

Part of #11123.

Notable changes from that document:

• Allows treating prerender rules as prefetch rules.

• Allows the presence of target_hint (a prerender-specific speculation rule key), but ignores it.

• Follows the precedent set by import maps a bit more closely in how it integrates with the script processing model. Normatively, it fires error events on the script element for the few things that cause complete parsing failures. (Chromium recently changed to do this.)

• Adds conformance requirements for speculation rules JSON documents, following the precedent set by import maps.

• Adds explicit callouts for when the implementation may report warnings for parsing failures, following the precedent set by import maps.

• Fixes a few minor bugs in the parsing algorithms. E.g., the previous text did not correctly deal with how CSS selector parsing returned a list, or called the slightly wrong algorithm for parsing URL patterns.

• Simplified the processing model for the Speculation-Rules HTTP header. The previous text added various concepts around pending fetches and periodic implementation-defined processing of those fetches, all designed to allow the implementation to de-prioritize these fetches in favor of other things that happen during loading. This version instead explicitly allows an implementation-defined wait time and uses a separate task source which user agents can deprioritize.

• Be explicit about when to consider speculative loads, instead of using vague language about "whenever something relevant changes".

• Give an explicit algorithm for how speculative load candidates are grouped and redundant ones coalesced, instead of stating that it must be done somehow.

• Add re-parsing of <script type=speculationrules> on base URL changes, to match the Chromium implementation.

• Slightly shorten some of the security and privacy considerations, e.g., by not mentioning mixed content concerns.

• At least two implementers are interested (and none opposed):
  • Chromium
  • Gecko
  • WebKit is in support of a subset of the functionality here
• Tests are written and can be reviewed and commented upon at:
  • https://wpt.fyi/results/speculation-rules (includes updates to error behavior mentioned above)
• Implementation bugs are filed:
  • Chromium: N/A
  • Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1969396
  • WebKit: https://bugs.webkit.org/show_bug.cgi?id=295193
  • Deno (only for timers, structured clone, base64 utils, channel messaging, module resolution, web workers, and web storage): N/A
  • Node.js (only for timers, structured clone, base64 utils, channel messaging, and module resolution): N/A
• Corresponding HTML AAM & ARIA in HTML issues & PRs: N/A
• MDN issue is filed: N/A, MDN already documents this well
• The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

---

/acknowledgements.html ( diff )
/browsers.html ( diff )
/browsing-the-web.html ( diff )
/document-lifecycle.html ( diff )
/document-sequences.html ( diff )
/iana.html ( diff )
/index.html ( diff )
/indices.html ( diff )
/infrastructure.html ( diff )
/interaction.html ( diff )
/links.html ( diff )
/nav-history-apis.html ( diff )
/references.html ( diff )
/scripting.html ( diff )
/semantics.html ( diff )
/urls-and-fetching.html ( diff )
/webappapis.html ( diff )
/speculative-loading.html ( diff )

[domenic]

The spec text here is done and ready for review!

Marking "do not merge yet" until I get the other prerequisites listed in the OP done (mostly PRs against other specs).

[domenic]

This is now fully ready for review, including all prerequisites. I'll keep "do not merge yet" to reflect that we need to get #11442 reviewed and merged first, but that's a pretty small change...

[annevk]

We generally don't support new headers through http-equiv because we don't want mutable policies. That seems worth documenting, but it doesn't seem specific to this feature.

[domenic]

I've added #11494 to document that in the overall definition of http-equiv.

[hsivonen]

I've added #11494 to document that in the overall definition of http-equiv.

That works. Thanks.

[domfarolino]

First pass, although I'm certainly less than half-way through. Mostly small things and questions so far.

[domfarolino]

Can this result in any prefetch request being canceled other than the one associated with the triggering of this algorithm to run in the first place? I'm asking because I know spec-wise we simplify things by "considering all speculative loads" in a state-less way whenever any individual link-specific thing happened, causing this algorithm to run. And I'm just trying to make sure it's not possible for a <a>'s href attribute to be mutated, and for this algorithm to run and cancel an unrelated speculative load (because we evaluate all the thingz), in a way that a smarter implementation would never do.

[domenic]

In spirit: no. But literally: yes.

This is because of the microtask queuing behavior. If you do

mutateLink1();
mutateLink2();
mutateLink3();

then mutateLink1() queues a microtask to "consider speculative loads", mutateLink2() and mutateLink3() do nothing, and by the time the microtask runs, "consider speculative loads" will cancel prefetches from all 3 links.

In the Chromium implementation, what this looks like is all the various steps that call "consider speculative loads" add to a list of dirty candidates, and so this step looks at the dirty list.

Do you think explaining this implementation strategy would be worth a note?

[domfarolino]

OK so it is possible for this to cancel multiple things beyond what the thing invoking this algorithm would do, but not so much "in a way that a smarter implementation would never do" since implementations achieve the same outcome just with a different, more stateful mechanism, if I'm correct. So the observable results shouldn't differ. I think that's worth a pretty brief note, yeah!

[domenic]

I expanded #note-consider-speculative-loads-stateless with more details; let me know what you think.

[domfarolino]

I'm confused why this is here? It's just a breadcrumb for when we eventually upstream prerendering? Are we likely to forget to do this, or inversely, are we like to remember to scan through all PRERENDERING: instances to ensure we completed the full upstreaming?

[domenic]

The intent is for it to be a breadcrumb for the future. I'm not sure we'll remember to scan through, but it shouldn't hurt?

[zcorpan]

I think this means that "requires": ["anonymous-client-ip-when-cross-origin"] in prerender will be honored, but would be ignored when prerender is supported as prerender. Is that intended?

[domenic]

Very interesting observation. This prerender -> prefetch conversation was not thought through in great detail, so the extra scrutiny is helpful.

The general intention for requirements is that any requirements expressed, which cannot be met, cause the speculative load to fail. So, instead of being ignored, it would cause the prerender to not happen. Which is a good way to fail.

The mechanism by which this happens needs to be in the https://wicg.github.io/nav-speculation/speculation-rules.html#speculation-rules-parsing . Similar to how we discard prefetch rules that include a target navigable name hint (since it indicates authoring confusion), we need to discard prerender rules that include a requirement on ACIWCO. I'll update that spec shortly.

[domfarolino]

Is there a difference between being parsed as a URL, and being parsed as a "URL pattern constructor string", and what is the latter? Is it basically just a scalar value string that doesn't make https://urlpattern.spec.whatwg.org/#process-a-urlpatterninit throw an error?

[domenic]

URLPatternInit is a dictionary with each component specified individually. In contrast, the string version in this bullet is one of the shorthands discussed in https://urlpattern.spec.whatwg.org/#introduction , e.g. "http{s}?://{:subdomain.}?shop.example/products/:id([0-9]+)#reviews". Note that this is not parseable as a URL.

Any thoughts on if this should be clearer somehow?

[domfarolino]

This is interesting to me. Do we use the same range of code points elsewhere in standards, for the same reason? Or is there no precedent for that that you can think of.

[domenic]

I can't find any other instances in HTML or Fetch. I guess it's the inverse of URL's C0 control percent-encode set?

Moving information from HTML/JavaScript/etc. into HTTP headers seems a bit novel, but I would be surprised if this is the very first time we're doing it... It might be the first time we're doing it while writing strict conformance requirements?

[domfarolino]

This is interesting... I guess because this can throw an exception, this technically introduces another way to run script synchronously during node removal, is that right? I'm imagining a scenario where some child node is removed from a speculation rules script that has already been "prepared". Synchronously during the removal of that child node, we'd find ourselves here. The post-connection steps do nothing, but if "update the speculation rules parse result" produces an exception (perhaps as a result of the removal), then #register-speculation-rules will immediately report that exception, thus invoking script.

I honestly don't know if this is a big deal—I just know that we don't places where we might run script synchronously during upon removal.

[domenic]

I think you're right, and this is somewhat unintended. I'm unsure if we should do anything about it... we could always queue a task for the error reporting, I guess? I think that would be inconsistent with other script errors, but maybe it's not so bad...

[domfarolino]

Should we tighten this condition to specifically require valid URL strings, to be consistent with the authoring requirements?

[domenic]

Values which are not parseable as URLs (like "https:notaport") will fail in a couple of steps, once we iterate over each item and parse it.

Values which are parseable but not valid (like "https:////////example.com/") should be conformance errors but not cause failure.

There's a separate issue you're pointing at here where we're inconsistent, in that we treat non-strings as fatal errors for the whole rule and non-parseable URLs as just something to be skipped over. This does match the current implementation. I think it's kind of OK, since that level of type confusion indicates a pretty bad authoring error, whereas maybe a non-parseable URL is something that could happen more easily? But I'm not sure.

Also, the current implementation mismatches this draft spec in that it checks for string-ness as part of the loop, instead of earlier. I guess I should fix that to match the implementation more closely, although it's only observable via console warning ordering.

[domfarolino]

Like the above instance, I'm not sure if this:

• Is OK as-is (which is confusing to me)
• Should be "Return null"
• Should be removed entirely

[domenic]

Let me know if this is still confusing after the above comment and the subsequent slight restructuring.