Add support for OSC 8, Hyperlink, to the console infrastructure

[stuaxo]

There is a new ansi code for hyperlinks in the console.
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda

It would be great if this was possible in the windows console.

[arcanis]

If it helps prioritising this feature, I'd like to use it to implement better error messages in Yarn: the package names would be clickable and point to the package websites.

[lypanov]

Could MSFT please comment on if this is possible for us to work on? As in, is this something that can be done only from the OSS side or will it also require changes to ConPTY in Windows itself? I find the architecture quite difficult to grok.

[zadjii-msft]

This is definitely something the community could work on - ConPTY is OSS too (in this repo 😉). I'd say it's probably not something a community member would want to just pick up - it would take a lot of changes. We'd need to change the TextBuffer to recognize where URLs are in the buffer. We'd need to probably change the Renderer and IRenderData interfaces to support rendering where a hyperlink is. Then, there'd probably need to be changes in the InteractivityWin32 project to support clicking the links. And this is all just for conhost.exe. To add it to the Terminal, you'd probably need to change TermControl instead of InteractivityWin32.

After I typed this up, it actually doesn't sound as bad as I thought initially.

Fortunately, I don't think there's any conpty changes necessary here 😄

DISREGARD THAT. I thought I was commenting in #574. This issue's a little different.

For conhost to support this VT sequence:

1. update OutputStateMachineEngine to support parsing this sequence
2. Add support to TextBuffer to know when a region of text is a hyperlink, and what the URL is for that region.
3. Add functions to AdaptDispatch, ConGetSet, OutputStream, getset.cpp, etc. to add/remove hyperlinks from the buffer
4. Add code in Renderer, RenderData to be able to identify when a region of text should be rendered as a hyperlink.
5. Modify GdiEngine to be able to draw hyperlinks (and possibly stub out other renderers as well. The VtEngine should probably just emit the text)
6. Modify InteractivityWin32 to check if a click is on a hyperlink in the buffer, and open the browser (or ShellExecute) when it is

Then for the Terminal, after 4 (but not needing 3/5/6):
7. Modify OutputStateMachineEngine to not only handle hyperlink VT sequences, but also pass them through to the terminal
8. Add support to TerminalDispatch to add/remove hyperlinks from the buffer
9. Modify DxEngine to be able to draw hyperlinks
10. Modify TermControl to check if a click is on a hyperlink in the buffer, and open the browser (or ShellExecute) when it is

[lypanov]

@zadjii-msft My current specific interest could reduce this list even further. I'd like to add support for the OSC hyperlink extension to the wezterm (https://github.com/wez/wezterm) Windows port.

Reading over the changes from https://github.com/microsoft/terminal/pull/891/commits to add support for OSC 10/11 sounds like it has direct parallels in the changes I'll need for OSC 8.

I understand from the maintainer of wezterm that the VT sequences themselves are not directly passed on to wezterm. Is this due to the lack of "parse and re-emit" support in this repository? Am I correct in saying that OutputStateMachineEngine would be the correct place to add such support to? Is this source file part of "conhost"? I've read the sentence "pass them through to the terminal" here from you and also several times in the docs but don't understand exactly how this can relate to 3rd party users of ConPTY.

Final question... how can wezterm know about my changes to conhost? Do I have to somehow embed the open source fork? (My assumption is that wezterm right now is using a conhost provided by Windows 10).

Please correct any horrible assumptions / failures in understanding I might have. Thank you so much!

[egmontkob]

Experiences from implementing it in gnome-terminal:

It was freaking easy to implement a demo, which was feature complete and everything, except the "tiny" issue that each character cell contained the entire target URL (in addition to the Unicode character and the SGR properties such as colors, boldness etc.), so the memory consumption went up from ~12 bytes per charcell to ~2k per charcell. Needless to say, this was a proof of concept, not meant to ever hit the codebase.

And then it was freaking hard to come up with a solution that consumes reasonable amount of memory, assuming reasonable use of the feature. The details probably don't really matter because gnome-terminal stores the onscreen data and the scrollback buffer in totally different formats, so whatever we did there probably wouldn't apply to WT. But basically for the onscreen bits we have 2 bytes per cell storing an index to a per-terminal pool of hyperlinks (with garbage collecting and stuff), and for the scrollback we store the URLs in runlength-encoding (i.e. once per continuous run of the same URL) in the corresponding disk file (just as we do the SGR attributes). Converting back and forth between the two, handling if the user scrolls back across the boundary etc. was a true PITA. I hope WT's architecture allows for something cleaner for you guys, e.g. 4 bytes per cell indexing to the pool which is common for the writable part as well as the scrollback (still need to figure out refcounting and/or GC, etc.).

Would be lovely to hear how much trouble it was to other terminals that implemented it :)

[lypanov]

I've been wondering about these things myself @egmontkob. The implementation I'm looking at is written in rust which I can imagine simplifies the GCing requirements you had. Based on a very quick perusal it looks like Hyperlinks are not instantiated per cell but shared and referenced via pointers. Not sure it uses a pool / arena as, well, I have no Rust skills.

[egmontkob]

To complete the picture, there are two more reasons which made it quite cumbersome in vte (gnome-terminal).

One is that we're paranoid about long-time memory fragmentation, so we try to avoid malloc/free (new/delete, etc.) as much as possible. E.g. even in the hyperlink pool, allocated memory for each hyperlink is grown on demand but isn't shrunk or freed until the terminal is closed.

The other one is: due to the way our scrollback is stored on disk, we don't really know what data (incl. hyperlinks) scrolls out for good at the top of the scrollback. I mean, knowing it would require an extra read which we don't want to do for performance reasons.

If a terminal doesn't suffer from these constraints, some simple refcounted pointer (e.g. std::shared_ptr) could be a good solution. One reference by the parser as long as it's the active URL, and one per character cell that got populated with it.

[zadjii-msft]

Okay yea lets do this.

Right now, all VT output goes through OutputStateMachine, who tries to parse it into actions to perform on the console buffer. When we do find a sequence we understand, we call methods on AdaptDispatch, which handles actually doing something to conhost. Since you're really only worried about adding support in another terminal, you won't need to worry about that too much.

When the parser encounters a string it doesn't understand, it's supposed to flush the string through to the connected terminal, if there is one. So, since conhost doesn't yet understand this sequence, and it's an OSC sequence, it should be hitting this block of code:

terminal/src/terminal/parser/OutputStateMachineEngine.cpp

Lines 752 to 757 in cc9d2ca

	// If we were unable to process the string, and there's a TTY attached to us,
	// trigger the state machine to flush the string to the terminal.
	if (_pfnFlushToTerminal != nullptr && !success)
	{
	success = _pfnFlushToTerminal();
	}

That's the code that should tell the state machine to "pass the string through" to the terminal, whatever application that might be. (You might be hitting #2011/#2665 while debugging this, but any subsequent printable character should cause the frame to flush, so I doubt you're hitting this too badly.)

I'd start debugging there.

It's definitely possible to use the OSS-built conhost to host conptys instead of the OS one - that's what the Windows Terminal does today. However, I don't think we've really had time to polish that story for external consumers right now. We're currently just building a .lib & .dll for the conpty APIs, and making sure to link that first, and that's a little janky. @DHowett-MSFT can go into greater details on that topic.

---

@egmontkob thanks for the great tips. That's pretty much the way I envisioned it - using a single ID in a `TextAttributeRun` to host the URL. We don't really have any difference between scrollback and viewport in the TextBuffer, so we don't need to worry about that too much. @miniksa is also interested in this, so I'm cc'ing him.

[miniksa]

This sounds reasonable. I think we'd also want to consider some sort of heuristic-based detection of hyperlinks (in addition to being able to surface ones declared with this VT code) to meet our goal of the hyperlink feature we dreamed of for our announce video.

I also agree that the concept of having a table of links and hanging some sort of attribute off the relevant area of the buffer would be sufficient.

I don't know if we'd do this in InteractivityWin32 though because we're probably not going to add interactivity features to the conhost UI. We'd probably surface the actual click behavior through the new Terminal UI.

[egmontkob]

I think we'd also want to consider some sort of heuristic-based detection of hyperlinks

Sure thing :) But that's quite a different story.

[edburns]

FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue.

[jquast]

This feature is demonstrated in the blessed python library documentation, as @edburns mentions,

• a dotted underline is displayed and cannot be clicked,
• the ⌘ key is then held to show URL at bottom of the window,
• and the mouse cursor indicates it is now clickable, 👆

[jzabroski]

FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue.

@edburns Not only does iTerm2 do it, but according to the Symfony blog, iTerm2 collaborated with GNOME Terminal on a hyperlinking standard. Might be worth asking them where that standard is published.

Rendering clickable hyperlinks is one of the most important missing features of Console apps and commands. Although most of the terminal emulators auto-detect URLs and allow to click on them with some key combination, it's not possible to render clickable text that points to some arbitrary URL.

However, some terminal emulators led by iTerm2 and GNOME Terminal, are working on a new specification to add support for this feature.

[DHowett-MSFT]

Might be worth asking them

Fortunately, @egmontkob's already weighed in on this issue with his authority and some implementation details 😄

[jzabroski]

😆 I just realized the first link in this post is about what I just posted about. Sorry.

[andrzejnovak]

So. this happening at some point? Probably last thing holding me back from switching.

[tracker1]

@egmontkob, for what it's worth, I know your comments were from a while ago... would probably be best to use another 12 bits (3-bytes per character total) for a number that is a reference to a list of links experienced in the terminal... that just adds to the list up to the 12-bit unsigned value max (4095)... then loop if the end is hit... could do an arbitrary limit of a lower number as well. This way you're only growing a little bit per character in addition to the list for the links themselves.

-- edit:

Only real risk would be for going back into a really long scroll buffer would potentially be a wrong link... but shouldn't be a huge issue in that. and would only really be a per-session issue as well.

[DHowett]

So, just to level-set for when this issue is closed by PR #7251:

This issue only tracks support for application-generated hyperlinks.
Automatic detection of hyperlinks is in a different workitem in this repo, and it's something we're still planning to have. Don't be alarmed when this issue closes that we don't have it.

We ended up going with something similar to #204 (comment), except using a SHORT instead of something 12 bits wide. Every attribute is now 16 bits longer, but because they're run-length encoded the common case buffer size doesn't grow all that much. There's a bit more memory usage in highly attribute-turbulent scenarios, but it isn't all that bad.

[jzabroski]

@DHowett Now's a good time to go to AirBnb. book availability at the last Blockbuster Video, and watch a VHS copy of Lawnmower Man. Because we just entered virtual reality! Thank you very much.

[stuaxo]

Fantastic work @PankajBhojwani thanks :)

[DHowett]

Anybody who'd like to track followup work, including automatic detection of URIs, should move on over to 👉 👉 #5001, the scenario issue tracking all further hyperlink efforts.

[ghost]

🎉This issue was addressed in #7251, which has now been successfully released as Windows Terminal Preview v1.4.2652.0.:tada:

Handy links:

• Release Notes
• Store Download