It would be awesome if this could be merged.

In the meantime you can just use the fork.

@soffes any plans on merging this in?

Backreferences from end to begin like \1 are not supported (I would like to know how Texmate does this).

Any progress on this?

Nope, I decided the current state was good enough (backreferences are usually used to recognize languages embedded within other languages, which usually slows parsing down quite a bit). Also for this you would have to match the begin and end part at the same time which is kind of tricky to implement.

If you want to have a go at this, PR's are always welcome.

Yes, of course. Lua uses this syntax:

(Multiline Comment)
begin: --[(=*)[
end: ]\1]

(Multiline String)
begin: [(=*)[
end: ]\1]

I am working on an iOS Lua Editor, your work really helped me a lot!

You are welcome!

Yes, this is another use of back-references. I see how this can be useful.

If you want to implement this, your best bet is probably to combine the begin and end expressions with a .* and match that instead of just the begin expression. For this you would have to take a look at Pattern.swift and Parser.swift.

@alehed I've got a question (or five), partner!

When using AttributedParsingOperation to perform incremental parsing, it seems as though there are only two options: insertion or deletion. How might one go about handling the ”replacement” of text? There are three replacement situations that I continually run into where I'm not sure how best to handle parsing and/or re-parsing.

For instance, in all of the examples below, assume the following is true:

let source = "Hello, World!"
let replacementRange = NSRange(location: 7, length: 6) // "World!"

Given source is the target string and replacementRange is the range to be replaced within the target string, the possible replacement situations are:

• Exact replacement: The replacement string is exactly the length of the range being replaced. Thus, the change in length between the strings is 0 (i.e. no characters have been “deleted” or “inserted” — in the traditional sense of the words).

  let replacement = "There!"
  let result = source.replacingCharacters(in: replacementRange, with: replacement)
  print("\(source) => \(result) (delta: \(result.length - source.length))") 
  // ~> Hello, World! => Hello, There! (delta: 0)
  

• Additive replacement: The replacement string is longer than the range being replaced. Thus, the change in length between the strings is a positive value (i.e. characters have been “inserted”).

  let replacement = "My Dudes and Dudettes!"
  let result = source.replacingCharacters(in: replacementRange, with: replacement)
  print("\(source) => \(result) (delta: \(result.length - source.length))") 
  // ~> Hello, World! => Hello, My Dudes and Dudettes! (delta: 16)
  

• Subtractive replacement: The replacement string is shorter than the range being replaced. Thus, the change in length between the strings is a negative value (i.e. characters have been “deleted”).

  let replacement = "Pal!"
  let result = source.replacingCharacters(in: replacementRange, with: replacement)
  print("\(source) => \(result) (delta: \(result.length - source.length))") 
  // ~> Hello, World! => Hello, Pal! (delta: -2)
  

In these situations, does one "break down" the replacement operation into its sub-operations? What about the first case, where the string has had no change in length? Does one re-parse the string in its entirety?

As far as background information goes, I simply have a custom NSTextView subclass that processes the changes as they happen. In shouldChangeText(in:replacementString:), I record the change (if the change should be allowed), then on didChangeText(), I enqueue the parsing operation in a custom queue like you suggested (e.g. chaining the previous operation to the next, or if no operations have been performed, creating the initial operation with the theme, language, etc.). Everything works great until I want to edit the text storage in some other method (like to implement convertSpacesToTabs() or something similar). I never know which range(s) should be considered “dirty.”

If none of this makes sense, please let me know, and I'll try my best to expound on or simplify my question(s). Basically, I just want to know how best to handle the “replacement” aspect of the incremental parsing operation, rather than raw insertion/deletion scenarios.

Thanks!

P.S. Your changes to the base SyntaxKit are badass, and they've been a great help to me!

Hi @benstockdesign, glad you like it.

Replacement is quite simple, first you delete then you insert. Firstly, this model is simple and always works. Secondly, since the library depends on Foundation and UIKit it is designed to integrate well with the rest of UIKit. For instance NSTextStorage can notify you every time an update happens. If you do a replacement in a UITextView it will first send you a deletion and afterwards an insertion notification. So you never have to deal with actual replacements.

If you observe a different behavior by NSTextStorage, please let me know.

@alehed Okay, that clears things up. I was just never sure. Oh, wait, one last question: what about in NSTextStorage when edits are coalesced (i.e. beginEditing(); insert(); delete(); delete(); insert(); endEditing())? Do you just attack the final change or all of the in-betweeners? By the way, I'm working on an editor for the Mac (I honestly think I could be the only one), so maybe there's a difference when it comes to some of the UI_____Class vs. NS_____Class stuff. I don't think there are too many differences with the Cocoa text system, but maybe I've overlooked something. Either way, thanks, again!

@benstockdesign according to my experience user edits are never coalesced, correct me if I'm wrong. If you are doing edits programmatically either do all the changes in succession, or maybe it is faster to just reload the whole thing once in the end. You might want to write a performance test for that.

The underlying class is NSTextStorage for both UITextView and NSTextView.