.NET Framework Bookmark and Share   
 index > Microsoft Codename 'Oslo' > Problem with final tokens in context
 

Problem with final tokens in context

Currently I'm implementing a DSL with Oslo, but I've got some trouble with making the MGrammar.
In my language I've defined some keywords, which I've marked as an "final token" and with the appropiate highlight tag before the token. This works correctly. But nowI want to create a token whereI can use the keyword in a different context. To make the example more concreteI will show some example of my language:

module home



site

    site/index.html : home()

end



def home()

  ....

end


Site is a keyword, but not when used inside a path, then it's correct in that context. I can imagine more examples in my language which would give this type of problems. So my question is how I can arrange my grammar to solve this problem, because now it raises an error? Currently the site token looks like this:

@{Classification["Keyword"]} final token SiteKeyword = "site";


and my path grammar looks like this:
        // ~[\ \t\n\r\.\/\\]+

        token PathElement = ^(

                '\u0020' | // Space Character

                '\u0009' | // Horizontal tab

                '\u000A' | // New Line

                '\u000D' | // Carriage Return

                '\u002E' | // Point

                '\u005C' | // Backslash

                '\u002F')+ // Slash

        ;        

        

        //{PathElement "/"}+ -> Directory

        syntax Directory 

            = item:PathElement 

                => item

            | list: PathElement "/" item:PathElement

                => [valuesof(list), item];

        

        syntax Filename = p:PathElement "." f:FileExt

                => Filename[p,f];

               

        syntax Path = Directory "/" Filename | Filename;
Captain007
Finally after reading alot of documentation and also looking at other implementations of this language in tools like ANTLR, SableCC, ASF-SDFetc. I found out it was a problem which occurs also in that type of languages and this has nothing to do with Oslo but with the grammar specification itself. Only in ASF-SDF we can express this correct, because there lexing and parsing is combined. Now i've created a little workaround by extending the language such that paths must start with [ and end with an ]. Because I'm a beginner with Oslo,I was also using token rules and syntax rules on any place and I've now restructured the grammar that all the tokens are in the beginning of the file followed by syntax rules only. This works better than just using it all around in the grammar.
  • Marked As Answer byCaptain007 Monday, July 27, 2009 1:29 PM
  •  
Captain007
I assume you've tried this already, but if you remove the "final" modifier from SiteKeyword, it parses correctly (for me) using the May 09 CTP, but the input syntax highlighting in Intellipad is wrong, since it still highlights "site" even though it's not matching as SiteKeyword (it's matching as PathElement).
'blog: http://diakopter.blogspot.com/ JSMeta: http://jsmeta.org/
  • Unmarked As Answer byCaptain007 Monday, July 27, 2009 7:57 AM
  • Marked As Answer byCaptain007 Monday, July 27, 2009 6:03 AM
  •  
Matthew Wilson _diakopter_
Yes I've tried this already and I also have the highlighting issue/bug which let me think my grammar was wrong. Because I can't see in Intellipad which rule/token has been matched, this confused me. I hope you guys are already working ona fix for this problem?!
Captain007
I don't know whether they're working on it, but there's not a public bug report for that issue on connect.microsoft.com/oslo I'd be glad to add one, unless you want to.

-Matthew
'blog: http://diakopter.blogspot.com/ JSMeta: http://jsmeta.org/
Matthew Wilson _diakopter_

If you could file a bug, yes. I don't have the rights to add one, I think. This is a product in development which can become better if this kind of bugs are resolved and feedback is given from to community to the developers.

Captain007
Sure, I'll do it. You should be able to create a bug report from this page once you've activated your liveid on Connect.

'blog: http://diakopter.blogspot.com/ JSMeta: http://jsmeta.org/
  • Unmarked As Answer byCaptain007 Monday, July 27, 2009 7:57 AM
  •  
Matthew Wilson _diakopter_
I've overlooked the grammar again andI now resolved this issue. It's not a bug but I did not read the MGrammar reference good enough. Tokens are always the first rules which are evaluated and my grammar uses syntax rules to express something that should be an token. So I changed the syntax rules to tokenrulesand now it works fine! Then you can also mark the keyword tokens to final and the result is that the keyword tokens have the least priority which results in correct highlighting.

Edit: Oops, list constructions cannot appear in tokens so this is also not the solution.
  • Marked As Answer byCaptain007 Monday, July 27, 2009 8:00 AM
  • Unmarked As Answer byCaptain007 Monday, July 27, 2009 9:15 AM
  •  
Captain007
Finally after reading alot of documentation and also looking at other implementations of this language in tools like ANTLR, SableCC, ASF-SDFetc. I found out it was a problem which occurs also in that type of languages and this has nothing to do with Oslo but with the grammar specification itself. Only in ASF-SDF we can express this correct, because there lexing and parsing is combined. Now i've created a little workaround by extending the language such that paths must start with [ and end with an ]. Because I'm a beginner with Oslo,I was also using token rules and syntax rules on any place and I've now restructured the grammar that all the tokens are in the beginning of the file followed by syntax rules only. This works better than just using it all around in the grammar.
  • Marked As Answer byCaptain007 Monday, July 27, 2009 1:29 PM
  •  
Captain007
I wouldn't call that a workaround. I'd explain it by saying it's usually a good idea to use quoting delimiters (as you did with the []) to create nested "contexts" (e.g. "double-quoted context") when your language allows identical strings ("site" vs "site") as both keywords and a substring of a literal value denotation.

I call that "nested context" since many programming languages allow interpolation (of some subset of the "outer" language) inside of quoted strings (and if interpolated/quoted strings are permitted in that language subset, it allows interpolation [within interpolation [within interpolation [... etc.]]].

One language that allows such multiple-nested interpolation without "requoting" (e.g. "hello &pass_thru("world")!") is Perl 6: seeQuoting and Parsingand the link to Synopsis 2at the bottom of that page. That language can permit such a thing 1) since the "sigils" that initiate the interpolated code span are required to be "escaped" if they are not intended to denote an interpolation span, 2) since there is effectively an arbitrarily deeply nested number of "lexer/parser" pairs processing the text, and 3) since in many cases, portions of a Perl 6 code file must be compiled, interpreted, or otherwise evaluated before parsing continues. See Synopsis 5: Regexes and Rules if you'd like to dive into the complex (and obscenely powerful) language-integrated language grammar in Perl 6.

Sorry for the somewhat OT discussion... :)

-Matthew

'blog: http://diakopter.blogspot.com/ JSMeta: http://jsmeta.org/
Matthew Wilson _diakopter_

You can use google to search for other answers

Custom Search

More Threads

• Find All Command For Intellipad
• Annotate the token with left(n) or right(n) ?
• Error in a code snippet in the release notes for the May 2009 CTP
• mmake: running a buildcommand from Intellipad on grammars and models using Makefiles
• Quadrant from Oslo May 2009 CTP gets Fatal Error
• Completing the pipeline; MSchema + (Custom MGrammar-based DSL + input) -> Repository w/data
• OsloTool, a must have for Oslo Developers!
• Grammar Editing in May CTP
• MGrammar XAML Samples - File Not Found
• "System".XXX.dll vs. "Microsoft".XXX.dll