Scripting revamp #659

filipw · 2016-11-04T13:59:50Z

This PR is an attempt to revamp/normalize the C# scripting support in Omnisharp.

Here is what is done:

removed all dependencies on scriptcs - based on Roslyn directly. Omnisharp makes no assumptions about "scripting dialect" any more. It tries to be as close to "standard" as possible. Also renamed things like ScriptCsProjectContext to ScriptProjectContext etc etc
scripting intellisense is now compatible with InteractiveScriptGlobals - same as used by csi.exe. This gives us same set of global properties as there, i.e. Args.
~~scripting is (for now) supported only as .NET Core scripting~~ update: you can specify net46 target in project.json or use no project.json at all to force "net46 scripting mode". If your project.json contains a target for netcoreapp1.0, then .NET Core language services will be used
dependencies for scripts are no longer defined by the old packages.config, instead they are pulled from project.json
no implicit using statements or assembly references are made - except for System namespace, mscorlib and System.Runtime (if needed).
suppress a couple of code check warnings which are not applicable to scripting code - CS8099 & CS1701

This PR is in line with my proposal here dotnet/vscode-csharp#23 (comment)

This is a breaking change of course for scripting story in Omnisharp, but I believe it was never really officially supported but more of an experimental stuff. I believe trying to normalize to "pure" Roslyn scripting is a step in a right direction here, especially as we can easily align with runners like csi.exe then or dotnet-script

filipw · 2016-11-04T14:02:45Z

This PR also - from the scripting perspective - removes Omnisharp's dependency on full desktop framework

david-driscoll · 2016-11-05T12:25:10Z

I am in favor of this change! Will see if I have time to look at it before summit... if not we'll sit down at summit and talk about it :)

DustinCampbell · 2016-11-10T09:19:53Z

src/OmniSharp.Script/ScriptProjectSystem.cs

-        IOmnisharpEnvironment Env { get; }
-        ScriptCsContext Context { get; }
-        ILogger Logger { get; }
+        private CSharpParseOptions CsxParseOptions { get; } = new CSharpParseOptions(LanguageVersion.CSharp6, DocumentationMode.Parse, SourceCodeKind.Script);


Use LanguageVersion.Default to ensure that the latest version of the language is used?

DustinCampbell · 2016-11-10T09:22:34Z

Thanks for making this change @filipw! I definitely agree it's a step in the right direction.

DustinCampbell · 2016-11-10T09:24:55Z

src/OmniSharp.Script/ScriptProjectSystem.cs

-            var localScriptServices = baseBuilder.ScriptName(csxPath).Build();
-            var processResult = localScriptServices.FilePreProcessor.ProcessFile(csxPath);
+
+            var fileParser = new FileParser(Context.RootPath);


Any reason not to just parse the file with Roslyn?

to be honest I would love to, and I tried but couldn't find the right APIs. In fact - unless I am mistaken - the necessary APIs (this?) are not available publicly. What we need here is to walk the dependency tree of #load directives and gather all the files and their using statements and so on so that we can construct ProjectInfo correctly. It seems the only way to publicly get there with Roslyn scripting APIs at the moment is to manually create a script compilation which I wanted to avoid

This can be done syntactically, correct? If so, SyntaxTree.ParseText(...) is what you're looking for.

yeah. The only "hurdle" is if I encounter a #load foo.csx directive, I need to follow it and pick up the foo.csx and that one might have it's own #loads and so on.

Scripting APIs would use this to resolve that whole dependency tree so maybe that would work, to parse the entry file and then follow it's #loads using ScriptSourceResolver recursively.

but that would be better than manual parsing, definitely

I guess I'm not seeing that as much of a hurdle. You could still have the file parser but use the SyntaxTree to find the using's, #load's, and #r's. You could keep the logic where it continues processing #load's. The real issue is that the simple text "parsing" isn't going to do all that good of a job. For example, it looks like it would return a weird malformed using if it countered a using static.

Right I see what you mean now. Of course, let me pluck the naive syntax parsing 👍

filipw · 2016-11-10T17:03:01Z

Thanks for making this change @filipw! I definitely agree it's a step in the right direction.

great! I also had a chat with Tomas yesterday and he suggested that it might be a better idea to treat all CSX files as single project, rather than each of them as a separate 1-file project (which is how it is implemented at the moment). I will try to incorporate this change too.

DustinCampbell · 2016-11-23T04:58:54Z

How is this going @filipw?

filipw · 2016-11-23T06:14:41Z

It's coming!

On Nov 23, 2016 5:58 AM, "Dustin Campbell" [email protected] wrote:

How is this going @filipw https://github.com/filipw?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#659 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABoZIeceDUScLBLqXA2uRCUKES0XQDZFks5rA8gPgaJpZM4KpkTV
.

filipw · 2016-11-24T09:00:22Z

I've rebased this and introduced better parsing of files (no more naive parser, instead use SyntaxTrees) as suggested. Also improved some of the overall logic.

Tested in the following scenarios:

.net core single-file CSX script with nuget references in project.json
.net core multi-file CSX script with nuget references in project.json
accessing script args using globals from InteractiveScriptGlobals (same global object as in csi.exe)
.net desktop CSX script with nuget references in project.json and extra GAC references in project.json
.net desktop CSX script with no nuget references (everything self contained via #r)

This is still a simple approach to the problem, but it's definitely an improvement over what's there at the moment. My suggestion would be to go ahead with this merge and I can progressively make this better as we go forward (for example I'd like to include this in the future, though behaviorally things are OK already).

Finally, this solution works best on net46 build of OmniSharp because when we are in netcoreapp1.0 it's impossible to i.e. infer some of the assemblies without running into a lot of problems with object has not been imported. However my understanding is that net46 will be the default mode of distributing OmniSharp

DustinCampbell

Looking good! I had just a couple of questions.

DustinCampbell · 2016-11-24T15:28:00Z

src/OmniSharp.Script/FileParser.cs

+            var scriptCode = File.ReadAllText(fullPath);
+
+            var syntaxTree = CSharpSyntaxTree.ParseText(scriptCode, CSharpParseOptions.Default.
+                WithPreprocessorSymbols("load", "r").


Is this line necessary? This would have the effect of implicitly adding the following code at that top of the tree:

#define load #define r

Is that what you meant?

ah my mistake I thought I needed to use this to make the parser be aware of #load and #r as valid code. will remove

DustinCampbell · 2016-11-24T15:29:09Z

src/OmniSharp.Script/FileParser.cs

+            var loads = syntaxTree.GetCompilationUnitRoot().GetLoadDirectives().Select(x => x.File.ToString());
+            foreach (var load in loads)
+            {
+                var filePath = load.Replace("\"", string.Empty);


Does this work on OSX/Linux?

yes, this just strips away the quotes, tested on Mac too. when I have i.e. #load "bar.csx" in my CSX and I use GetXXXDirectives() method on the compilation root, the result string comes back with quotes inside i.e."bar.csx" not bar.csx and I want to strip it away to be able to concat the relative path and traverse these resources

Oh sorry. I misread the slash in there. :-) I glanced at it and read the slash, not the double-quote. 😄

DustinCampbell · 2016-11-24T15:30:38Z

src/OmniSharp.Script/FileParserResult.cs

+            LoadedScripts = new HashSet<string>();
+        }
+
+        public HashSet<string> Namespaces { get; }


FYI that read-only auto-props can have initializers. So you could just have this with no constructor:

public HashSet<string> Namespaces { get; } = new HashSet<string>(); public HashSet<string> References { get; } = new HashSet<string>(); public HashSet<string> LoadedScripts { get; } = new HashSet<string>();

Your choice.

DustinCampbell · 2016-11-24T15:32:14Z

src/OmniSharp.Script/ScriptProjectSystem.cs


-namespace OmniSharp.ScriptCs
+namespace OmniSharp.Script
 {
    [Export(typeof(IProjectSystem))]


Add Shared attribute so that MEF creates this as a singleton.

DustinCampbell · 2016-11-24T15:34:59Z

src/OmniSharp.Script/ScriptProjectSystem.cs


-            var scriptServices = baseScriptServicesBuilder.Build();
+            var runtimeContexts = File.Exists(Path.Combine(Env.Path, "project.json")) ? ProjectContext.CreateContextForEachTarget(Env.Path) : null;


If I understand what this is doing, it'll probably need to change in the future for csproj .NET Core. I might not understand though. 😄

yes correct I will want to change it in the future. There is no real official project model for scripting so at the moment the idea is that if you have a project.json it will be used to determine your dependencies and we can load them in for you to provide language services. We will in the future switch to csproj so that you can have <packagereference> there.

In all other cases, you get a no-frills service with mscorlib/system.runtime only and all your needed assemblies must be #r-ed manually by you

Sounds good.

DustinCampbell · 2016-11-24T15:35:26Z

src/OmniSharp.Script/ScriptProjectSystem.cs

+            // if we have no context, then we also have no dependencies
+            // we can assume desktop framework
+            // and add mscorlib
+            if (runtimeContexts == null || !runtimeContexts.Any())


if (runtimeContexts?.Any() == false)

DustinCampbell · 2016-11-24T15:37:10Z

src/OmniSharp.Script/ScriptProjectSystem.cs


-                Context.ScriptPacks.UnionWith(scriptPackSession.Contexts.Select(pack => pack.GetType().ToString()));
+            // inject all inherited assemblies
+//#if NET46


Can this be deleted?

DustinCampbell

Looks great with your latest changes.

filipw · 2016-11-24T18:03:29Z

@DustinCampbell everything should be OK now, all the review feedback applied - thanks!

filipw · 2016-11-24T18:07:17Z

DustinCampbell · 2016-11-24T18:35:33Z

Merging! 😄

filipw · 2016-11-24T18:44:50Z

yay - thanks 😄

gpuido · 2017-01-08T09:21:07Z

Sounds great !

what should I put in project.json to get started with something simple like the project in your gif ?

#659 (comment)

filipw · 2017-01-08T10:42:02Z

what should I put in project.json to get started with something simple like the project in your gif ?

If you use CSI to execute your scripts, the project.json can be empty, just {} (it's only needed to "light up" OmniSharp). You will be responsible for bringing in any assembly references you need using #r directives. This is the regular CSI experience.

If you use a .NET Core script runner (CSI is not .NET Core compatible at the moment) like https://github.com/filipw/dotnet-script, then you can put any package references you need into the project.json too.

gpuido · 2017-01-08T13:36:41Z

Thanks a lot filipw,

I got some of the frustation out with dotnet-script running .NET Core
but I'm on win10 and I'd like to target dotnet4.6, is this possible ?

CSI is not .NET Core compatible at the moment

So the 'regular CSI experience' is running on regular .net46 ?
because when I hit F5 vscode ask me to select an environment but only offer to select .NET Core.

DustinCampbell · 2017-01-09T02:49:14Z

If I'm understanding you correctly, the platforms that the debugger in VS Code supports (which is just CoreCLR) is completely orthogonal to what environment your scripts run in.

filipw · 2017-01-09T07:38:53Z

I got some of the frustation out with dotnet-script running .NET Core
but I'm on win10 and I'd like to target dotnet4.6, is this possible ?

no, at the moment that runner only exists to run .NET Core scripts.
In fact, it was built only to bridge the current gap, because CSI doesn't support .NET Core

So the 'regular CSI experience' is running on regular .net46 ?

correct, CSI is for running scripts as net46 only. It runs on mono (with some hiccups) too.

because when I hit F5 vscode ask me to select an environment but only offer to select .NET Core.

yep, as Dustin said, the F5 debugging in VS Code only works with CoreCLR, so you can use it to work with scripts running with dotnet-script, but not running with CSI.

gpuido · 2017-01-09T10:16:01Z

Got it now, Thanks guys.

This attempt to normalize the C# scripting support in Omnisharp is much welcomed.

DustinCampbell approved these changes Nov 10, 2016

View reviewed changes

DustinCampbell reviewed Nov 10, 2016

View reviewed changes

filipw force-pushed the feature/scripting-revamp branch 2 times, most recently from 022fb27 to f1e2491 Compare November 24, 2016 08:49

DustinCampbell reviewed Nov 24, 2016

View reviewed changes

filipw added 10 commits November 24, 2016 18:57

remove scriptcs

b886ff5

folder rename

ba285b8

scripting WIP

a5dbed5

updated file parsing logic

e5c92d5

general clean up

7d4c7e4

added scriptcs attribution

83ba62f

better file parsing and overall logic

46e88d7

some extra scripting reference logic

7df0560

updated to roslyn 2.0.0-rc

5f22e94

applied code review feedback

11f9e10

filipw force-pushed the feature/scripting-revamp branch from e8e99d2 to 11f9e10 Compare November 24, 2016 17:57

DustinCampbell approved these changes Nov 24, 2016

View reviewed changes

fixed build

9c45a68

DustinCampbell merged commit db3478f into OmniSharp:dev Nov 24, 2016

filipw deleted the feature/scripting-revamp branch November 24, 2016 18:44

filipw mentioned this pull request Nov 30, 2016

Unable to use multiple #load in CSX file #689

Closed

filipw mentioned this pull request Dec 20, 2016

Support "csx" dotnet/vscode-csharp#23

Closed

filipw mentioned this pull request Feb 8, 2017

Further improvements to CSX support #760

Merged

filipw mentioned this pull request Jun 7, 2018

Packages from scriptcs_packages.config are ignored in editor dotnet/vscode-csharp#2370

Closed


		var scriptServices = baseScriptServicesBuilder.Build();
		var runtimeContexts = File.Exists(Path.Combine(Env.Path, "project.json")) ? ProjectContext.CreateContextForEachTarget(Env.Path) : null;

Scripting revamp #659

Scripting revamp #659

Conversation

filipw commented Nov 4, 2016 • edited Loading

filipw commented Nov 4, 2016

david-driscoll commented Nov 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DustinCampbell commented Nov 10, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filipw commented Nov 10, 2016

DustinCampbell commented Nov 23, 2016

filipw commented Nov 23, 2016

filipw commented Nov 24, 2016

DustinCampbell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filipw Nov 24, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DustinCampbell left a comment

Choose a reason for hiding this comment

filipw commented Nov 24, 2016

filipw commented Nov 24, 2016

DustinCampbell commented Nov 24, 2016

filipw commented Nov 24, 2016

gpuido commented Jan 8, 2017

filipw commented Jan 8, 2017

gpuido commented Jan 8, 2017

DustinCampbell commented Jan 9, 2017

filipw commented Jan 9, 2017

gpuido commented Jan 9, 2017

filipw commented Nov 4, 2016 •

edited

Loading

filipw Nov 24, 2016 •

edited

Loading