Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conversion hangs on large project #524

Closed
samcoder1 opened this issue Feb 14, 2020 · 31 comments · Fixed by #535
Closed

Conversion hangs on large project #524

samcoder1 opened this issue Feb 14, 2020 · 31 comments · Fixed by #535

Comments

@samcoder1
Copy link

I have a relatively large project (maybe 50-100 vb files) and when I initiate the conversion it says
phase 1 of 2
.......
phase 2 of 2
......
Then it just hangs. I tried waiting it out for about 12 hours and it never completes. I tried it with a small project and it worked fine. I tried it using VS2017 and VS2019.
Any ideas on what could be wrong?
Did I not wait long enough?
Is there a way to debug the source code and point to a different .vbproj?
@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 15, 2020 •
edited
Loading

Hi thanks for posting. In each phase it should list the files. So something is definitely wrong. I would guess there's something special about the project file or its location which stops it finding anything to convert. If you can share any project type details that would help a lot

e.g. If the path contains any of these strings it will be ignored:
https://github.com/icsharpcode/CodeConverter/blob/master/ICSharpCode.CodeConverter/Shared/ProjectConversion.cs#L124

e.g. If the vb is embedded within various ASP style pages (anything not ending in vb) it won't work.

@GrahamTheCoder
Copy link
Member

If you want to debug the source code, you can follow the instructions from the readme:
https://github.com/icsharpcode/CodeConverter#buildingrunning-from-source

Once you start the Vsix project in debug mode and load your project, I suggest setting a breakpoint here
https://github.com/icsharpcode/CodeConverter/blob/master/ICSharpCode.CodeConverter/Shared/ProjectConversion.cs#L82

You should see a document for each ".vb" file in the project

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 15, 2020 •
edited
Loading

Could you clarify whether there are any file names mentioned in either phase, or whether the "......." is literal? Is there a particular file it hangs on? I've just found an issue where large files take a long time in phase 1 and attempted to mitigate it, let me know if it improves with update 7.8 just released.

Even without the source code checked out, you could start another Visual Studio instance, click Debug...Attach to Process, and attach to the other "devenv.exe", then hit the pause button and have a look at the stack trace (and paste it here). Note: You may need to look at the Debug->Windows->Parallel stacks to find a relevant stack trace that mentions the code converter.

@samcoder1
Copy link
Author

Thanks for the responses. When I said "...." above, that was where the output listed all of the .vb files in the project. Phase 1 and Phase 2 listed all project files, however it would just hang after that.
Debugging didn't work. I did a comparison on the files listed in phase 1 and phase 2 and I discovered that there were 10 files missing in phase 2. One of them was the My Project/Settings.Designer.vb file. Since I didn't have any settings saved, I deleted it and ran it again. This time only 5 files were missing but only one file overlapped those two lists. I tried converting those files individually and they converted fine but I can't get the entire project to convert successfully. It just hangs when it's almost complete.

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 17, 2020 •
edited
Loading

Thanks for the extra info. Could you give any more details on what difficulty you're having attaching a debugger? If you'd like you can call ([email protected]) on hangouts, or email me an id for some other service (skype etc.) and either screenshare or talk through debugging.

Ah, also, did you update the extension to the new version (7.8)?

@GrahamTheCoder
Copy link
Member

It's possible there's some issue during converting the project file itself. One thing you could try, is deleting all files from the project (obviously make sure you have a copy saved somewhere else e.g. git).
Then try converting the project when it has absolutely no contents and see if it works. If not, could you sanitise the project file by changing any names you want to keep private and post it here?

@samcoder1
Copy link
Author

I didn't update to 7.8 but I decided to convert each file one at a time and then load it into a new project. So far I came across one file that references MongoDB, System.Collections.Specialized and Serilog but just hangs when converting that file. I came across another file that doesn't import anything and realized there are self-references in both of those classes. The first instantiates itself within a public method and the second calls a Public Shared object.
Is it possible that self references cause it to spiral into a death loop?

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 17, 2020 •
edited
Loading

Possible - there are various bits of logic that follow references (e.g. trying to find the constructors not called by others). Are you able to post a version of either file for me to debug? If you can't post the whole thing, perhaps try deleting about half at a time then converting to see which bit causes the issue, and just post that bit? Thanks!

@samcoder1
Copy link
Author

samcoder1 commented Feb 17, 2020 •
edited by GrahamTheCoder
Loading 

Imports System.Collections.Specialized
Imports System.Configuration
Imports System.Net.Http
Imports System.Web
Imports DTS.Logging.MongoDB
Imports DTS.Logging.MongoDB.Configuration
Imports Serilog

''' <summary>
''' 
''' </summary>
''' <remarks></remarks>
Public Class LoggingDebugging

    Public Shared EnabledLogTypes As NameValueCollection = Util.GetConfigSection("Logging")
    Public Shared EnabledLogs As NameValueCollection = Util.GetConfigSection("enabledLogs")
    'Private Shared sempaphore As Object = New Object()
    ''' <summary>
    ''' Proxy method to set the LogInstance properties and add LogInstance class to LogBuffer
    ''' </summary>
    ''' <param name="logType"></param>
    ''' <param name="message"></param>
    ''' <param name="traceKey"></param>
    ''' <param name="user"></param>
    ''' <param name="TS"></param>
    ''' <returns></returns>
    ''' <remarks></remarks>
    Public Shared Function Log(ByVal logType As String, ByVal message As String, Optional ByVal traceKey As Long = Nothing, Optional ByVal user As String = Nothing, Optional ByVal TS As Long = Nothing) As Long

        Dim LogTS As Long

        If IsNothing(traceKey) Or traceKey = 0 Then
            Throw New ETDMSException(Util.t("traceKey must not be null. Please modify your method to either create a new " +
                                       "traceKey if it is the start of a request, or use an existing traceKey if it is " +
                                       "part of a request."))
        End If

        If LoggingDebugging.EnabledLogTypes(logType) = "True" Then

            Dim loginstance As New LogInstance(m_logType:=logType, m_message:=message, m_user:=user, m_TS:=TS)

            LogTS = LogAggregator.AddLogInstance(loginstance, traceKey)


        Else
            'If logType = "query" Then
            '    Dim LogEntry As New LogInstance(m_logType:=logType, m_message:=message, m_user:=user, m_TS:=TS)
            '    LogTS = LoggingDebugging.AddLogBuffer(Props:=LogEntry, traceKey:=traceKey)
            'End If
            LogTS = Now.Ticks()
        End If

        Return LogTS

    End Function



    ''' <summary>
    ''' Fluh out from Logbuffer
    ''' </summary>
    ''' <param name="userPin"></param>
    ''' <param name="traceKey"></param>
    ''' <remarks></remarks>
    Public Overloads Shared Sub FlushLog(ByVal userPin As String,
                                         Optional ByVal traceKey As String = Nothing,
                                         Optional ByRef esTime As String = Nothing,
                                         Optional ByRef request As HttpRequest = Nothing,
                                         Optional ByRef message As HttpRequestMessage = Nothing)

        Dim logitem As LogItem = Nothing
        LogAggregator.TryGet(userPin, logitem, traceKey, esTime, request, message)

    End Sub


    ''' <summary>
    ''' Converts ticks to Date and Time
    ''' </summary>
    ''' <param name="ticks"></param>
    ''' <returns></returns>
    ''' <remarks></remarks>
    Public Shared Function convertTickstoDateTime(ByVal ticks As Long) As String
        Dim dt As New DateTime(ticks)
        Return Format(dt, "MM/dd/yyy hh:mm:ss.fffffff")
    End Function

    ''' <summary>
    ''' Converts ticks to time
    ''' </summary>
    ''' <param name="ticks"></param>
    ''' <returns></returns>
    ''' <remarks></remarks>
    Public Shared Function convertTickstoTime(ByVal ticks As Long) As String
        Dim timespan As New TimeSpan(ticks)
        Return String.Format("{0}:{1}:{2}:{3}",
            timespan.Hours.ToString("D2"), timespan.Minutes.ToString("D2"),
            timespan.Seconds.ToString("D2"), timespan.Milliseconds.ToString("D3"))
    End Function

    ''' <summary>
    ''' Converts ticks to Date and Time with 24 hrs
    ''' </summary>
    ''' <param name="ticks"></param>
    ''' <returns></returns>
    ''' <remarks></remarks>
    Public Shared Function convertTickstoLongDateTime(ByVal ticks As Long) As String
        Dim dt As New DateTime(ticks)
        Return dt.ToString()
    End Function

    ''' <summary>
    ''' Reads configuration values for serilog
    ''' </summary>
    Public Shared Sub Configure()

        'Serilog.Log.Logger = New Serilog.LoggerConfiguration().ReadFrom.AppSettings().CreateLogger()

        Dim settings As StructuredMongoDBSettings = ConfigurationManager.AppSettings.MongoDBSettings()
        Serilog.Log.Logger = New Serilog.LoggerConfiguration() _
            .ReadFrom.AppSettings() _
            .Enrich.WithProperty("Application", "user-service-dev") _
            .Enrich.WithProperty("Stage", "local") _
            .WriteTo.StructuredMongoDB(settings) _
            .CreateLogger()

        Serilog.Log.Information("logger initialized")

    End Sub

End Class

@GrahamTheCoder
Copy link
Member

I tried against 7.7 and 7.8 in Visual Studio, and 7.8 on the web converter just to check but it converted in a couple of seconds on all of them. Could be related to other files or something else about the system setup.

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 17, 2020 •
edited
Loading

If you want to debug, try this or let me know which step you get up to:

You'll need two visual studio instances, one which will run the conversion, one which will attach to debug that process. Once it's hanging, do this:

image
2. Attach to the hanging VS
image
3. Pause
image
4.
image
5.
image
I need to see what's on the stack at the very top of the image

@GrahamTheCoder GrahamTheCoder mentioned this issue Feb 18, 2020
Closed
@kuxiaochouyu
Copy link

HI
I have the same issue, and I find that the possible problem is the conversion of design files. However, when I convert form1.designer.vb file alone, it will also hang.

@kuxiaochouyu
Copy link

I built a small project and found that it could be converted. But there are no events in C#'s design file( such as this.button.Click += new System.EventHandler(this.button_Click);).
I think the event is the most important for me. Because in the current project, there are thousands of events that cannot be changed one by one.

@samcoder1
Copy link
Author

Nothing is showing up on parallel stacks.
image
This is the other VS instance:
image
When I try to pause the debugger, it always tries to look into a .NET library.
image
The worst part about it is the worker thread goes rogue. It causes my CPU to almost max out and won't stop even if I close Visual Studio. I have to kill VS in Task Manager.

@samcoder1
Copy link
Author

Here is something interesting. When I ran the code through the online converter, it didn't copy all of the imports over. And I am using them.
image
It may be due to an issue of the converter not knowing that the System.Collections.Specialized.NameValueCollection should be used like an array. VB uses parenthesis and C# uses brackets. I noticed in the converter it kept the parenthesis.
image

@samcoder1
Copy link
Author

Also, it doesn't change Now.Ticks() to DateTime.Now.Ticks.
image

@samcoder1
Copy link
Author

In the second file I had issues with, the following wasn't converted properly
If startDate = endDate Then startDate = DateAdd(DateInterval.Second, 1, startDate) endDate = DateAdd(DateInterval.Day, 1, startDate) End If if (startDate == endDate) { startDate = DateAdd(Microsoft.VisualBasic.DateInterval.Second, 1, startDate); endDate = DateAdd(Microsoft.VisualBasic.DateInterval.Day, 1, startDate); }
It should be
if (startDate == endDate) { startDate = startDate.AddSeconds(1); endDate = endDate.AddDays(1); }

@GrahamTheCoder GrahamTheCoder mentioned this issue Feb 18, 2020
Closed
@GrahamTheCoder
Copy link
Member

@samcoder1

In the parallel stacks screenshot I can see that you weren't paused. Can you wait until it's definitely hanging, then pause, then look at parallel stacks please. Don't worry about the msccorlib message, that's just trying to show the spot you're paused on one of the stacks.

Re: Missing usings, brackets, "DateTime": I've filed that as #529 It will never get it perfect since it won't have access to all the project references, but it shouldn't remove things that it doesn't recognize, and should really get DateTime.Now (or DateAndTime.Now at least), same for the DateAdd function.

@GrahamTheCoder GrahamTheCoder mentioned this issue Feb 19, 2020
Closed
@samcoder1
Copy link
Author

Ok, I paused it when was hanging after printing the Phase 2 of 2 message. Here is the parallel stack:
image
image
image

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 20, 2020 •
edited
Loading

@samcoder1 Great, thanks for all your efforts helping with this - that's just the sort of thing I need. It probably narrows it down to the "simplification" phase (which is phase 2) which gets rid of redundant qualifications, rather than the bit after it. Would you mind doing this a few times:

  • Click "play"
  • Wait about a minute
  • Click pause
  • Paste the stacks each time (it's the full stack of the big one on the right hand side that's of most interest - especially any lines mentioning DocumentExtensions. I think in the previous screenshot some lines might be hidden by the zoomed out view that you can hide with the icon in the bottom right).

Another user (Peter) has reported a similar issue and sent me his entire solution. But when I convert it, the ~250 files just take about 10 minutes in total. I have a fast SSD and 16GB of RAM, and I'm wondering if that's hiding the issue for me, so I'm interested in the other stats. Would you mind:

  • Running a conversion without the debugger attached (i.e. just one visual studio open like when you originally saw the bug)
  • Wait until the output log hasn't updated for a couple of minutes (i.e. definitely hanging)
  • Open up the performance tab of task manager on the "disk" page, and leave it 30 seconds for the graphs to fill up.
  • Send a screenshot of that

My hypothesis is that you'll see a lot of hard disk activity for one of these reasons from most to least likely:

  • Your RAM is being maxed out and the system is paging parts to disk
  • Roslyn is heavily caching the syntax trees on disk (I think any over 4K get cached on disk)
  • The above option, plus an antivirus tool getting involved trying to scan all the cache files

Finally (if you aren't tired of sending stuff!) could you send me the information from your suspect there's a detail about Visual Studio version affecting this, but it could be some other detail like RAM or disk.
Could you let me know which Visual Studio version you're using by going to Help...About and pasting the results of "copy info"? Thanks!

@kuxiaochouyu If you can provide any similar information on the hanging problem, that'd also be very helpful.

One other thing that could be useful is to see the result of running a profiler while it's hanging:
https://docs.microsoft.com/en-us/visualstudio/profiling/profiling-feature-tour?view=vs-2019#post_mortem

@Peter-Simpson
Copy link

Hi Graham,

I don't know whether this will help but I attached a debugger to my VS instance when it was trying to convert the project from C# to VB and it stopped with a 'System.Collections.Generic.KeyNotFoundException' in mscorlib.dll exception

I couldn't see any reference to your code anywhere in the stack trace unfortunately.

Regards, Peter

Stack trace:

Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.ErrorReporting.FatalError.Report(System.Exception exception, System.Action<System.Exception> handler) Line 212 C#
Microsoft.CodeAnalysis.CSharp.Workspaces.dll!Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync(string replacementText, Microsoft.CodeAnalysis.ISymbol renamedSymbol, Microsoft.CodeAnalysis.ISymbol renameSymbol, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId> referencedSymbols, Microsoft.CodeAnalysis.Solution baseSolution, Microsoft.CodeAnalysis.Solution newSolution, System.Collections.Generic.IDictionary<Microsoft.CodeAnalysis.Location, Microsoft.CodeAnalysis.Location> reverseMappedLocations, System.Threading.CancellationToken cancellationToken) Unknown
[Resuming Async Method]
[Native to Managed Transition]
mscorlib.dll!System.Collections.Generic.Dictionary<System.__Canon, System.__Canon>.this[System.__Canon].get(System.__Canon key) Line 183 C#
Microsoft.CodeAnalysis.CSharp.Workspaces.dll!Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync.AnonymousMethod__3(Microsoft.CodeAnalysis.Location t) Unknown
System.Core.dll!System.Linq.Enumerable.WhereSelectArrayIterator<System.__Canon, System.__Canon>.MoveNext() Unknown
System.Collections.Immutable.dll!System.Collections.Immutable.ImmutableArray<Microsoft.CodeAnalysis.Location>.Builder.AddRange(System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.Location> items) Line 278 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder<System.__Canon>.AddRange(System.Collections.Generic.IEnumerable<System.__Canon> items) Line 477 C#
Microsoft.CodeAnalysis.CSharp.Workspaces.dll!Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync(string replacementText, Microsoft.CodeAnalysis.ISymbol renamedSymbol, Microsoft.CodeAnalysis.ISymbol renameSymbol, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId> referencedSymbols, Microsoft.CodeAnalysis.Solution baseSolution, Microsoft.CodeAnalysis.Solution newSolution, System.Collections.Generic.IDictionary<Microsoft.CodeAnalysis.Location, Microsoft.CodeAnalysis.Location> reverseMappedLocations, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.CodeAnalysis.CSharp.Workspaces.dll!Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync(string replacementText, Microsoft.CodeAnalysis.ISymbol renamedSymbol, Microsoft.CodeAnalysis.ISymbol renameSymbol, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId> referencedSymbols, Microsoft.CodeAnalysis.Solution baseSolution, Microsoft.CodeAnalysis.Solution newSolution, System.Collections.Generic.IDictionary<Microsoft.CodeAnalysis.Location, Microsoft.CodeAnalysis.Location> reverseMappedLocations, System.Threading.CancellationToken cancellationToken) Unknown
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolver.AddDeclarationConflictsAsync(Microsoft.CodeAnalysis.ISymbol renamedSymbol, Microsoft.CodeAnalysis.ISymbol renameSymbol, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId> referencedSymbols, Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolution conflictResolution, System.Collections.Generic.IDictionary<Microsoft.CodeAnalysis.Location, Microsoft.CodeAnalysis.Location> reverseMappedLocations, System.Threading.CancellationToken cancellationToken) Line 238 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolver.Session.IdentifyConflictsAsync(System.Collections.Generic.HashSet<Microsoft.CodeAnalysis.DocumentId> documentIdsForConflictResolution, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.DocumentId> allDocumentIdsInProject, Microsoft.CodeAnalysis.ProjectId projectId, Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolution conflictResolution) Line 436 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolver.Session.ResolveConflictsAsync() Line 149 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.ConflictEngine.ConflictResolver.ResolveConflictsAsync(Microsoft.CodeAnalysis.Rename.RenameLocations renameLocationSet, string originalText, string replacementText, Microsoft.CodeAnalysis.Options.OptionSet optionSet, System.Func<System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.ISymbol>, bool?> hasConflict, System.Threading.CancellationToken cancellationToken) Line 67 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.Renamer.RenameAsync(Microsoft.CodeAnalysis.Rename.RenameLocations locations, string newName, System.Func<Microsoft.CodeAnalysis.Location, bool> filter, System.Func<System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.ISymbol>, bool?> hasConflict, System.Threading.CancellationToken cancellationToken) Line 76 C#
Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.Renamer.RenameSymbolAsync(Microsoft.CodeAnalysis.Solution solution, Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId symbolAndProjectId, string newName, Microsoft.CodeAnalysis.Options.OptionSet options, System.Func<Microsoft.CodeAnalysis.Location, bool> filter, System.Func<System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.ISymbol>, bool?> hasConflict, System.Threading.CancellationToken cancellationToken) Line 104 C#
[Resuming Async Method]
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine) Line 1090 C#
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Line 1070 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Line 977 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0() Line 288 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask) Line 810 C#
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations() Line 3617 C#
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Line 2363 C#
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 490 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.CodeAnalysis.Rename.RenameLocations>.SetResult(Microsoft.CodeAnalysis.Rename.RenameLocations result) Line 608 C#
[Completed] Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.RenameLocations.FindAsync(Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId symbolAndProjectId, Microsoft.CodeAnalysis.Solution solution, Microsoft.CodeAnalysis.Options.OptionSet optionSet, System.Threading.CancellationToken cancellationToken) Line 134 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine) Line 1090 C#
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Line 1070 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Line 977 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0() Line 288 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action action, bool allowInlining, ref System.Threading.Tasks.Task currentTask) Line 810 C#
mscorlib.dll!System.Threading.Tasks.Task.FinishContinuations() Line 3617 C#
mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Line 2363 C#
mscorlib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result) Line 490 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<Microsoft.CodeAnalysis.Rename.RenameLocations.SearchResult>.SetResult(Microsoft.CodeAnalysis.Rename.RenameLocations.SearchResult result) Line 608 C#
[Completed] Microsoft.CodeAnalysis.Workspaces.dll!Microsoft.CodeAnalysis.Rename.RenameLocations.AddLocationsReferenceSymbolsAsync(Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId symbolAndProjectId, Microsoft.CodeAnalysis.Solution solution, System.Threading.CancellationToken cancellationToken) Line 218 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(object stateMachine) Line 1090 C#
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 980 C#
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) Line 928 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run() Line 1070 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.OutputAsyncCausalityEvents.AnonymousMethod__0() Line 977 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Runtime.CompilerServices.TaskAwaiter.OutputWaitEtwEvents.AnonymousMethod__0() Line 288 C#
mscorlib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke() Line 1123 C#
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.ExecuteWorkItemHelper() Line 681 C#
mscorlib.dll!System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() Line 715 C#
mscorlib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 820 C#
mscorlib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() Line 1161 C#

When I moved on 1 step I got this message:

This exception was originally thrown at this call stack:
System.Collections.Generic.Dictionary<TKey, TValue>.this[TKey].get(TKey)
Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync.AnonymousMethod__3(Microsoft.CodeAnalysis.Location)
System.Linq.Enumerable.WhereSelectArrayIterator<TSource, TResult>.MoveNext()
System.Collections.Immutable.ImmutableArray.Builder.AddRange(System.Collections.Generic.IEnumerable)
Microsoft.CodeAnalysis.PooledObjects.ArrayBuilder.AddRange(System.Collections.Generic.IEnumerable) in ArrayBuilder.cs
Microsoft.CodeAnalysis.CSharp.Rename.CSharpRenameConflictLanguageService.ComputeDeclarationConflictsAsync(string, Microsoft.CodeAnalysis.ISymbol, Microsoft.CodeAnalysis.ISymbol, System.Collections.Generic.IEnumerable<Microsoft.CodeAnalysis.FindSymbols.SymbolAndProjectId>, Microsoft.CodeAnalysis.Solution, Microsoft.CodeAnalysis.Solution, System.Collections.Generic.IDictionary<Microsoft.CodeAnalysis.Location, Microsoft.CodeAnalysis.Location>, System.Threading.CancellationToken)

And VS terminated on the next step leaving this message in the output:

The thread 0x6240 has exited with code 0 (0x0).
'devenv.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\PrivateAssemblies\Runtime\Microsoft.VisualStudio.Debugger.Runtime.dll'.
The program '[0x179C] devenv.exe' has exited with code -2146232797 (0x80131623).

@GrahamTheCoder
Copy link
Member

Hi Peter. Thanks, I'm not sure it's related to hanging, but it could be useful in two ways. First, I've noticed I'd got muddled which direction you were converting previously. I'll try the conversion in the intended direction (C# to VB) on your solution sometime soon.

Second: It is indeed a roslyn bug and a roslyn error handler that intentionally crashes the process, but I believed I'd worked the issue in the most recent release sufficiently to prevent a crash. So it seems like I must have missed a case. For general reference, the roslyn repo is tracking removing the hard-crash as dotnet/roslyn#32785

@Peter-Simpson
Copy link

Peter-Simpson commented Feb 22, 2020 via email

@GrahamTheCoder
Copy link
Member

Hi Peter. I've made a further attempt at preventing Roslyn's hard crashes, and fixed a converter bug that was triggering the root exception that was leading to a crash. There's a CI build you can try here if you'd like:

https://icsharpcode.visualstudio.com/icsharpcode-pipelines/_build/results?buildId=930&view=artifacts&type=publishedArtifacts

I don't have any improvements around the hanging yet. But I did find that when converting your 11k line DiagnosticForm to C# then back to VB, I managed to cause it to appear to hang in the C#->VB direction. It actually came back and completed the conversion after a couple of minutes, but I imagine on different hardware or an even larger file the wait could be substantially longer (though the 12 hours mentioned by @samcoder1 is extreme!)

So I suspect running the simplifier on large files is causing the hanging. I'll look at the options for mitigating this and then post here with an update.

@Peter-Simpson
Copy link

Peter-Simpson commented Feb 22, 2020 via email

@GrahamTheCoder
Copy link
Member

Thanks for getting back to me, great to hear it's working.
The issue you mention sounds like the case conflict resolver got a bit carried away. I'll try to get that fixed before the next release since it should be easy - I've added it to my list in #498 just so I remember

I really appreciate all the help!

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Feb 24, 2020 •
edited
Loading

I have a theory about what's going on in this case now. I believe the algorithm used in Roslyn for simplification is linear(ish) in the best case, but quadratic in the worst case.
i.e. If there's a shortage of RAM and garbage collector kicks in very often, a 10k line file might take 10k times longer to simplify in the worst case.
(Note: It won't actually be lines that is the crucial scaling factor here)

I believe this is due to using WeakReference to cache semantic models and thus having to recompute them very frequently which is a very slow operation, and can be slower for larger files, projects and solutions depending how well other caching is working (usually badly when GCs are happening frequently).

To combat this, there are a few things I could do without making any changes to roslyn:

  • Quick wins:
    • Put a timeout on the simplification process of 1 minute per file
    • Convert files in descending order of size, giving more free memory to use in simplification.
  • Bigger changes:
    • Write out simplified files as they complete, rather than waiting for all before writing to disk (a bit of an architectural change may be required to make this work)

@Peter-Simpson
Copy link

I used to think that my 16Gb PC was huge, but no longer given discussion on this thread and my recent experience trying to manipulate images from large astronomical cameras!

@GrahamTheCoder GrahamTheCoder mentioned this issue Feb 29, 2020
Merged
@GrahamTheCoder
Copy link
Member

Hi @samcoder1, @kuxiaochouyu, @Peter-Simpson,

I've some significant changes to the converter, one of which is to reduce memory usage in the hope of fixing this issue. If any of you could try out the latest CI build it'd be really helpful:

  • Uninstall the current version of the code converter extension
  • Close all Visual Studio windows
  • Download and unzip the "drop" artifact from here
  • Run Vsix\bin\Release\ICSharpCode.CodeConverter.VsExtension.vsix to install it

It now converts the files in descending order of size, so it's the first few files in phase 2 that are most likely to take a long time. Big files still take 10 minutes on my setup, and could easily take more on others. Let me know whether it works for you. If it's still making no progress after 30 minutes, then it'd be useful to know the memory usage, and CPU core usage e.g.
image
image
image

Thanks,
Graham

@Peter-Simpson
Copy link

Peter-Simpson commented Mar 1, 2020 via email

@GrahamTheCoder
Copy link
Member

Thanks for getting back to me so fast. That's enough information for me to be happy the performance changes are going in the right direction.

In general the C#->VB conversion is lower quality, so I'm not too surprised. Generally a lot of the resulting compile errors will be caused by a small number of root issues. I'll use your solution to reproduce some of the issues over the next few weeks. No pressure, but if you're interested in contributing any fixes yourself let me know and I'd be happy to help you get started. Thanks for all your help on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Async Enumerable icsharpcode/CodeConverter
4 participants
@GrahamTheCoder @samcoder1 @Peter-Simpson @kuxiaochouyu