Observations on latest state

[scottkleinman]

I've been through the latest changes on lexos3 and noted a bunch of mostly superficial issues that we should look at. Many are quick fixes, so I'm just providing a running tally here:

Plotly (all tools that use it)

• Do we need "toggle spike lines" and "produced by plotly" in the toolbar?

Upload

• I uploaded one file on a slowish connection, and I didn't get an indicator that anything was happening. Perhaps show the spinner icon before loading starts.

Manage

• Merge selected Documents and Apply Class to Selected Documents have been disabled in the context menu.

Rolling Windows

• Before the graph is rendered, it would be a good idea to show the processing icon.
• When the graph is rendered, the window needs to autoscroll down to make it visible.
• Alternatively, we could put all the options in a collapsible panel. That would require slightly more work, but some people might like to toggle access to the options if they want to re-generate the graph.
• It might be a good idea to have an option to smooth the graph (see https://plot.ly/python/reference/#scatter-line-smoothing and https://stackoverflow.com/questions/38274102/is-there-any-way-to-make-a-plotly-scatter-have-smooth-lines-connecting-points).

Statistics

• In the sidebar, I believe "inter quartile" should be one word.
• It would be great to have question mark popups next to interquartile range and standard deviation. They can have boilerplate defintions for now, but eventually we could put in links to In the Margins discussion. Just thinking about making things easier for Humanities students...
• Below the sidebar, it might be good to put a button to scroll down to document-level statistics.
• The output of the anomalies might need some testing. I noticed that multiple documents with anomalies were not always displayed in order (sometimes doc1, doc2, doc3, sometimes doc1, doc3, doc2). The output may need to be run through a sort filter.

Hierarchical Clustering

• Nice job fixing the padding on the right side of the dendrogram. However, clicking the autoscale button in the toolbar gets rid of it. Perhaps we should remove the autoscale button?
• Vertical labels get cut off after 10 characters, which makes the dendrogram hard to interpret. If you zoom out, the labels get separated from the dendrogram leaves.

K-Means

• The Voronoi display could have an option to toggle the centroids off.
• I encountered an error whenever I tried to generate a Voronoi display with more than 10 clusters.

Topwords

• This tool does not seem to be operational yet (clicking "Get Topwords" does nothing).

Content Analysis

• I believe that this is not yet operational, but I wonder if the title could be changed to something more descriptive of what the tool does?

[WeiqiNs]

@scottkleinman Thank you so much for testing and reporting, we will work on fixing these issues.

[WeiqiNs]

#775 This fixes the Topwords tool.

[scottkleinman]

@Weiqi97 No problem! I changed the bullet points above to check boxes if people want to check off the fixes.

[WeiqiNs]

It makes sense that we want to smooth the rolling window graph result. However, in the implementation, we used ScatterGL instead of Scatter. ScatterGL implemented WebGL to increase the plotting speed. Unfortunately, when I went through the reference for ScatterGL, I didn't see the same smooth option that was listed in Scatter.

[ealitt]

Content analysis is operational and generates three graphs; word counts based on dictionaries, corpus counts, and file counts. We are still trying to come up with a better name. Maybe "Opinion Mining" (which is closer to sentiment analysis)? Mark's contacting the people who wrote the paper on content analysis for their opinion.

[danny98m]

Removing things from the plotly toolbar has actually proven to be quite difficult. Here is a relevant link. Since we are using python and js together to render the plotly plot its more difficult than just the example provided.

[ealitt]

@danny98m I found similar answers too about hiding parts of the toolbar (or modebar) using python (link). If we were generating the plotly graphs using only javascript, this would be possible (link). We'll just have to live with the default options for now.

[danny98m]

@dihydrogenmonoxide21 also feels kinda shady to remove the produced with plotly button haha.

[mathewleblanc]

@scottkleinman I just tried uploading a large file (~220MB) and there is a small loading icon near the bottom of the drag and drop area. This only appears in Firefox, though, and not Chrome. It may be some default feature in the file uploader that does not work in Chrome. I will try to add our usual loading icon instead.

[phuens]

RE: Merge selected Documents and Apply Class to Selected Documents have been disabled in the context menu.

@scottkleinman any help here? the right-click context menu does not show the Apply Class to Selected Docs upon entry; however, if we select some docs (on the manage page), then the Apply Class to Selected docs choice is available ... (??)

[scottkleinman]

On all of the above, today was a travel day. I'll get caught up tomorrow and see if I can help out.

[scottkleinman]

@phuens, have a look at line 821 of scripts_manage.js. It seems to work if I change this line to

num_rows_selected = table.rows('.selected').data().length

Can you test this solution to make sure?

[scottkleinman]

@mathewleblanc, there is a very peculiar comment in the code here. I'm not sure if it's related. But I think our normal icon, perhaps with a delay of a couple of seconds so that humans can catch it, will do the trick. From what I can tell, the code is in there and should be working. It may ultimately be a good idea to go to a third-party library like dropzone.js, but that's for another summer.

[scottkleinman]

For the various plotly toolbar issues. I only looked at hierarchical clustering, but I successfully modified line 96 of dendro_model.py to the following:

config = {
    'showLink': False,
    'displaylogo': False,
    'modeBarButtonsToRemove': [
        'autoScale2d',
        'toggleSpikelines'
    ]
}
div = plot(figure, config=config, output_type='div', include_plotlyjs=False)

This gets rid of the Plotly logo, the autoscale, and the toggle spike lines buttons. (The showLink may not be necessary; I think it defaults to false in the latest versions of Plotly).

When I tested this, the nice padding between the rightmost leaf and the end border of the graph was gone. I'm not sure if that's related.

[scottkleinman]

For further thoughts on plotly dendrograms, see issue #789.

[mathewleblanc]

@scottkleinman Merge Selected document and Apply Class to Selected Documents is definitely acting odd. In Manage one must deselect and then reselect the documents and then these options are enabled. Possibly related, the button to download selected documents does not appear unless using the same method.

Additionally, downloading documents does not work. The first batch of files downloads fine, but using the reset button and uploading new files, the first group of files will download and not the new set. This is done through a Flask route window.location = '/downloadScrubbing' that I am not familiar with. At the moment the only way to download a new batch of files is to clear one's cache and re-upload.

[scottkleinman]

@mathewleblanc Look at my comment to @phuens above in this thread. That seems to fix the problem for me. Basically, the code is not identifying the active documents. If you find the code to show the download icon and substitute the line above, I imagine that will work as well.

In terms of the downloading issue, we may need to insert a random number into the download link. I'll have a look when I get a chance.

[phuens]

@scottkleinman Thank you for the help with the code. As of right now, we have an issue of a pop-up error-modal that appears whenever you try to merge documents. This error is present in the master server and lexos3 server as well. I figured that removing line 550 of scripts_manage.js lets us get rid of the pop-up error-modal when you merge. I just wanted to confirm this with you because I don't think we need that line but I am not 100% sure.

[scottkleinman]

@phuens It is hard for me to diagnose this. I just pulled the master branch, and all files get de-selected when I right click. Same thing in refactor/manage branch. On both servers the documents remain selected, but the merge option is greyed out when I right-click. This wasn't the case yesterday. So there is something bigger going on here. What branch are you working in?

[mleblanc321]

dang; @phuens this is a priority

[phuens]

@scottkleinman The right click merge option is working on lexos3 server. We updated the lexos3 server yesterday. The manage page seems to be working fine on lexos3 sever.

[scottkleinman]

@phuens Are you sure? I just tried it on the server, and it's still de-deselecting documents when I right-click.

[phuens]

@scottkleinman Could you be able to check the manage page again since we updated the lexos3 server. It seems to be working well on everyone's computer here.

[mleblanc321]

@scottkleinman @phuens @Weiqi97 -- dang, it is alil' random; seems to deselect at odd times, but not always ...

[Myles-Trevino]

The definite issues in the original checklist are solved.