Build global issues view

[miketaylr]

Essentially the Flask/Backbone guts of whatever gets designed in #169

• issues displayed
  • refactor issue models
  • write view code for issues to be shown
  • pull in "team" Figure out best way to work with "Team" metadata for issue on issue list #288
• issues filtering
  • issue count in filters Issue categories count #291
  • make reusable dropdown component
  • teach dropdown component about logged in state
  • is-active class for filter button
  • make filter dropdown interactive
  • clicking ouside filter closes it
  • make filter dropdown update search query
  • make clicking filter button update search query
  • add toggle behavior to "remove" filter
• issues sorting
  • make sort dropdown update search query
• issues search
  • add endpoint for search API
  • normalize data returned from search API to match what our model expects
  • update/re-render issues based on clicking filter (depends on having search in place)
  • arbitary search
  • clicking on label updates search & issues
  • If searchbox is empty, clear search results (return to all issues) #308, if searchbox is empty, render default
  • automatically search as user types (debounced)
• make sure updating search via filter + dropdown isn't mutually destructive
• remove transparent "search for everything" approach. (meaning, every filter/sort operation shouldn't update the search box... even if we end up using the search API for that, I think it will be too confusing)
• clicking on category button (untriaged, needs diagnosis, etc.) updates results.
• pagination
  • return Link header with XHR responses
  • write logic to track rel=next, rel=previous, etc. from client
  • when at first, disable previous, when at last disable next
  • re-render when asking for next page
  • add pagination support to relevant API endpoints
  • generalized Link href parsing and an endpoint to handle that.

[miketaylr]

is roughly what we'll be building.

[miketaylr]

Just brain dumping/storming here:

We have API endpoints for untriaged, needs diagnosis, ready for outreach and site contacted at: https://github.com/webcompat/webcompat.com/blob/master/webcompat/api/endpoints.py#L86-L148

We could easily make another one for "closed".

It would also be nice to grab all issues and just do the filtering on the clientside--but this becomes less feasible as we grow. There is an option to get them all, via ?state=all. But we have to consider download size in the case that we have thousands of issues one day.

OK, scratch that. GitHub will actually paginate this for us, even if we ask for all (see Link header):

$ curl -I https://api.github.com/repos/webcompat/web-bugs/issues?state=all
HTTP/1.1 200 OK
Server: GitHub.com
Date: Tue, 23 Sep 2014 16:24:04 GMT
Content-Type: application/json; charset=utf-8
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 58
X-RateLimit-Reset: 1411493038
Cache-Control: public, max-age=60, s-maxage=60
ETag: "b60f7fe2fc277672d58dc20aa47fc4a6"
Vary: Accept
X-GitHub-Media-Type: github.v3
Link: <https://api.github.com/repositories/17914657/issues?state=all&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?state=all&page=12>; rel="last"
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Security-Policy: default-src 'none'
Content-Length: 92394
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
X-GitHub-Request-Id: 425AD868:76F9:10C78C69:54219EA4
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
Vary: Accept-Encoding
X-Served-By: c436b2b44345c72ff906059f604991e7

[karlcow]

ooooh. ❤️

Link: <https://api.github.com/repositories/17914657/issues?state=all&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?state=all&page=12>; rel="last"

[miketaylr]

Ditching "Team" for now, #288.

[miketaylr]

So we've got two API sources for this: issues API and the search API. I'm going to attempt to get all the non-search data from the Issues API and leave the search API for user-initiated search.

(It's probably possible to do everything via the search API, but the rate limit would kill us.)

[karlcow]

@miketaylr when I was working on optimizing the code for #294 I realized that /api/issues/untriaged is in fact a filtered out version of the first page of all issues. And the same goes for the others. I don't know why I didn't realize that before. I guess that was the essence of your comment in
#266 (comment)
I need another pair of eyes for christmas ;) 👀

Trying to understand a bit more below.

https://api.github.com/repos/karlcow/webcompat-test/issues?labels=needsdiagnosis
https://api.github.com/repos/karlcow/webcompat-test/issues?labels=contactready
https://api.github.com/repos/karlcow/webcompat-test/issues?labels=sitewait
https://api.github.com/repos/karlcow/webcompat-test/issues

We never really have access to all issues.

• All issues is for now 8 pages. Link: <https://api.github.com/repositories/17914657/issues?page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?page=8>; rel="last"
• Sitewait is 2 pages Link: <https://api.github.com/repositories/17914657/issues?labels=sitewait&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?labels=sitewait&page=2>; rel="last"
• Contactready is 2 pages. Link: <https://api.github.com/repositories/17914657/issues?labels=contactready&page=2>; rel="next", <https://api.github.com/repositories/17914657/issues?labels=contactready&page=2>; rel="last"
• Needsdiagnosis is 1 page long. As of today it's the only one we fully retrieve.

Tell me if I'm wrong. In our current UI, how do we navigate older issues than the recent ones. Currently the home page is showing only the most recent ones. Luckily, we do not really offer the option to show the oldest ones first. We could not do it in the current design of the API.

Not sure there's an easy solution though.

Question:

• what do you call non search data?

[miketaylr]

We never really have access to all issues.

Not from the home page. As it's designed, it's just a "bite-sized" preview of our issues.

In our current UI, how do we navigate older issues than the recent ones.

From the homepage we don't. On the /issues page that we're building here in #266, there will be pagination and sorting, etc. Part of this task will be to add page/Link traversal to the API endpoints that the homepage uses. Or to make a more generic mechanism and ditch them altogether. 💣 We'll see.

Not a reflection of the final product, but should give you an idea:

[miketaylr]

what do you call non search data?

Sorry, very vague language on my part. In my mind the /issues page does three things: 1) filter issues 2) search issues 3) sort issues. 3 is probably just an operation on the results of 1 and 2. So non-search data is the result from a "filter" operation--clicking on "Needs Diagnosis", for example.

The point of my comment was just me thinking out loud--we can get all the same results from the Issues API with the Seach API. But it's way more expensive to do so. Probably OK if we could force all users to login, but not possible for the proxied "anonymous" approach.

[karlcow]

For the Issues page, aka, /issues, let me see if I understand correctly in terms of User actions and how it would work on the API side.

1. filter issues: aka click on [contactready] and get the first page of contactready. it means a call to /api/issues/contactready and github will send back a paginated JSON.
2. search issues: Take the search box input, send it to github and gets back the result to be displayed. (to be built)
3. sort issues: the list is displayed and we reorganize the order of them.

Sorting issue is orthogonal to search and filter in my understanding and it is where the difficulty is without a local dump of all issues (all of that if I'm not making a fool of myself 🌵 living dangerously).

Let's say you displayed all the untriaged. Untriaged is a reduced version of the paginated all issues case. It's a list with holes. It also means that if you sort them by oldest, for example, you can't fech the last paginated link because it might contain no untriaged issues at all. You could go fetch one by one links starting from the oldest until you build up the 10 links you need for displaying in the UI, but that would be consuming resources. It's one of the issues.

Or Maybe for untriaged we need to use github search all the time.

I wonder if we have similar issues for the contactready/sitewait/etc cases. Need to think about it.

Also background thought, I wonder if it would be expensive to store all of that on the client side in one storage bucket so we could play with the data. Like how long does it take to fetch the 8 current pages of all issues compared to all the subsequent requests to create the searches, etc. :/ I guess it depends on the type of user.

[miketaylr]

Hmm, apparently my comment I wrote a while back (yesterday?) didn't stick.

Or Maybe for untriaged we need to use github search all the time.

I think so.

I wonder if we have similar issues for the contactready/sitewait/etc cases. Need to think about it.

Shouldn't be a problem. We can ask the Issues API to only return (paginated) issues filtered by a single label.

I wonder if it would be expensive to store all of that on the client side in one storage bucket so we could play with the data.

Would be a cool experiment. I considered this a few months back, https://github.com/pamelafox/lscache

[miketaylr]

OK, coming back to this after a week or travel and hacking on other things. Question for @calexity and @magsout:

If you check out staging, when you load the page you get all open issues loaded into the issues view. If you click on a filter button (untriaged, ready for outreach, etc. buttons) it will re-render the issues view with those categories.

How does the user get back to "all"? Should we click on the selected button to toggle it off and go back that way? Right now it only toggles if you click a different category.

[miketaylr]

(note: untriaged doesn't work yet--need to hook that up via the search API tomorrow).

[calexity]

Great question @miketaylr! It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

I was thinking you would just select (or click) View all open issues again and it would reset, but that might be more trouble than it's worth for a user.

[miketaylr]

It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

I like that suggestion. I think for now I'm going to add the "toggle" behavior, and we can tweak it after we launch.

[magsout]

@calexity @miketaylr

Great question @miketaylr! It actually might make sense to have all the filters selected by default and then you would deselect any you didn't want to see.

yes, I see it like that too

[miketaylr]

Just deployed arbitary search over issues to http://staging.webcompat.com/issues

Next up: sorting, etc.

[magsout]

@miketaylr 👍

[magsout]

@miketaylr You think it is possible to automatically search after typing ~3 letters. And just as like to reset automatically when the inuput empty?

[miketaylr]

@magsout I was thinking about that... it's also related to the question of needing a "search" button. Right now it just searches on "enter"/submit/whatever. I think it wouldn't be an issue for logged in users, but for anonymous people it will be too many requests for @webcompat-bot. The Search API rate limit is very limited (docs say 20 searches a minute, HTTP response says 30 a minute).

Thinking out loud... would it be super terrible to disable search for non-logged in users and have a "Log in to search" placeholder?

[miketaylr]

Ping @calexity for thoughts on previous comment about restricting search to logged in users. For comparison, we do require users to log in to set labels or comment on and close issues.

[calexity]

I'd rather have search open but if it's going to cause an issue with
Github, its fine to say login with github to search.

Also, I forgot to call out that I used "Reported Bugs" because we use the
term bug on the homepage. I'd like to replace using "issues" with "bugs "
for clarity and consistency. Would also need to replace Browse issues on
home with Reported Bugs.
On Oct 24, 2014 8:37 AM, "Mike Taylor" [email protected] wrote:

Ping @calexity https://github.com/calexity for thoughts on previous
comment about restricting search to logged in users. For comparison, we do
require users to log in to set labels or comment on and close issues.

—
Reply to this email directly or view it on GitHub
#266 (comment)
.

[miketaylr]

Thanks for the feedback Alexa. The other option is to just be smart about rate limits and show a message if we do run into them saying "hang on for N seconds before searching again." Or something. :) We might ship the first version requiring login, then work our way up to it, depending on how much effort it would be (I think @karlcow put in most of the plumbing for that already).

[calexity]

Excellent - a limit would be a solid solution. A regular user (not a bot)
would only submit a few search requests at a time.

On Fri, Oct 24, 2014 at 1:09 PM, Mike Taylor [email protected]
wrote:

Thanks for the feedback Alexa. The other option is to just be smart about
rate limits and show a message if we do run into them saying "hang on for N
seconds before searching again." Or something. :) We might ship the first
version requiring login, then work our way up to it, depending on how much
effort it would be (I think @Karlpro put in most of the plumbing for that
already).

—
Reply to this email directly or view it on GitHub
#266 (comment)
.

[miketaylr]

Related ticket: #309

[miketaylr]

Need to investigate: 27482c7#commitcomment-8359530

[miketaylr]

Oh, maybe https://twitter.com/githubstatus/status/527583627123912704.

We're actively mitigating a large DDoS attack.

Just don't tell @karlcow. ^_^

[miketaylr]

Oh, maybe https://twitter.com/githubstatus/status/527583627123912704.

Nope, just me writing bugs. Fixed now.

[miketaylr]

OK, #312 is out the door. Will file smaller bugs to track enabling the rest of the features. We should probably figure out our milestone process--seems like it would be useful to help us ship the rest of this feature.

[magsout]

We should probably figure out our milestone process--seems like it would be useful to help us ship the rest of this feature.

yes.. agree

[miketaylr]

I've created a Milestone "Ship full issues page" to track the smaller parts.

[miketaylr]

But let's close this issue. The milestone will serve as the "meta".