newscat provides fast and accurate news content / article extraction. It is trained to extract the contents of news articles, while excluding clutter like image and video captions, editorial side notes, related content, teasers, advertisements, user comments and meta data.
To download newscat, run
go get github.com/slyrz/newscat
This will download the source code of newscat and, if not present,
newscat's only non-standard build dependency - the html package from the
go.net networking libraries.
Then run
go build github.com/slyrz/newscat
to build newscat. This should produce a newscat binary file in your
$GOPATH/bin directory.
newscat accepts file paths and HTTP URLs as command line arguments.
newscat [PATH|URL]...
If no arguments are passed, newscat expects HTML written to its standard input.
newscat < PATH
It prints the extracted article text to standard output. If you want
properly formatted paragraphs, pipe newscat's output to the fmt command.
newscat ... | fmt
300 news articles were gathered by crawling top submissions from various news-related Reddit communities. A golden standard was created for every HTML page by manually adding a custom HTML5 data attribute to the elements containing relevant content.
The data set was split into training and test data. 50 randomly chosen news articles were used to train newscat. The remaining 250 news articles were used to evaluate the content extraction quality.
For each news article, we compared the element-level predictions with the real labels and calculated precision, recall and the balanced F-score.
The above figure shows the resulting F-scores ordered in increasing magnitude. The x-axis shows the percentage of articles whose F-scores fall below the value indicated by the y-axis. In other words: the percentiles.
newscat is released under MIT license. You can find a copy of the MIT License in the LICENSE file.