Code

Some taken some given#202

Code

Open

HjalmarrSv wants to merge 110 commits intoattardi:masterattardi/wikiextractor:masterfrom HjalmarrSv:masterHjalmarrSv/wikiextractor:masterCopy head branch name to clipboard

+794-183

Commits

Commits on Dec 13, 2019

• Compacted Headers

  Show description for e2d332e
  HjalmarrSv

  authored
  e2d332e

• Update WikiExtractor.py

  HjalmarrSv

  authored
  053fde3

• Update WikiExtractor.py

  HjalmarrSv

  authored
  59e597c

• Update WikiExtractor.py

  HjalmarrSv

  authored
  5870040

• fixed added errors

  HjalmarrSv

  authored
  d2e6330

• works but errors may exist as well as quirks

  Show description for eeda954
  HjalmarrSv

  authored
  eeda954

• Added Example

  HjalmarrSv

  authored
  8d10750

• Added 2 examples

  HjalmarrSv

  authored
  3b6a710

Commits on Dec 14, 2019

• -- spacefree now also removes line = " "

  Show description for ef9c07a
  HjalmarrSv

  authored
  ef9c07a

Commits on Dec 20, 2019

• Changed --spacefree to --squeeze-blank (same as in cat --squeeze-blank ) + magic word

  Show description for 7b2abd4
  HjalmarrSv

  authored
  7b2abd4

Commits on Dec 23, 2019

• Update

  Show description for de81a98
  HjalmarrSv

  authored
  de81a98

• Update README.md

  HjalmarrSv

  authored
  866e073

Commits on Dec 28, 2019

• --no-title as alias for --titlefree

  Show description for 22da7ea
  HjalmarrSv

  authored
  22da7ea

• --no-title should work as --titlefree

  HjalmarrSv

  authored
  8378de6

• --no-templates as alias for --no_templates

  Show description for dc56429
  HjalmarrSv

  authored
  dc56429

• temporarily change --no-title to -no-title

  Show description for 7198d0a
  HjalmarrSv

  authored
  7198d0a

Commits on Dec 29, 2019

• Update WikiExtractor.py

  Show description for 67a0ce7
  HjalmarrSv

  authored
  67a0ce7

• About cleaning

  HjalmarrSv

  authored
  49ad988

• Update README.md

  HjalmarrSv

  authored
  bb68ed7

• Update README.md

  HjalmarrSv

  authored
  4f192ed

Commits on Dec 31, 2019

• spelling

  Show description for d2ce1a8
  HjalmarrSv

  authored
  d2ce1a8

• Added mediawiki links in comments

  HjalmarrSv

  authored
  ea31c41

Commits on Jan 3, 2020

• Implement josecannete/wikiextractorforBERT

  Show description for 67903ab
  HjalmarrSv

  authored
  67903ab

• Fix: separate docs

  HjalmarrSv

  authored
  a32e7b1

• --for-bert

  Show description for 3325153
  HjalmarrSv

  authored
  3325153

Commits on Jan 4, 2020

• Added better example

  Show description for 8bed747
  HjalmarrSv

  authored
  8bed747

• Update README.md

  HjalmarrSv

  authored
  b4420f3

• Update README.md

  HjalmarrSv

  authored
  703249a

• Concatenate many files to one file

  Show description for 58d4352
  HjalmarrSv

  authored
  58d4352

• Template expansion

  Show description for bfda827
  HjalmarrSv

  authored
  bfda827

• broken middoc interwiki link

  Show description for d7eb406
  HjalmarrSv

  authored
  d7eb406

• Augmented key regex to catch plus/minus signs

  Show description for a4e4fdc
  HjalmarrSv

  authored
  a4e4fdc

• New example for Bert and json

  Show description for 82f6d3c
  HjalmarrSv

  authored
  82f6d3c

• Force 'utf-8' encoding

  Show description for 5ae4bcd
  HjalmarrSv

  authored
  5ae4bcd

• Remove inline flags from the middle of a regex

  Show description for c6b5d39
  HjalmarrSv

  authored
  c6b5d39

• Update README.md

  HjalmarrSv

  authored
  41fdefc

Commits on Jan 5, 2020

• Option to restrict to specific pages by title

  Show description for b2c4a47
  HjalmarrSv

  authored
  b2c4a47

Commits on Jan 6, 2020

• --max_articles

  Show description for 2f80d1c
  HjalmarrSv

  authored
  2f80d1c

• bug fixes

  HjalmarrSv

  authored
  9683248

• max_articles, remove-html-tags in tested example

  Show description for 761f67b
  HjalmarrSv

  authored
  761f67b

• remove mapframe, maplink

  Show description for aef4bc8
  HjalmarrSv

  authored
  aef4bc8

• discard score tags

  Show description for 7c37e62
  HjalmarrSv

  authored
  7c37e62

Commits on Jan 7, 2020

• cgi.escape with html.escape

  Show description for fe2b6f2
  HjalmarrSv

  authored
  fe2b6f2

• Handle broken pipe and keyboard interrupts

  Show description for 02757c6
  HjalmarrSv

  authored
  02757c6

• fixed the last introduced bug

  HjalmarrSv

  authored
  050b586

• Rollback of code for termination

  Show description for a63ab3a
  HjalmarrSv

  authored
  a63ab3a

Commits on Jan 8, 2020

• --verbose and <BR>

  Show description for 0b9091b
  HjalmarrSv

  authored
  0b9091b

• bug fix + a bit of removing of debug comments

  HjalmarrSv

  authored
  ea73ede

• Update README.md

  HjalmarrSv

  authored
  e190f33

• Update README.md

  HjalmarrSv

  authored
  5ab7485

Commits on Jan 9, 2020

• Update README.md

  HjalmarrSv

  authored
  48dc351

• templates_only option and dropNested within and between ref tags

  Show description for ff7cf4d
  HjalmarrSv

  authored
  ff7cf4d

Commits on Jan 12, 2020

• currently no install

  HjalmarrSv

  authored
  3255fae

Commits on Jan 16, 2020

• --raw, --abstract_only

  Show description for eaf3d34
  HjalmarrSv

  authored
  eaf3d34

• Update README.md

  HjalmarrSv

  authored
  6088107

• bugfixes

  HjalmarrSv

  authored
  d831277

Commits on Jan 19, 2020

• fixed breaking errors

  Show description for a3cc036
  HjalmarrSv

  authored
  a3cc036

• Example on cirrus dump

  HjalmarrSv

  authored
  7047392

• text only option

  HjalmarrSv

  authored
  de50af0

• Example on text only

  HjalmarrSv

  authored
  7b8a9d1

Commits on Jan 20, 2020

• Remove code making gzip conditional

  Show description for e399a97
  HjalmarrSv

  authored
  e399a97

• Add urlbase to verbosity

  Show description for e3657c1
  HjalmarrSv

  authored
  e3657c1

• only one space before caret

  Show description for d875198
  HjalmarrSv

  authored
  d875198

Commits on Jan 21, 2020

• remove broken sentences

  Show description for c4321d0
  HjalmarrSv

  authored
  c4321d0

Commits on Jan 23, 2020

• --sentences and --raw

  Show description for 8a78aea
  HjalmarrSv

  authored
  8a78aea

• fix tab error

  Show description for 24e6e85
  HjalmarrSv

  authored
  24e6e85

• --sentences, now at least 2 per article

  Show description for 6c64e75
  HjalmarrSv

  authored
  6c64e75

Commits on Jan 24, 2020

• Update README.md

  HjalmarrSv

  authored
  00a322e

• bug fix

  HjalmarrSv

  authored
  8bb7497

Commits on Jan 25, 2020

• Make visible what is not supported currently

  Show description for 3b8bb70
  HjalmarrSv

  authored
  3b8bb70

• Added switch: '__NOGLOBAL__'

  HjalmarrSv

  authored
  6d755bf

Commits on Jan 27, 2020

• Added {{grammar: ...}}

  Show description for 49e3cd8
  HjalmarrSv

  authored
  49e3cd8

• disambiguate command line parameter

  HjalmarrSv

  authored
  34ee0dc

• Update WikiExtractor.py

  HjalmarrSv

  authored
  fd1c7ca

• revert on 'small'

  Show description for bbdb4a8
  HjalmarrSv

  authored
  bbdb4a8

Commits on Jan 28, 2020

• basic support for 'formatnum'

  Show description for 62d2b06
  HjalmarrSv

  authored
  62d2b06

• added example

  HjalmarrSv

  authored
  7ccd3f8

• refresh comments

  HjalmarrSv

  authored
  63278c7

Commits on Jan 29, 2020

• --decimalcomma; decimal separator comma (,) for formatnum

  Show description for 0950b89
  HjalmarrSv

  authored
  0950b89

• bug fix

  HjalmarrSv

  authored
  8910474

• a little better formatnum

  Show description for a729546
  HjalmarrSv

  authored
  a729546

• version update

  Show description for fbb1199
  HjalmarrSv

  authored
  fbb1199

• clean up templatestyles

  Show description for effcc94
  HjalmarrSv

  authored
  effcc94

Commits on Feb 2, 2020

• preparing for #dateformat tags

  HjalmarrSv

  authored
  063d3d1

• clean '! ...' - lines

  Show description for 40bd592
  HjalmarrSv

  authored
  40bd592

• this way it actually works

  Show description for c235357
  HjalmarrSv

  authored
  c235357

• set 'options.cleaned = True'

  Show description for 970f94b
  HjalmarrSv

  authored
  970f94b

• Additional cleaning needed

  HjalmarrSv

  authored
  7a824b8

Commits on Feb 3, 2020

• improve cleaning

  HjalmarrSv

  authored
  56d9852

• basic formatdate and dateformat parsing

  HjalmarrSv

  authored
  43a9b9d

• remove multiples of '/n'

  Show description for c4136d0
  HjalmarrSv

  authored
  c4136d0

• remove empty ( )

  Show description for d90b9e7
  HjalmarrSv

  authored
  d90b9e7

Commits on Feb 5, 2020

• Update comments in code

  HjalmarrSv

  authored
  9c7d502

Commits on Feb 6, 2020

• use raw, especially where there are \

  Show description for 01668eb
  HjalmarrSv

  authored
  01668eb

• expand formatnum functionality

  Show description for 3f672b0
  HjalmarrSv

  authored
  3f672b0

• the obvious bugs

  Show description for 1b2a695
  HjalmarrSv

  authored
  1b2a695

• too buggy - works when function shortcut, else not

  Show description for e974947
  HjalmarrSv

  authored
  e974947

Commits on Feb 7, 2020

• Update comments: Lua wants!

  HjalmarrSv

  authored
  3133906

Commits on Feb 28, 2020

• fix for revid

  Show description for d0f2529
  HjalmarrSv

  authored
  d0f2529

Commits on Feb 29, 2020

• creates article files instead

  Show description for 4a4650d
  HjalmarrSv

  authored
  4a4650d

• Update README.md

  HjalmarrSv

  authored
  392fddf

• Adding comments

  HjalmarrSv

  authored
  bc781f2

Commits on Mar 2, 2020

• If / gives problems in an os that uses \ in dir path

  Show description for 86adb42
  HjalmarrSv

  authored
  86adb42

Commits on Mar 7, 2020

• fix for en, new folder structure

  Show description for 1f89728
  HjalmarrSv

  authored
  1f89728

• Update README.md

  HjalmarrSv

  authored
  69d0e76

Commits on Mar 8, 2020

• Updated

  Show description for 1f70817
  HjalmarrSv

  authored
  1f70817

• Update README.md

  HjalmarrSv

  authored
  f3a1892

Commits on Mar 13, 2020

• Update cirrus-extract.py

  HjalmarrSv

  authored
  0654bab

Commits on May 19, 2020

• Update cirrus-extract.py

  Show description for fbeef46
  HjalmarrSv

  authored
  fbeef46

Commits on Oct 13, 2021

• Update README.md

  HjalmarrSv

  authored
  c423d2d