Skip to content
This repository has been archived by the owner on Feb 22, 2020. It is now read-only.

Commit

Permalink
fix(preprocess): fix offset in sentence splitter
Browse files Browse the repository at this point in the history
  • Loading branch information
hanhxiao committed Sep 2, 2019
1 parent d21a6a1 commit d94329b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion gnes/preprocessor/text/split.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,6 @@ def apply(self, doc: 'gnes_pb2.Document') -> None:
c = doc.chunks.add()
c.doc_id = doc.doc_id
c.text = f[:self.max_sent_len]
c.offset_1d = ci
c.offset = ci
c.weight = len(c.text) / len(doc.raw_text)
c.offset_nd.extend([s, e])

0 comments on commit d94329b

Please sign in to comment.