Skip to content

Commit

Permalink
Initialize stream writer support
Browse files Browse the repository at this point in the history
- Add 4 new functions: new_stream_writer, add_table, set_row and flush
- Update unit tests and docs for the function
- Upgrade the dependencies package version
- Using form template for GitHub issues
  • Loading branch information
xuri committed Jan 7, 2025
1 parent c7199a6 commit 4d0e021
Show file tree
Hide file tree
Showing 12 changed files with 436 additions and 125 deletions.
50 changes: 0 additions & 50 deletions .github/ISSUE_TEMPLATE/bug_report.md

This file was deleted.

81 changes: 81 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Bug report
description: Create a report to help us improve
body:
- type: markdown
attributes:
value: |
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
- type: textarea
id: description
attributes:
label: Description
description: Briefly describe the problem you are having in a few paragraphs.
validations:
required: true

- type: textarea
id: reproduction-steps
attributes:
label: Steps to reproduce the issue
description: Explain how to cause the issue in the provided reproduction.
placeholder: |
1.
2.
3.
validations:
required: true

- type: textarea
id: received
attributes:
label: Describe the results you received
validations:
required: true

- type: textarea
id: expected
attributes:
label: Describe the results you expected
validations:
required: true

- type: input
id: py-version
attributes:
label: Python version
description: |
Output of `python --version`:
placeholder: e.g. 3.8.10
validations:
required: true

- type: input
id: excelize-version
attributes:
label: Excelize version or commit ID
description: |
Which version of Excelize are you using?
placeholder: e.g. 0.0.1
validations:
required: true

- type: textarea
id: env
attributes:
label: Environment
description: Environment details (OS, Microsoft Excel™ version, physical, etc.)
render: shell
validations:
required: true

- type: checkboxes
id: checkboxes
attributes:
label: Validations
description: Before submitting the issue, please make sure you do the following
options:
- label: Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
required: true
- label: The provided reproduction is a minimal reproducible example of the bug.
required: true
50 changes: 0 additions & 50 deletions .github/ISSUE_TEMPLATE/feature_request.md

This file was deleted.

30 changes: 30 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: Feature request
description: Suggest an idea for this project
body:
- type: markdown
attributes:
value: |
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
- type: textarea
id: description
attributes:
label: Description
description: Describe the feature that you would like added
validations:
required: true

- type: textarea
id: additional-context
attributes:
label: Additional context
description: Any other context or screenshots about the feature request here?

- type: checkboxes
id: checkboxes
attributes:
label: Validations
description: Before submitting the issue, please make sure you do the following
options:
- label: Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.
required: true
140 changes: 139 additions & 1 deletion excelize.py
Original file line number Diff line number Diff line change
Expand Up @@ -439,10 +439,89 @@ def py_value_to_c_interface(py_value):
return py_value_to_c(interface, types_go._Interface())


class StreamWriter:
sw_index: int

def __init__(self, sw_index: int):
self.sw_index = sw_index

def add_table(self, table: Table) -> Optional[Exception]:
"""
Creates an Excel table for the stream writer using the given cell range
and format set.
Note that the table must be at least two lines including the header. The
header cells must contain strings and must be unique. Currently, only
one table is allowed for a stream writer. The function must be called
after the rows are written but before 'flush'.
Args:
table (Table): The table options
Returns:
Optional[Exception]: Returns None if no error occurred,
otherwise returns an Exception with the message.
Example:
For example, create a table of A1:D5 on Sheet1:
.. code-block:: python
err = sw.add_table(excelize.Table(range="A1:D5"))
"""
lib.StreamAddTable.restype = c_char_p
options = py_value_to_c(table, types_go._Table())
err = lib.StreamAddTable(self.sw_index, byref(options)).decode(ENCODE)
return None if err == "" else Exception(err)

def set_row(
self,
cell: str,
values: List[Union[None, int, str, bool, datetime, date]],
) -> Optional[Exception]:
"""
Writes an array to stream rows by giving starting cell reference and a
pointer to an array of values. Note that you must call the 'flush'
function to end the streaming writing process.
Args:
cell (str): The cell reference
values (List[Union[None, int, str, bool, datetime, date]]): The cell
values
Returns:
Optional[Exception]: Returns None if no error occurred,
otherwise returns an Exception with the message.
"""
lib.StreamSetRow.restype = c_char_p
vals = (types_go._Interface * len(values))()
for i, value in enumerate(values):
vals[i] = py_value_to_c_interface(value)
err = lib.StreamSetRow(
self.sw_index,
cell.encode(ENCODE),
byref(vals),
len(vals),
).decode(ENCODE)
return None if err == "" else Exception(err)

def flush(self) -> Optional[Exception]:
"""
Ending the streaming writing process.
Returns:
Optional[Exception]: Returns None if no error occurred,
otherwise returns an Exception with the message.
"""
lib.StreamFlush.restype = c_char_p
err = lib.StreamFlush(self.sw_index).decode(ENCODE)
return None if err == "" else Exception(err)


class File:
file_index: int

def __init__(self, file_index):
def __init__(self, file_index: int):
self.file_index = file_index

def save(self, *opts: Options) -> Optional[Exception]:
Expand Down Expand Up @@ -1660,6 +1739,65 @@ def new_sheet(self, sheet: str) -> Tuple[int, Optional[Exception]]:
err = res.err.decode(ENCODE)
return res.val, None if err == "" else Exception(err)

def new_stream_writer(
self, sheet: str
) -> Tuple[Optional[StreamWriter], Optional[Exception]]:
"""
Returns stream writer struct by given worksheet name used for writing
data on a new existing empty worksheet with large amounts of data. Note
that after writing data with the stream writer for the worksheet, you
must call the 'flush' method to end the streaming writing process,
ensure that the order of row numbers is ascending when set rows, and the
normal mode functions and stream mode functions can not be work mixed to
writing data on the worksheets. The stream writer will try to use
temporary files on disk to reduce the memory usage when in-memory chunks
data over 16MB, and you can't get cell value at this time.
Args:
sheet (str): The worksheet name
Returns:
Tuple[Optional[StreamWriter], Optional[Exception]]: A tuple
containing stream writer object if successful, or None and an
Exception if an error occurred.
Example:
For example, set data for worksheet of size 102400 rows x 50 columns
with numbers:
.. code-block:: python
import excelize, random
f = excelize.new_file()
sw, err = f.new_stream_writer("Sheet1")
if err:
print(err)
for r in range(2, 102401):
row = [random.randrange(640000) for _ in range(1, 51)]
cell, err = excelize.coordinates_to_cell_name(1, r, False)
if err:
print(err)
err = sw.set_row(cell, row)
if err:
print(err)
err = sw.flush()
if err:
print(err)
err = f.save_as("Book1.xlsx")
if err:
print(err)
err = f.close()
if err:
print(err)
"""
lib.NewStreamWriter.restype = types_go._IntErrorResult
res = lib.NewStreamWriter(self.file_index, sheet.encode(ENCODE))
err = res.err.decode(ENCODE)
if err == "":
return StreamWriter(res.val), None
return None, Exception(err)

def new_style(self, style: Style) -> Tuple[int, Optional[Exception]]:
"""
Create the style for cells by a given style options, and returns style
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/xuri/excelize-py
go 1.20

require (
github.com/xuri/excelize/v2 v2.9.1-0.20241229043028-3f6ecffcca89
github.com/xuri/excelize/v2 v2.9.1-0.20250105013731-af422e17009b
golang.org/x/image v0.23.0
)

Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ github.com/tiendc/go-deepcopy v1.2.0 h1:6vCCs+qdLQHzFqY1fcPirsAWOmrLbuccilfp8UzD
github.com/tiendc/go-deepcopy v1.2.0/go.mod h1:toXoeQoUqXOOS/X4sKuiAoSk6elIdqc0pN7MTgOOo2I=
github.com/xuri/efp v0.0.0-20241211021726-c4e992084aa6 h1:8m6DWBG+dlFNbx5ynvrE7NgI+Y7OlZVMVTpayoW+rCc=
github.com/xuri/efp v0.0.0-20241211021726-c4e992084aa6/go.mod h1:ybY/Jr0T0GTCnYjKqmdwxyxn2BQf2RcQIIvex5QldPI=
github.com/xuri/excelize/v2 v2.9.1-0.20241229043028-3f6ecffcca89 h1:KlbACs1A0q8+TJ3keiTh6ir/L0HBFwNe416d1c+dez4=
github.com/xuri/excelize/v2 v2.9.1-0.20241229043028-3f6ecffcca89/go.mod h1:NBRx6e5FHFx4mHLiYG1QBONNvNNSs/wrtzS+h56/A6k=
github.com/xuri/excelize/v2 v2.9.1-0.20250105013731-af422e17009b h1:IgzLFLfCA10Lu5Ac9XgC8wBvPAVqHe2ilgIA8nouiPc=
github.com/xuri/excelize/v2 v2.9.1-0.20250105013731-af422e17009b/go.mod h1:NBRx6e5FHFx4mHLiYG1QBONNvNNSs/wrtzS+h56/A6k=
github.com/xuri/nfp v0.0.0-20240318013403-ab9948c2c4a7 h1:hPVCafDV85blFTabnqKgNhDCkJX25eik94Si9cTER4A=
github.com/xuri/nfp v0.0.0-20240318013403-ab9948c2c4a7/go.mod h1:WwHg+CVyzlv/TX9xqBFXEZAuxOPxn2k1GNHwG41IIUQ=
golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
Expand Down
Loading

0 comments on commit 4d0e021

Please sign in to comment.