'read_csv' returns wrong number of records

[edorid]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

I did comparison with CSV python standard library, and the number of records read by polars was not matched.
The file attached is actually database dump, but I already obscure it. But the output still the same.

issue.csv

Reproducible example

import polars as pl
import csv

if __name__ == "__main__":
    with open("issue.csv", 'r') as f:
        reader = csv.reader(f)
        lines = 0
        for row in reader:
            lines += 1
    
    print("Lines: %s" % lines)

    db_df = pl.read_csv("issue.csv")
    print("Lines: %s" % len(db_df.rows()))

Output:

Lines: 5
Lines: 1334

### Expected behavior

The number of records should be the same (records=5).

### Installed versions

<details>

---Version info---
Polars: 0.16.4
Index type: UInt32
Platform: Linux-5.15.0-60-generic-x86_64-with
Python: 3.9.16 (main, Dec 10 2022, 13:47:19)
[GCC 10.3.1 20210424]
---Optional dependencies---
pyarrow:
pandas:
numpy:
fsspec:
connectorx:
xlsx2csv:
deltalake:
matplotlib: ```

[alexander-beedie]

Thanks for the bug report; just to clarify, from the obfuscated data it appears that you are loading logged/multi-line SQL queries?

As a temporary workaround, you could do the following to get the data into polars:

with open( "issue.csv", "r" ) as f:
    iter_csv = csv.reader( f )
    columns = next( iter_csv )

    df = pl.DataFrame( 
        data = iter_csv, 
        schema = columns,
    )
    # ┌──────────┬────────────────────┬─────────────────────┐
    # │ type     ┆ dbname             ┆ dump                │
    # │ ---      ┆ ---                ┆ ---                 │
    # │ str      ┆ str                ┆ str                 │
    # ╞══════════╪════════════════════╪═════════════════════╡
    # │ database ┆ connections-prod   ┆ --                  │
    # │          ┆                    ┆ x x x.x_x_x_x_x_x x │
    # │          ┆                    ┆  x x.x,             │
    # │          ┆                    ┆  ...                │
    # │ database ┆ content-prod       ┆ --                  │
    # │          ┆                    ┆ -- x x x            │
    # │          ┆                    ┆ --                  │
    # │          ┆                    ┆                     │
    # │          ┆                    ┆ -- x x x x 11.16... │
    # │ database ┆ notifications-prod ┆ --                  │
    # │          ┆                    ┆ -- x x x            │
    # │          ┆                    ┆ --                  │
    # │          ┆                    ┆                     │
    # │          ┆                    ┆ -- x x x x 11.16... │
    # │ database ┆ users-prod         ┆ --                  │
    # │          ┆                    ┆ -- x x x            │
    # │          ┆                    ┆ --                  │
    # │          ┆                    ┆                     │
    # │          ┆                    ┆ -- x x x x 11.16... │
    # └──────────┴────────────────────┴─────────────────────┘

[edorid]

yes, it's multi-line SQL queries generated from pg_dumpall

[ritchie46]

Right, polars succeeds if we set max_threads to 1. So we have trouble finding a valid point to split up the file. I will look into this.

This does work:

pl.read_csv("/home/ritchie46/Downloads/issue.csv", n_threads=1)