Speedup df.schema.

[ghuls]

No description provided.

[ghuls]

This fixes very slow df.schema with 1M columns (now runs in 1 second) #2755

[ritchie46]

Yes, much better. :)

[alexander-beedie]

FYI: this is even faster still (~25% or so), as it avoids temporary assignment into "c" and "dtype" ;)

return dict( zip(self.columns, self.dtypes) )

(The difference between the two calls is quite notable if you look at the output after disassembly via dis)

[ghuls]

In [24]: %time columns = dfs.columns
CPU times: user 0 ns, sys: 170 ms, total: 170 ms
Wall time: 169 ms

In [25]: %time dtypes = dfs.dtypes
CPU times: user 257 ms, sys: 451 ms, total: 709 ms
Wall time: 705 ms

In [26]: %time schema = dfs.schema
CPU times: user 905 ms, sys: 159 ms, total: 1.06 s
Wall time: 1.06 s

In [28]: %time s1 = { c: d for c, d in  zip(columns, dtypes) }
CPU times: user 245 ms, sys: 11.2 ms, total: 256 ms
Wall time: 254 ms

In [29]: %time s2 = dict(zip(columns, dtypes))
CPU times: user 174 ms, sys: 1.21 ms, total: 175 ms
Wall time: 174 ms

Getting dtypes is relatively slow compared wtih just getting column names.

[ritchie46]

This is a good improvement for now. We can further improve by creating the schema on the rust bindings side. Then we don't have to allocate to lists/vecs and create a dict directly.