Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Several projection improvements #2795

Merged
merged 4 commits into from
Feb 28, 2022
Merged

Several projection improvements #2795

merged 4 commits into from
Feb 28, 2022

Conversation

ritchie46
Copy link
Member

@ritchie46 ritchie46 commented Feb 28, 2022

The lazy computation graph has schema for every node. This PR makes sure we actually use it and therefore save a lot column name lookups.

Test run

use polars::prelude::*;

fn main() -> Result<()> {
    let cols = (0..100000)
        .map(|i| Series::new(&format!("{i}"), [i]))
        .collect();
    let df = DataFrame::new_no_checks(cols);
    dbg!("created df");
    let names = df.get_column_names();
    dbg!("created names");

    let exprs: Vec<_> = (0..1000).map(|i| col(names[names.len() - 1 - i])).collect();

    for i in 0..10 {
        dbg!(i);
        df.clone().lazy().select(exprs.clone()).collect().unwrap();
    }

    Ok(())
}

Master

         38.066,31 msec task-clock                #   10,185 CPUs utilized          
             9.725      context-switches          #  255,475 /sec                   
                85      cpu-migrations            #    2,233 /sec                   
            17.976      page-faults               #  472,229 /sec                   
   117.932.283.930      cycles                    #    3,098 GHz                    
    43.582.787.522      instructions              #    0,37  insn per cycle         
    12.368.895.419      branches                  #  324,930 M/sec                  
         1.413.148      branch-misses             #    0,01% of all branches        

       3,737638335 seconds time elapsed

      37,864670000 seconds user
       0,181020000 seconds sys

This PR

            551,74 msec task-clock                #    1,040 CPUs utilized          
               569      context-switches          #    1,031 K/sec                  
                65      cpu-migrations            #  117,810 /sec                   
            17.982      page-faults               #   32,592 K/sec                  
     1.705.472.128      cycles                    #    3,091 GHz                    
     2.583.767.201      instructions              #    1,51  insn per cycle         
       497.490.863      branches                  #  901,681 M/sec                  
           436.293      branch-misses             #    0,09% of all branches        

       0,530435470 seconds time elapsed

       0,496682000 seconds user
       0,059751000 seconds sys

@ritchie46
Copy link
Member Author

@ghuls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant