Skip to content
This repository has been archived by the owner on Aug 20, 2024. It is now read-only.

Add BufferedCustomFileEmission #2334

Merged
merged 1 commit into from
Aug 21, 2021
Merged

Add BufferedCustomFileEmission #2334

merged 1 commit into from
Aug 21, 2021

Conversation

jackkoenig
Copy link
Contributor

Uses virtual method .getBytesBuffered: Iterable[Array[Byte]] to optimize
file emission.

This is intended to enable faster annotation serialization. In particular this will be used to make .fir emission from Chisel support > 2 GiB .fir files. Currently. Chisel serializes the .fir to a String which has a max size of 2 GiB. We could use the CustomFileEmission API but it is 9-12x slower using just Iterable[Byte] so this PR is intended to make it much faster via buffering.

Contributor Checklist

  • Did you add Scaladoc to every public function/method?
  • [NA] Did you update the FIRRTL spec to include every new feature/behavior?
  • Did you add at least one test demonstrating the PR?
  • Did you delete any extraneous printlns/debugging code?
  • Did you specify the type of improvement?
  • Did you state the API impact?
  • Did you specify the code generation impact?
  • Did you request a desired merge strategy?
  • Did you add text to be included in the Release Notes for this change?

Type of Improvement

  • performance improvement
  • new feature/API

API Impact

Expands the API with a new subclass of CustomFileEmission called BufferedCustomFileEmission

Backend Code Generation Impact

No impact

Desired Merge Strategy

  • Squash

Release Notes

Introduce BufferedCustomFileEmission which enables Annotation writers to optimize serialization performance for their annotations.

Reviewer Checklist (only modified by reviewer)

  • Did you add the appropriate labels?
  • Did you mark the proper milestone (1.2.x, 1.3.0, 1.4.0) ?
  • Did you review?
  • Did you check whether all relevant Contributor checkboxes have been checked?
  • Did you mark as Please Merge?

@jackkoenig jackkoenig requested a review from seldridge August 20, 2021 23:53
@jackkoenig
Copy link
Contributor Author

I'm working on benchmarking serialization using this, will report back once I have numbers.

Uses virtual method .getBytesBuffered: Iterable[Array[Byte]] to optimize
file emission.
@jackkoenig jackkoenig force-pushed the bufferedcustomfileemission branch from 13ee399 to 5494e66 Compare August 20, 2021 23:58
Copy link
Member

@seldridge seldridge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@jackkoenig jackkoenig added this to the 1.4.x milestone Aug 21, 2021
@jackkoenig jackkoenig added the Please Merge Accepted PRs that are ready to be merged. Useful when waiting on CI. label Aug 21, 2021
@mergify mergify bot merged commit dc2fbad into master Aug 21, 2021
mergify bot pushed a commit that referenced this pull request Aug 21, 2021
Uses virtual method .getBytesBuffered: Iterable[Array[Byte]] to optimize
file emission.

(cherry picked from commit dc2fbad)
@mergify mergify bot added the Backported This PR has been backported to marked stable branch label Aug 21, 2021
@jackkoenig
Copy link
Contributor Author

I'm working on benchmarking serialization using this, will report back once I have numbers.

The numbers look great, completely erases the 9-12x slowdown and actually for the specific case of emitting .fir from Chisel3, it appears to make emission faster over the Chisel 3 built-in emitter. See PR to follow in Chisel 3 using this new API.

@jackkoenig jackkoenig deleted the bufferedcustomfileemission branch August 21, 2021 00:20
mergify bot added a commit that referenced this pull request Aug 21, 2021
Uses virtual method .getBytesBuffered: Iterable[Array[Byte]] to optimize
file emission.

(cherry picked from commit dc2fbad)

Co-authored-by: Jack Koenig <[email protected]>
@mwachs5
Copy link
Contributor

mwachs5 commented Aug 21, 2021

Awesome!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Backported This PR has been backported to marked stable branch Please Merge Accepted PRs that are ready to be merged. Useful when waiting on CI.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants