Add DNMD into src/native #107961
Open
Add DNMD into src/native #107961
Commits on Oct 24, 2022
-
Static lib for reading ECMA-335 metadata Confirmed works against all .NET 6.0 BCL assemblies.
-
Commits on Oct 26, 2022
-
Create table/column index enum
Add a debugging process for validating using the correct enumeration when using an enumeration column index.
-
Portable PDB support can be enabled by building with the DNMD_PORTABLE_PDB define.
Commits on Oct 27, 2022
Commits on Oct 28, 2022
-
Still fails to compile on non-WIndows due to missing PE image data structures.
-
-
Udate public API and remove CorTokenType
Commits on Oct 30, 2022
-
Start implementation for interfaces
Builds with no obvious regressions in the mddump scenario.
Commits on Oct 31, 2022
-
-
-
Create CoTaskMemAlloc and malloc smart pointers
Implemented minimal IMetaDataDispenser
-
Commits on Nov 1, 2022
-
Add find logic for sorted tables in dnmd
Testing for several methods on IMetaDataImport
Commits on Nov 2, 2022
-
Add impl and test for EnumMembers
Add test for ResetEnum Update HCORENUM implementation to be a singly linked list to reduce allocations. Remove md_row_distance()
-
-
Commits on Nov 3, 2022
Commits on Nov 5, 2022
-
Change low level UserStrings API Add walk UserStrings heap API
Commits on Nov 7, 2022
Commits on Nov 10, 2022
Commits on Nov 11, 2022
Commits on Nov 12, 2022
-
-
-
Add project to compare perf and profile
Always generate symbols
-
Commits on Nov 14, 2022
Commits on Nov 19, 2022
-
Create simple PAL for Utf16ToUtf8 transcoding. - Only supported on Windows presently. Add build notes in readme.md
-
-
Commits on Nov 20, 2022
-
EnumMethodSemantics Bug create_query_context to compute end row properly. Reset the data to read in query context.
-
GetMemberRefProps Add a test that only runs on the corlibs. This is helpful for APIs that expensive to call.
Commits on Nov 21, 2022
-
Reduced static allocation substantially for MD load.
-
Improve performance test for expansion
Remove unnecessary checks in dnmd APIs Add GetScopeProps perf scenario Attempt to transcode immediately instead of querying for expected length first.
Commits on Nov 29, 2022
Commits on Nov 30, 2022
Commits on Dec 3, 2022
-
GetCustomAttributeByName benchmark
Performance of baseline was initially better. Remove branches and simplified functions that didn't need to construct new complex data structures so early.
-
-
Commits on Dec 11, 2022
-
Add IL project for testing Add global.json for IL SDK
Commits on Jan 10, 2023
-
Update test projects to reference native output bin Fix /W4 build for MSVC
Commits on Jan 11, 2023
Commits on Jan 13, 2023
Commits on Jan 14, 2023
Commits on Jan 17, 2023
-
-
-
-
Update managed tests to only use IMetaDataImport2 as there's no tests…
… where we just need IMetaDataImport
-
-
Commits on Jan 19, 2023
Commits on Jan 20, 2023
-
Fix some incorrect API usage with assuming 0 means invalid.
Commits on Jan 21, 2023
Commits on Jan 23, 2023
-
Add predicate to query functions to avoid
loop if out_loop is 1. Mark coded_index_map as static
Commits on Jan 27, 2023
-
Move regperf to a regnative location for shared
native test assets.
Commits on Jan 29, 2023
-
Add DNCP submodule Remove built-in COM interop .NET APIs.
Commits on Feb 1, 2023
-
-
Linux builds if utf16 <=> utf8 conversion
functions are implemented.
-
-
-
Add main CI job that builds, tests, and reports test failures as anot…
…her check entry.
Commits on Feb 3, 2023
Commits on Feb 11, 2023
-
-
Merge pull request dotnet#15 from jkoritzinsky/fix-mac-build
Set some loader and build settings to make the macos build work better.
-
Compare baseline results to current results by value instead of by po…
…inter for byte blobs (dotnet#14) * Compare baseline results to current results by value instead of by pointer for byte blobs. This will allow us to still use this same baseline/current testing methodology for our emit tests, where the baseline implementations move the data around on creation if they're in edit mode. * Use static array instead of vector when appropriate Co-authored-by: Aaron Robinson <[email protected]>
Commits on Feb 12, 2023
Commits on Feb 27, 2023
-
* Create LICENSE.md * Update readme * Ignore all changes to markdown files
Commits on Feb 28, 2023
Commits on Mar 29, 2023
-
Add support for the uncompressed stream, indirection tables, and mini…
…mal delta images (dotnet#22) * Add support for reading in the not-in-the-spec uncompressed table stream and recognize a minimal delta image to enable reading it in correctly. * Support recognizing which fields may refer to indirect table indicies. * Define indirect pointer column names * Implement handling of indirect tables at the IMetadataImport layer, with one small helper function in DNMD to handle determining if a column in a table is indirect and which column to use to resolve the actual target. * Produce an EnC diff with Roslyn APIs directly so we can test the delta tables and still have a good in-VS test running experience. * Make the delta image builder instantiated lazily to fix macos build. * Move composite image creation to native and fix some bugs that our tests caught. * Add IIDs for IMetaDataEmit and IMetaDataEmit2 * Add options guid. * Explicitly use portable pdbs so the EnC emit doesn't fail on non-Windows platforms * Trim excess null terminators from the strings heap. This is done by both Mono and CoreCLR and is required for EnC minimal diffs to apply correctly when the padding is present.
Commits on Mar 31, 2023
-
Fixes based on fuzzing with afl++ (dotnet#25)
* Check metadata offset in PE Check supported image types * Add missing PE defines * Handle PE section headers
Commits on Jun 22, 2023
-
Update build infrastructure to install as a modern CMake package (dot…
…net#26) This PR updates the build of DNMD in a few ways: - Export the dnmd targets as package components. - Ensure that dncp is consumed through its exported interface in all targets - Consume icu through CMake modules instead of direct linking - Update dnmd.h to be buildable without including any other headers (useful for prototyping using dnmd in Mono) - Add a dependency on dncp from dnmd itself to complete the above.
Commits on Jul 28, 2023
-
-
Explicitly validate that the user is asking for a sorted column when …
…searching on a sorted table (dotnet#28)
Commits on Jul 30, 2023
-
Change the interfaces library to use a tear-off model to make adding …
…additional interfaces based on the parameters passed to the dispenser easier (dotnet#29)
Commits on Aug 11, 2023
-
Use macros to templatize search functions (dotnet#31)
* Use macros to templatize search functions * Update macro definition * Review feedback
Commits on Aug 14, 2023
-
Add a writing API and implement the md_apply_delta API in terms of it (…
…dotnet#30) * Create basic entrypoint API for creating a read-write handle on an existing metadata image. * Split up headers and make sure dnmd.h can build if it is included as the first #include. * Add in support for tear-offs and centralize refcounting in a base "controlling IUnknown" type to avoid having to have both ref-counting rules (composed and otherwise) in the same object. * Provide tear-off for emit * Move IUnknown implementation to be provided by a template base type instead of by a macro. * Fix build with new headers * Fix some simple refcounting errors and typos found with some basic smoke testing. * Make dnmd.h not have any dependencies on any other headers being implicitly included before it is included. * Ensure we fully clean-up all allocations by the editor when destroying the handle. * Implement proof-of-concept API for creating an indirection table to validate basic flow of the editor API design. * Implement some basic writing logic and fix query context creation to allocate memory if a writable query is created over a currently-readonly table in an editable context. * Update based on rebase. * Finish implementing table resizing. * Add include lost in rebase * Don't include dncp in dnmd.hpp. DNMD should not depend on dncp at all. * Add set APIs for heap columns * Use create_cursor directly for less overhead. * Add row insertion and creation of indirection tables. * Add support for creating new tables in the append method if a table does not exist initially. * Add internal API for copying heaps over from another image. Logic ported from the mddump_delta_support branch in the main repo. * Remove fixed TODO and add an assert (this was fixed in the caller) * Fix calculating the writable data address. * Add table sorting validation checks Validate that calls to md_find_range_from_cursor only try to find a range for the primary key column. Calling md_find_range_from_cursor with a non-primary-key on a sorted table causes inconsistent behavior. Validate that a sorted table is still sorted when key columns are written to. This design imposes a requirement on the user to write the columns for a new row in order of the sorting keys, but it also enables the user to not have to consider if the table is sorted (and more importantly, we don't need to expose that concept to the user). * Remove completed TODOs * Fix build breaks * Refactor check for sorted blob columns * Handle edge cases for heaps (no heap and too large a heap) * Start putting real method bodies into the IMetaDataEmit API. * Add string splitting and add more emit apis * Make mdhandle_t images always editable, lazily mark as edited. Refactor editor and delta APIs to have a better split. * Fix build failures in emit * Implement a few more emit APIs. * Fix comparison * More APIs and fix build * Hide some struct definitions in source files instead of headers. Make the editor object lazily initialized and make "editor != NULL" the mark for "is this image edited". * Style cleanup. * First pass implementing delta application API. * Various fixes to get minimal test passing * Various fixes * Fix Linux build and fix one place where I missed a malloc call. * Make sure we always initialize list columns, even in the "add a child element" case. Re-create the query context when we're validating sort order. * Fix setting up the indirection table (and pointing the parent table to it) Add some comments about future required work around handling already-created indirection tables. * Fix bugs relating to maintaining the indirection tables and shifting row references. We now correctly apply deltas that require us to introduce and update indirection tables. As a result, our delta application is fully-featured (to my knowledge). * Add comment about adding to the composite enc log * Modify table allocation and heap resizing to ensure that flags and table layouts in memory are consistent with how they'd be represented on disk (simplifies saving). * Add validation of Enc edit chains. * Fix creating the indirection table now that we size it correctly based on the actual target table size. Also, move insert_row_into_table to use a 1-based row index. It was the only place using a 0-based index, which was really confusing. * Remove some unnecessary casts and add some casts to ensure that we widen before doing a possibly-overflowing operation. * Remove changes in dnmd_interfaces. We'll make those changes in a follow-up PR * Add some comments and move one API to be internal * Clean up handling the high bit. * Move the query context out to its own file and rename to "access" context as it now also has write support. Move the writing functions out of query.c and in to write.c * Fix appending for guid heap * Refactor APIs that were used to implement delta application to be more self-contained in the API surface now that we can reach into cursors more easily. * Disable binary search lookups when a row-add operation is in progress. Reimplement sorting validation with cursors instead of query access contexts. It's significantly cleaner and easier to understand. * Move list column initialization into the public row-add API. Now all columns will be initialized to traversable/readable values after all row-add calls. * Don't validate sorting on column write when we're adding a row. Instead only validate at the end of the row-add operation. * Rewrite how we add rows to tables so that we have a few simple operations that only do basic work like allocating, shifting, initializing, updating references (append, add before, add after) and one complex operation (add element to list) that handles the crazy logic with indirection tables. With this design, it is not possible to insert rows into a table in such a way to break layout of the table and tables that need special handling only work with the special handling functions (and ones that don't are blocked from the special handling functions). * Fix adding an item to a list in the following cases: - The list is not the first empty list in a set of lists - The list is not the only empty list at the end of the table. * Ensure that the lists in the Param table are always sorted by their Sequence column. * PR feedback. * Fix Windows Release build failures. * Clang/GCC warn (which we turn into error) for unused functions. static functions are included in this warning, but "inline" functions are not. * Explicitly ignore composed value in release builds. * Add missing const * Change back to static and just suppress the warning * Apply suggestions from code review Co-authored-by: Aaron Robinson <[email protected]> * More error checks. * Add a qsort_s implementation adapted from CoreCLR's utilcode and use it to sort the parameter list. We'll conditionally use our implementation when an implementation does not exist on the platform. * Change signature of compare callback. * MSVC's qsort_s is just ever so slightly incompatible with the standard definition, so just hard-code the sort that we need instead of fighting it and adding a ton of CMake goo. * Revert changes to configure.cmake * Change the quicksort out for a simple insertion-sort-like algorithm as we're likely do already be in order for most items. * Build * Apply suggestions from code review Co-authored-by: Aaron Robinson <[email protected]> --------- Co-authored-by: Aaron Robinson <[email protected]>
Commits on Aug 15, 2023
Commits on Aug 16, 2023
-
Create API for accessing raw column data. (dotnet#34)
* Create API for accessing raw column data. The API is not written in a performant manner. Co-authored-by: Jeremy Koritzinsky <[email protected]>
Commits on Aug 24, 2023
-
Implement save APIs and provide an mdmerge tool that composes an imag…
…e from deltas and writes it to disk instead of dumping to console. (dotnet#35) Co-authored-by: Aaron Robinson <[email protected]>
Commits on Aug 30, 2023
Commits on Sep 2, 2023
-
Introduce a marker interface and move ownership to it (dotnet#39)
Co-authored-by: Aaron Robinson <[email protected]>
Commits on Sep 11, 2023
Commits on Sep 13, 2023
Commits on Nov 28, 2023
-
Remove tool source duplication (dotnet#42)
* Remove tool source duplication * Various nits across codebase * Fix decompressing of signed u32 * Fix bug in md_parse_local_constant_sig().
Commits on Nov 29, 2023
-
Change the "data to write" parameters in
md_set_column_value_as_*m……ethods to be const. (dotnet#43)
Commits on Nov 30, 2023
-
Add a new md_add_new_row_to_sorted_list function to insert a row into…
… a list column's target table in the correctly sorted position so we do not need to re-sort later. (dotnet#44)
Commits on Dec 5, 2023
-
Move the "get MethodDefSig from a MethodRefSig" functionality into dn…
…md::interfaces (dotnet#45) Co-authored-by: Aaron Robinson <[email protected]>
Commits on Dec 6, 2023
-
Initial infrastructure for SymReader support (dotnet#46)
* Comment out non-standard pragmas. * Remove from anonymous namespace.
-
* Update to .NET 8 * Disable test workflow
Commits on Jan 4, 2024
-
Fix the version length field when saving (dotnet#51)
We need to write the length of the aligned buffer, not the length of the version string. Discovered in icsharpcode/ILSpy#3068
Commits on Jan 10, 2024
-
Implement signature importing (dotnet#48)
* Bring over the partial implementation of TypeDef importing from the imetadataemit branch and make it a free function that operates on DNMD handles. * Add SHA1 pal implementation * Generate public key token from public key * Finish implementation of TypeDef importing * Sketch out ImportReferenceToTypeRef * Port over strong name token improvements from my CoreCLR branch. * Fix name shadowing * Implement more of TypeRef importing, with todos and comments for the remaining parts. * Finish TypeRef importing. * Implement the remainder of signature importing (TypeSpec and higher level signatures) * Fix qualified base member access * Fix PAL errors * Fix headers and use E_FAIL instead of E_UNEXPECTED (which isn't available in DNCP) * Fix typo caught by static analysis * Initialize variable to avoid a maybe-uninitialized warning that isn't accurate * Apply suggestions from code review Co-authored-by: Aaron Robinson <[email protected]> * Comment about initialization * Implement importing TypeRef to self as TypeDef * Correctly handle implicit ExportedType rows for types in the source Assembly referenced by a nil ResolutionScope in the source module. * Refactor adding AssemblyRef rows to always add to both the module and assembly (as the CoreCLR does) * Use non-deprecated oneshot SHA API on OpenSSL and use API with the same shape from CommonCrypto * Implement PR feedback * make md_added_row_t constructor explicit and add back removed comment --------- Co-authored-by: Aaron Robinson <[email protected]>
Commits on Jan 18, 2024
-
Add DNMD API to create a new metadata handle and implement IMetaDataD…
…ispenser::DefineScope (dotnet#49) Co-authored-by: Aaron Robinson <[email protected]>
Commits on Jan 26, 2024
Commits on Apr 10, 2024
Commits on Jun 18, 2024
-
Provide single-threaded (no-locking) implementation of IMetadataEmit …
…family of APIs. (dotnet#52)
Commits on Jul 23, 2024
Commits on Oct 31, 2024
-
-
Change the DNMD APIs to act on single rows and bring other
perf improvements from the CoreCLR + DNMD experiment to DNMD. (dotnet#55)
-
Commits on Nov 1, 2024
Commits on Nov 6, 2024
-
Integrate DNMD into the runtime repos build.
Convert testing harness to C++11. Convert DNMD PALs to use the minipal and the COM minipal. Integrate DNMD testing into the CLR_Tools_Tests job.
-
Remove the implicit conversion from span<T> to T* and instead introdu…
…ce a .data() member, matching the C++20 design for span<T>
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.