TypeDB 3.0.5

Download from TypeDB Package Repository:

Distributions for 3.0.5

Pull the Docker image:

docker pull typedb/typedb:3.0.5

New Features

• Dramatically improve error messages with source query pointers

  TypeDB now shows the source of an error in the context of the original query, wher it is possible. In general, we aim to show a detailed error message in context of the original query whenever the error arises in the compilation phase of the query.

  Example of the improved error format:

  [QEX2] Failed to execute define query.
  Near 4:29
  -----
          define
          attribute name value string;
  -->     entity person owns name @range(0..10);
                                  ^
        
  -----
  Caused by: 
        [DEX25] Defining annotation failed for type 'person'.
  Caused by: 
        [COW4] Concept write failed due to a schema validation error.
  Caused by: 
        [SVL34] Invalid arguments for range annotation '@range(0..10)' for value type 'Some(String)'.
  

• Apply typedb error macro to top-level packages

  TypeDB now reports extended error stack traces for all error types in the compiler and the intermediate representation builder, improving debuggability and ease-of-use.

Bugs Fixed

• Introduce role-player deduplication & Expand BDD coverage
  Introduce role-player deduplication for when specified together in a single links constraint. i.e. $r links (my-role: $p, my-role: $q); will not use the same edge twice to satisfy each sub-constraint.
  Writing them as separate links constraint $r links (my-role: $p); $r links (my-role: $q); will not de-duplicate.

• Fix relates double specialization
  Fix the behavior of relates specialization, featuring:

  • Unblocked double specialization:

  define 
    relation family-relation relates member @abstract;
    relation parentship sub family relation, relates parent as member, relates child as member; # Good!
  

  • Fixed validations for multi-layered specializations:

  define
    relation family-relation relates member @abstract, relates pet @abstract;
    relation parentship sub family relation, relates parent as member;
    relation fathership sub parentship, relates father as member; # Bad!
    relation fathership sub parentship, relates fathers-dog as pet; # Good!
  

  • Better definition resolution and error messaging.
  • Change the inner terminology for generated relates for specialization: "root" and "non specializing" are substituted by "explicit", and "specializing" is substituted by "implicit".

Code Refactors

Other Improvements

• Replace unwraps with panics for more detailed error messages

• Update automation.yml: return test_update

• Bugfix: direction of indexed relation instruction in lowering

  We fix the logic in the lowering of query plans to executables that determines the direction of indexed relation instructions.

TypeDB 3.0.4

Download from TypeDB Package Repository:

Distributions for 3.0.4

Pull the Docker image:

docker pull typedb/typedb:3.0.4

New Features

Bugs Fixed

• Add correct version for CLI --version flag. Substitute encryption-related config crashes by user errors.
  Update the --version flag to return the correct version of the server when requested.
  Prevent the server from crashing when an incorrect encryption configuration is supplied, stopping it gracefully and returning a defined error instead.

Code Refactors

• Add 'dec' suffix to display of decimal
  Add 'dec' suffix to display of decimal

Other Improvements

• Add 'dec' suffix to display of decimal (#7326)

• Re-enable behaviour tests
  Enables tests for various read & write behaviour.

• Cleanup todos
  Replaces usages of the todo macro with either errors which will be returned, or custom macros if the line is unreachable. This reduces server crashes due to panics when execution reaches the todo.

TypeDB 3.0.3

Download from TypeDB Package Repository:

Distributions for 3.0.3

Pull the Docker image:

docker pull typedb/typedb:3.0.3

New Features

Bugs Fixed

• Fix TypeDB reporting crashes when CA certificates are not found
  We fix TypeDB reporting crashes when CA certificates are not found on the host. An additional Sentry (our crash reports endpoint) warning will be reported, but it no longer affects the server's availability.

Code Refactors

Other Improvements

• Make windows distribution directory structure consistent with other platforms
  Makes the directory structure of the windows distribution consistent with other platforms.

TypeDB 3.0.2

Download from TypeDB Package Repository:

Distributions for 3.0.2

Pull the Docker image:

docker pull vaticle/typedb:3.0.2

New Features

Bugs Fixed

• Cartesian product fix

  We fix the issue where a join on an indexed relation would skip all additional data within the join variable.

• Fix rollback and remove dependency on typedb-common
  We fix the rollback feature, which used to hang the transaction operated from an external driver, not allowing the operations to proceed correctly.

  Additionally, we remove an outdated Bazel dependency on typedb-common, which is no longer needed for TypeDB 3.0.

Code Refactors

• Improve TypeDB error macro

  We refactor the typedb_error! macro to avoid the use of meaningful parentheses and position-dependence.

• Update grouped reduction syntax and add reduction tests

  We use the new grouped-reduction syntax. So we no longer use 'within':

  match $x...; $y ...;
  reduce $count = count($x) within $y;
  

  and we now write:

  match $x...; $y ...;
  reduce $count = count($x) groupby $y;
  

• Remove action metrics reporting to Posthog if reporting is disabled
  We remove action metrics reporting for all remote diagnostics endpoints if reporting is disabled. Previously, while not sharing confidential information, the server could still send the full action metrics snapshot to one of the endpoints, which contained information about user actions: numbers of opened connections, opened transactions, executed queries, etc.

Other Improvements

• Extend query planning to functions & introduce IAM benchmark
  Extends query planning to consider functions. Non-recursive functions add up planning cost estimates of every triggered function body. Recursive function planning currently just sets the recursive call cost to 1.

• Implement release pipeline for windows
  Implements a release pipeline for windows which uses cargo instead of bazel to build TypeDB.

• Small cost model fix

  When we, e.g., look up an attribute of specific type of a bound entity, then the cost should be proportional to the average number of attributes of that type of the entity (not average number of all attributes of the entity).

• Add Sentry for critical error reporting and fix panics on diagnostics initialization
  We add integration with Sentry to allow critical errors (e.g. panic!s) reporting. This information will help us see and eliminate unexpected TypeDB Server crashes.
  Please use the -diagnostics.reporting.errors boolean option to disable this feature if its work is undesirable (note that it will reduce the efficiency of our maintenance).

  Additionally, an overflow subtraction bug that sometimes affects diagnostics initialization and leads to crashes is fixed.

• Add a more user-friendly taken address error
  Introduce a more user-friendly and explicit "Address already in use" error while running a second instance of the server with the same addresses and monitoring enabled:

  Ready!
  WARNING: Diagnostics monitoring server could not get initialised on 0.0.0.0:4104: 'error creating server listener: Address already in use (os error 48)'
  Exited with error: [SRO7] Could not serve on 0.0.0.0:1729.
  Cause: 
           tonic::transport::Error(Transport, Os { code: 48, kind: AddrInUse, message: "Address already in use" })
  

• Add more value types to typeql tests

• Return concept bdds, fix build

TypeDB 3.0.0

Download from TypeDB Package Repository:

Distributions for 3.0.0

Pull the Docker image:

docker pull typedb/typedb:3.0.0

Announcement

This last year of rewriting TypeDB 2.x into TypeDB 3.0 has been an adventure of learning, iterating, and lots of fun for our whole team.

We're thrilled to say we've achieved our first tranche of goals:

• Initial testing shows TypeDB 3.0's performance is comparable-to or surpasses MongoDB in transactional workloads (a 3-5x performance gain over TypeDB 2.x!)
• TypeDB 3.0 leverages its new Rust codebase to greatly increase correctness, reduce memory footprint, and improve performance
• It features a simplified architecture that opens the door for adding new features and optimisations quickly

Not only that, but we've upgraded TypeQL to version 3.0 to directly address long-standing requests:

• Rules have been replaced with Functions: functions are more flexible, easier to reason about, and much more familiar to programmers. Functions are like subqueries you can re-use and invoke whenever you want. We think you'll love them!
• Query pipelining: combined with more powerful expressions and functions, you can now build read, write, or transformation pipelines that do everything you need without coming back to the client. This is a triple threat: more readable, more maintainable, and more performant.
• Data constraints: we welcome the arrival of Cardiniality, value Range, and value Enumeration restrictions, among others. These have been our top requested features for over a year and now they're at your fingertips! TypeDB will now automatically validate that your data has the exact connectivity and shape you require.

There's so much more than this, and we hope you'll dig into TypeDB 3.0 to try it out.

Top 10 TypeDB 3.0 Features

1. Goodbye sessions, hello transactions: TypeDB 3.0 eliminates sessions and simplifies your interactions to use 3 types of transactions. Use read, write, or schema transactions when reading data, modifying data, or exclusively modifying schema & data, respectively. Transactions are still ACID up to snapshot isolation, and support concurrent reads and writes.

2. Standard return types - Rows or Documents: TypeDB 3.0 simplifies the answers to be either Rows or Documents. Rows will feel familiar to SQL users, though still contain our traditional Concept data types. Documents can be structured and build using the enhanced fetch clause.

3. Enhanced schema language: We've streamlined define, undefine, and introduce redefine to modify schemas - while restructuring the definition language to be shorter, more consistent, and more understandible:

TypeDB 2.x:

define
  person sub entity,
    owns age,
    plays friendship:friend;
  friendship sub relation,
    relates friend;
  age sub attribute, value string;

  abstract-friendship sub relation,
    relates friend, abstract;


undefine
  abstract-friendship relates friend, abstract; # does this undefine the `abstract`, the `friend` role, or the `abstract-friendship`?
  

TypeDB 3.x:

define
  entity person,
    owns age,
    plays friendship:friend;
  relation friendship,
    relates friend;
  attribute age, value string;
  relation abstract-friendship,
    relates friend @abstract;
  entity child, owns age;
  entity dog;


undefine
  @abstract from abstract-friendship relates friend; # clearly removes @abstract
  owns age from child; # clearly removes only the age ownership from the child type
  dog; # clearly undefines the dog type

4. Query pipelines: Pipelines are best illustrated. Let's say we want to assign a tax credit for a person called Bill, corresponding to how many children he has.

In TypeDB 2.x, we have to split this into multiple queries that do redundant work, and perform multiple network round trips. In addition, we have to split some of our logic between the database and the application, which damages maintainability!

# count how many children Bill has
children = tx.query().get_aggregate('match $p isa person, has email "[email protected]"; (child: $child, parent: $p) isa parentship; get $child; sum;').resolve()
# compute tax credit
tax_credit = 1000*children;
# assign tax credit
tx.query().insert(f'match $p isa person, has email "[email protected]"; insert $p has tax_credit {tax_credit};')

In TypeDB 3.0, this is streamlined into one query pipeline:

tx.query("""
  match $p isa person, has email "[email protected]"; parentship (child: $child, parent: $p);
  reduce $count = count($child) within $p;
  match let $tax_credit = 1000 * $count;
  insert $p has tax_credit == $tax_credit;
""")

This is a simple pipeline: we could continue to chain operations, such as inserts and deletes, to build complex transformations.

5. Reduce and aggregation operations: As the previous example illustrates, what used to be a get; sum; operation is now a reduce clause! This is a much more expressive way of aggregating values and allows using them in future operations.

6. Enhanced Fetch: fetch clauses now are structured exactly how they will return JSON objects:

match
    $p isa person, has email $email, has age $age; $email == "[email protected]";
fetch {
    "email": $email, 
    "tax-identifier": $p.tax_id,
    "names": [ $p.name ],
    "age_next_year": $age + 1,
    "total_salary": (
        match
        $p has salary $salary;
        return sum($salary);
    ),
    "children_ages": [
        match
        parentship (child: $child, parent: $p);
        fetch {
            "age": $child.age,
        };
    ],
    "all_attributes": { $p.* }
};

This will return a stream of JSON documents that look like this:

{
    "email": "[email protected]", 
    "tax-identifier": "123-45-6789",
    "names": [ "bill", "billstone" ],
    "age_next_year": 51,
    "total_salary": 50000,
    "children_ages": [
      { "age": 10 },
      { "age": 13 },
    ],
    "all_attributes": { 
      "email": "[email protected]",
      "age": 50,
      "tax_id": "123-45-6789",
      "names" [ "bill", "billstone" ],
      "salary": [ 10000, 40000 ],
    }
};

{
    "email": "[email protected]",
    "tax-identifier": "123-45-6789",
    "names": [
        "bill",
        "billstone"
    ],
    "age_next_year": 51,
    "total_salary": 50000
    "children_ages": [
        { "age": 10 },
        { "age": 12 }
    ],
    "all_attributes": {
        "age": 50,
        "email": "[email protected]",
        "name": [
            "bill",
            "billstone"
        ],
        "salary": [
            10000,
            40000
        ],
        "tax_id": "123-45-6789"
    },

}

7. Functions: functions take a set of arguments, and return either a stream or a tuple of concepts. Functions can contain any read pipeline!

define
fun mean_salary($p: person) -> double:
  match $p has salary $salary;
  return mean($salary);

8. Let-bindings and inline expressions: expressions are now bound to a new variable (using the same $ syntax - no more ? variables!) using the 'let' keyword. If you don't need to name your result, you can inline expressions:

match
  $p has salary $annual_salary;
  let $monthly_salary = $annual_salary / 12;
  parentship (parent: $p, child: $child);
  $child has age > (10 + 8);

9. Data constraints: TypeDB 3.0 ships with built-in controls for cardinality, values, ranges, and abstractness, best shown by example:

define
  entity person, 
    owns email, # NEW: unless specified, ownerships have a default cardinality of @card(0..1) (exactly 0 or 1 attributes)
    owns name @card(0..),  # specifically relaxed cardinality of 0 to infinity
    owns tax_id @card(1, 1), # require exactly 1 tax_id to be owned 
    owns age,
    owns gender;

    plays parentship:parent, # NEW: unless specified, played roles have an implicit @card(0..) (any number of connections allowed)
    plays parentship:child;

  relation parentship,
    relates child, # NEW: unless specified, relation roles have a default cardinality of @card(0..1) (exactly 0 or 1 of the role)
    relates parent @card(1..2);  # specialised relation roles to require 1 or 2 parents for each parentship relation

  attribute age, value integer @range(0..); # require all ages to be of values equal to or greater than 0
  attribute gender, value string @values("male", "female", "other"); # restrict the domain of values to an exact set

10. New built-in value types: TypeDB 3.0 ships with a larger set of value types:

• integer (renamed from long in 2.x)
• double
• boolean
• string
• NEW: date, representing a date, without a time
• datetime
• NEW: datetime-tz, representing a date time with timezone
• NEW: duration
• NEW: Fixed-Decimal numbers

Aggregated release note ---------------

New Features

• User authentication and network encryption

  User authentication

  Bring user authentication and user management functionalities into TypeDB Core through the following changes:

  • Implement user management functionality, handling user creation, retrieval, update, and deletion.
  • Implement authentication protocol - TypeDB Core drivers are now required to supply credentials when making a connection

  System database

  Implement a system database, a special database that the server will use to store various systems-related information. The primary motivation is to store database server users for authentication purpose, but in the future can be extended to store other system-related information.

  Network encryption

  Bring n...

TypeDB 3.0.0-alpha-10

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-10

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-10

New Features

• IID constraint

  We add support for IID constraints in match clauses.

  Example:

  match $x iid 0x1E001A0000000012345678;
  

Bugs Fixed

• Fix greedy planner to sort on correct vars

  Currently, the greedy planner picks a plan in form of an ordering of constraints and variables. An added constraint produces variables if it relates those variables and they haven't been added to the plan at an earlier stage.

  The greedy planner picks constraints based on their "minimal" cost for a directional lookup, that may sort on either of one or more variables that the constraint relates. However, it does not record which variable the lookup should be sorted on. This means, as it stands, produced variables are added in random order after their constraint to the ordered plan.

  This PR aims to ensure that we sort on the variable for the appropriate direction, by adding them first.

• Guarantee transaction causality

  Under concurrent write workloads, causality can appear to be violated. For example:

  - Open tx1, tx2
  - start tx1 commit (validation begins)
  - start tx2 commit (validation begins and ends, no conflict with tx1!)
  - open tx3
  - end tx1 commit (validation finishes)
  

  When we open tx3, we end up with a snapshot that is actually from before tx1, even though tx2 has committed - because we don't know the status of tx1 yet, and in our current simplified model, transaction are assigned a linear commit order decided at WAL write time, the read watermark remains before tx1 until it finishes validating.

  In this scenario, a client that commits a transaction can actually end up opening the next transaction snapshot before the last commit that successfully returned.

  After this change, when opening a transaction, TypeDB wait until the currently pending transactions all finish, guaranteeing we see the latest possible data version. The assumption is that 1) validation in general is a small amount of time and 2) is fully concurrent, so the wait time should be very small, and only occur under large concurrently committing transactions.

Code Refactors

• Optimisations from typeql

  Updates typeql for parsing optimisations.

Other Improvements

• Add BDD tests for functions

  Add basic BDD tests for functions & implement validation.

• Frugal type seeder

  Optimises parts of the type seeder to avoid unnecessary effort.

• Improve Bytes ergonomics by allowing automatic dereference to byte slices

TypeDB 3.0.0-alpha-9

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-9

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-9

New Features

• Exit on panic

  In Rust, whenever a panic is encountered in a background thread, the main thread is normally unaffected until it either joins the background thread, or attempts to lock a mutex that was held by the panicking thread. If that communication never occurs (e.g. no locks were held by a worker), the server may end up in an invalid state.

  We override the default behaviour to exit from the process with code 1.

• Stabilise fetch and introduce fetch functions
  We introduce function invocation in fetch queries. Fetch can call already existing functions:

  match
    $p isa person;
  fetch {
    "names": [ get_names($p) ],
    "age": get_age($p)
  };
  

  and also use local function blocks:

  match
    $p isa person;
  fetch {
    "names": [
      match
        $p has name $n;
        return { $n };
    ],
    "age": (
      match
        $p has age $a;
        return first $a;
    )
  };
  

  In the examples above, results collected in concept document lists ("names": [ ... ]) represent streams (purposely multiple answers), while single (optional) results ("age": ...) represent a single concept document leaf and do not require (although allow) any wrapping.

Moreover, we stabilize attribute fetching, allowing you to expect a specific concept document structure based on your schema. This way, if the attribute type is owned with default or key cardinality (@card(0..1) or @card(1..1)), meaning that there can be at most one attribute of this type, it can be fetched as a single leaf, while other cardinalities force list representation and make your system safe and consistent. For example, a query

match
  $p isa person;
fetch {
  $p.*
};

can automatically produce a document like

{
  "names": [ "Linus", "Torvalds" ],
  "age": 54
}

with

define
  entity person 
    owns name @card(1..), 
    owns age @card(1..1);

Additionally, fetch now returns attributes of all subtypes x sub name; y sub name; for $p.name, just like regular match queries like match $p owns name $n.

With this, feel free to construct your fetch statements the way you want:

match
  $p isa person, has name $name;
  fetch {
    "info": {
      "name": {
        "from entity": [ $p.person-name ],
        "from var": $name,
      },
      "optional age": $p.age,
      "rating": calculate_rating($p)
    }
  };

to expect consistent structure of the results:

[
  {
    "info": {
      "name": {
        "from entity": [ "Name1", "Surname1" ],
        "from var": "Name1"
      },
      "optional age": 25,
      "rating": 19.5
    }
  },
  {
    "info": {
      "name": {
        "from entity": [ "Name1", "Surname1" ],
        "from var": "Surname1"
      },
      "optional age": 25,
      "rating": 19.5
    }
  },
  {
    "info": {
      "name": {
        "from entity": [ "Bob" ],
        "from var": "Bob"
      },
      "optional age": null
      "rating": 28.783
    }
  }
]

• Implement query executable cache

  We implement the cache (somewhat arbitrarily limited to 100 entries) for compiled executable queries, along with cache maintanance when statistics change significantly or the schema updates.

  Query execution without any cache hits still looks like this:

  Parsing -> Translation (to intermediate representation) -> Annotation -> Compilation -> Execution
  

  However, with a cache hit, we now have:

  Parsing -> Translation ---Cache--> Execution
  

  skipping the annotation and compilation/planning phases, which take significant time.

  Note that schema transactions don't have a query executable cache, since keeping the cache in-sync when schema operations run can be error prone.

  The query cache is a structural cache, which means it will ignore all Parameters in the query: variable names, constants and values, and fetch document keys. Most production systems run a limited set of query structures, only varying values and terms, making a structural cache like this highly effective!

• Introduce naive retries for suspended function calls
  Introduces retries for suspended function calls.
  Functions may suspend to break cycles of recursion. Restoring suspend points is essential to completeness of recursive functions, and thus correctness of negations which call them.

• Query execution analyser

  We implement a useful debugging feature - a query analyzer. This is similar to Postgres's Explain Analyze, which produces both the query plan plus some details about the data that has flowed through the query plan and the time at each step within it.

  Example output:

  Query profile[measurements_enabled=true]
    -----
    Stage or Pattern [id=0] - Match
      0. Sorted Iterator Intersection [bound_vars=[], output_size=1, sort_by=p0]
        [p0 isa ITEM] filter [] with (outputs=p0, )
      ==> batches: 158, rows: 10000, micros: 6407
  
      1. Sorted Iterator Intersection [bound_vars=[p0], output_size=2, sort_by=p1]
        Reverse[p1 rp p0 (role: __$2__)] filter [] with (inputs=p0, outputs=p1, checks=__$2__, )
      ==> batches: 854, rows: 39967, micros: 75716
  
    -----
    Stage or Pattern [id=1] - Reduce
      0. Reduction
      ==> batches: 1, rows: 10000, micros: 116035
  
    -----
    Stage or Pattern [id=2] - Insert
      0. Put attribute
      ==> batches: 10000, rows: 10000, micros: 5890
  
      1. Put has
      ==> batches: 10000, rows: 10000, micros: 54264
  

  When disabled, profiling is a no-op (no strings are created, locks taken, or times measured), though there is still some cost associated with cloning Arcs containing the profiling data structures around.

  To enable query profiling, the easiest way (for now) is to enable TRACE logging level for the executor package, currrently configured in //common/logger/logger.rs:

  .add_directive(LevelFilter::INFO.into())
  // add:
  // .add_directive("executor=trace".parse().unwrap())
  

  Alternatively, just set the enable boolean to true in the QueryProfile::new() constructor.

Bugs Fixed

• Variable position fix

• Disable variables of category value being narrow-able to attribute
  Disable variables of category value being narrowed to attribute

  $name = "some name"; $person has name == $name; # Fine
  $name == "some name"; $person has name $name; # Also Fine
  $name = "some name"; $person has name $name; # Disallowed
  

• Fix disjunction inputs when lowering
  The disjunction compiler explicitly accepts a list of variables which are bound when the disjunction is evaluated. This also fixes a bug where input variables used in downstream steps would not be copied over.

• Fixes early termination of type-inference pruning when disjunctions change internally
  Fixes early termination of type-inference pruning when disjunctions change internally

Code Refactors

• Update FactoryCI images to typedb-ubuntu

• Rename org to 'typedb' in deployment.bzl

• Rename org to 'typedb' in CircleCI docker job

• Boxed large result Error variants

  We follow some best practices to box the large Error variants of Result types, which should optimise both the stack size and the amount of memory that has to be copied between function calls.

• Decimal to double cast

  We can now use decimals in double expressions.

• Optimise has comparators and implement Executable Display

  We inline comparators into HasReverse executors, as a continuation of #7233, which inlined comparators into Isa executors. Both optimisations reduce the search space by using the value ranges provided by constants in the query or previously executed query steps. We also improve the query planner's statistics and cost function.

Other Improvements

• Add cargo check to CI

  We add a job to Factory CI which verifies that the Cargo configuration is valid and buildable.

• Return query operations errors on commit
  We return errors from the awaited operations on commit as the commit error. It can be useful if you want to run a number of queries from your TypeDB Driver like:

  let queries = [........];
  let mut promises = vec![];
  for query in queries {
      promises.push(transaction.query(query));
  }
  
  let result = transaction.commit().await;
  println!("Commit result will contain the unresolved query's error: {}", result.unwrap_err());

• Update FactoryCI images to typedb-ubuntu

• Fix structural equality test

• Implement structural equality

  We implement a form of Structural Equality, which check the equality of two queries, in Intermediate Representation format, which allows us to compare to queries for equality while ignoring user-written variable names and function names and constants. This is a building block for the next step, which will introduce a executable cache, allowing TypeDB to skip type inference and planning for queries that already exist in the cache.

• Deleted BUILD_java

• Commit generated Cargo.tomls + Cargo.lock

  We commit the generated cargo manifests so that TypeDB can be built as a normal cargo project without using Bazel.

  Integration and unit tests can be run normally, and rust-analyzer behaves correctly. Behaviour tests currentl...

TypeDB 3.0.0-alpha-8

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-8

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-8

New Features

• Is constraint

  We implement the is constraint support in queries:

  match
      $x isa company;
      $y isa person;
      $z isa person;
      not { $y is $z; };  # <---
      $r1 isa employment, links ($x, $y);
      $r2 isa employment, links ($x, $z);
      select $x, $y, $z;

• Introduce 3.0 undefine queries
  We introduce undefine queries with the new "targeted" syntax to enable safe and transparent schema concept undefinitions.

If you want to undefine the whole type, you can just say:

undefine 
  person;

If you want to undefine a capability from somewhere, use undefine **capability** from **somewhere**.
It consistently works with owns (person is preserved, only the owns is undefined):

undefine
  owns name from person;

annotations of owns (person owns name is preserved, only @regex(...) (you don't need to specify the regex's argument) is undefined):

undefine
  @regex from person owns name;

and any other capability, even specialisation:

undefine
  as parent from fathership relates father;

Want to undefine multiple concepts in one go? Worry not!

undefine
  person;
  relates employee from employment;
  @regex from name;
  @values from email;

The error messages in all definition queries (define, redefine, and undefine) were enhanced, and the respective query/language BDD tests were introduced in the CI with concept unit tests.

Additional fixes:

• answers of match capability queries like match $x owns $y, match $x relates $y, and $match $x plays $y now include transitive capabilities;
• define lets you write multiple declarations of an undefined type, specifying its kind anywhere, even in the last mention of this type;
• failures in schema queries for schema transactions no longer lead to freezes of the following opened transactions;
• no more crashes on long string attributes deletion with existing has edges;

Bugs Fixed

• Fix outputs from earlier disjunction branches affecting later branches
  Fixes outputs from earlier disjunction branches affecting later branches

• Ensure sorted join executor respects selected outputs

  Before this change, an intersection executor (which is used for any number of sorted iterators, not necessarily a join) would write into the output row all items in the constituent executors, so long as they have a row position assigned. That would happen even if those values were not selected for future steps, which prevented deduplication of answers from being handled correctly.

Code Refactors

• Simplify attribute encoding

  We remove separate prefixes for attribute instances and instead introduce a value type prefix after the attribute type encoding.

  Encoding before:

  [String-Attribute-Instance][<type-id>][<value>][<length>]
  

  After:

  [Attribute-Instance][<type-id>][String][<value>][<length>]
  

  Note: this change breaks backwards compatibility

• Refactor after basic functions
  Refactors the executor to be have more simple, flat instructions than less, complex ones.

Other Improvements

• Carry correct types into type list executor

  In cases where the list of types is dependent on the context (e.g. {$t label T;} or {$t label U;};, the type list executor (role name, explicit label, or kind) should only be producing the types in the list defined by the constraint. That was handled correctly when the type constraint was used as a post-check, but incorrectly during direct execution.

• Update spec.md

• Introduce 3.0 value and as match constraints
  We introduce value match constraint for querying for attribute types of specific value types:

  match
    $lo value long;
  

  We introduce as match constraint for querying for role types specialisation:

  match
    $relation relates $role as parentship:child;
  

  which is equivalent to:

  match
    $relation relates $role;
    $role sub parentship:child;
  

  but is useful for shortening your queries, making it similar to define definitions, and in case of anonymous variables:

  match
    $relation relates $_ as parentship:child;
  

• Optimise comparators

  We optimise the execution of queries containing comparators (>/</==/...) where these can be used to restrict the data search space. This PR sets up general infrastructure for using restrictions when searching the data, but uses it only for optimising ISA lookups - follow up work will optimise HAS operations as well.

• Add type annotation to fix build in circleci

• Update spec: fix list insert, add value pattern, add as! pattern

• Bugfix: do not reuse cartesian product iterators between inputs

  We can't reuse the iterators created from the previous input row, as they would iterate over different tuples. Now, when we take a new input, we reset the cartesian iterators.

• Update spec.md (fix sssnake expresssions)

• Green match BDD redux

  We resolve the remaining flaky failures in match BDDs.

• Update banner.png

• Update many unit & integration tests & enable in CI
  Updates many unit & integration tests & enables them in CI

TypeDB 3.0.0-alpha-7

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-7

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-7

New Features

• Expression support

  We add expression execution to match queries:

  match
      $person_1 isa person, has age $age_1;
      $person_2 isa person, has age $age_2;
      $age_2 == $age_1 + 2;

• Basic streaming functions

  Implement function execution for non-recursive stream functions which return streams only.
  Recursive functions & Non-stream functions will throw unimplemented.

  These can currently only be used from the preamble. Sample:

             with
              fun get_ages($p_arg: person) -> { age }:
              match
                  $p_arg has age $age_return;
              return {$age_return};
  
              match
                  $p isa person;
                  $z in get_ages($p);

• Implement tabling machinery for cyclic functions

  Introduces machinery needed to support finding the fixed-point of cyclic function calls. Cyclic functions now run, and return an incomplete set of results followed by an error. It is possible that the planner chooses a plan.

Bugs Fixed

• Minor query planner bug fixes

  1. Fix the issue where an iterator would attempt to use a variable before assigning to it.
  2. Let role name constraint behave as a post-check.

Code Refactors

• Resolve warnings

Other Improvements

• Update protocol to 3.0.0-alpha-7 release

• 3.0 Rename Ok.Empty to Ok.Done. Add query_type to the Done response

  We add a query_type field to all the Query.InitialRes.Ok protobuf messages to support retrieval of this information for any QueryAnswer on the client side.
  Additionally, the Ok.Empty message was renamed to Ok.Done.

• Fix and add to CI concept BDD tests

  We fix data commit error collection and checks and add concept package BDD tests to CI.

TypeDB 3.0.0-alpha-6

Install

Download from TypeDB Package Repository:

Distributions for 3.0.0-alpha-6

Pull the Docker image:

docker pull vaticle/typedb:3.0.0-alpha-6

New Features

• Partial disjunction support

  We introduce partial support for disjunctions in queries:

  match
      $person isa person;
      { $person has name $_; } or { $person has age $_; };

• Fetch execution

  We implement fetch execution, given an executable pipeline that may contain a Fetch terminal stage.

  Note: we have commented out match-return subqueries, fetching of expressions ({ "val" : $x + 1 }), and fetching of function outputs ("val": mean_salary($person)), as these require function-body evaluation under the hood - this is not yet implemented.

Bugs Fixed

• Fix document answers streaming for fetch
  We fix document answers streaming for fetch in order to complete the first version of fetch queries execution.

Code Refactors

• Function compilation preparation & trivial plan implementation
  Prepares higher level packages for compiling functions.

• Fetch annotation, compilation, and executables

  We implement Fetch annotation, compilation and executable building, rearchitecting the rest of the compiler to allow for tree-shaped nesting of queries (eg. functions or fetch sub-pipelines).

• Fetch annotations framework

  We implement further refactoring, which pull Fetch into Annotations and Executables, without implementing any sub-methods yet.

Other Improvements

• Add database name validation for database creation
  We add validation of names for created databases. Now, all database names should be valid TypeQL identifiers.

• Enable connection BDDs in CI
  We update some of the steps for BDDs to match the updated definitions.
  Additionally, we add connection BDDs to the factory CI to make sure that this piece of the system is safe and stable.

• Refactor compiler

  We refactor the compiler to standardize naming, removing the usage of 'program', and introducing 'executable' as the second stage of compilation.

• Refactor pipeline annotations

  We implement the next step of Fetch implementation, which allows us to embed Annotated pipelines into Fetch sub-queries and into functions. We comment out the code paths related to type inference for functions, since functions are now enhanced to include pipelines.