-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(engine)!: syncing published templates #1222
Closed
ksrichard
wants to merge
49
commits into
tari-project:development
from
ksrichard:feature/published-template-sync
Closed
Changes from all commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
4d181b7
impl in progress
ksrichard a0c833d
impl in progress
ksrichard bbf020d
impl done
ksrichard 1bea85d
fix test
ksrichard 62e6f99
format
ksrichard e20f4bf
added new publish template substate
ksrichard 5778757
impl in progress
ksrichard 7f20941
impl in progress
ksrichard 00c3ef0
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard e0de43a
impl in progress
ksrichard fe9807b
impl in progress
ksrichard 0450d49
updating templates db almost done
ksrichard 7b3a6b4
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard a7b90b5
impl in progress
ksrichard 8e29347
renamed PublishedTemplate substate to Template and almost done adding…
ksrichard 9cb0766
Template publishing implemented, new templates are represented as sub…
ksrichard 94598d0
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard fb1bf11
small update
ksrichard b752270
CI fixes
ksrichard 29436dd
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard cbaef1b
small fixes
ksrichard 29f1fde
small fix
ksrichard 63a531b
small fix
ksrichard 40d9936
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard 1e949ed
small fixes
ksrichard 5bb6223
added placeholder TODOs
ksrichard 716d6aa
removed unused import
ksrichard 221ca6c
clippy + bug fix + minor code cleanup + reduce mem usage
sdbondi 890188f
fix unncessary heap allocations in engine
sdbondi 83f117c
support for base layer registration
ksrichard d48d25f
Merge remote-tracking branch 'origin/feature/publish-template-as-subs…
ksrichard d8c90de
clippy fixes
ksrichard 15a81a8
Merge remote-tracking branch 'origin/feature/publish-template-as-subs…
ksrichard 6ac8a73
Merge branch 'development' into feature/publish-template-as-substate
ksrichard 14d9f6a
impl in progress
ksrichard 1ece847
small fix
ksrichard 6dab228
Merge remote-tracking branch 'origin/feature/publish-template-as-subs…
ksrichard 07f8895
fixed auto registration of local templates
ksrichard 31a41d7
Merge remote-tracking branch 'origin/feature/publish-template-as-subs…
ksrichard 3a029b7
Merge remote-tracking branch 'origin/development' into feature/publis…
ksrichard efe6ee3
impl in progress
ksrichard 88cc8f0
impl in progress
ksrichard 0d1274b
impl in progress
ksrichard bff83cc
impl in progress
ksrichard f5b4920
impl in progress
ksrichard 11437e5
impl in progress
ksrichard 584c069
impl almost done, needs extra error handling
ksrichard 037035a
almost fully done syncing implementation, small changes left
ksrichard afdb18b
added todo
ksrichard File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,13 +20,25 @@ | |
// WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE | ||
// USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
||
use std::{collections::HashMap, convert::TryFrom, fs, sync::Arc}; | ||
use std::{ | ||
collections::HashMap, | ||
convert::TryFrom, | ||
fs, | ||
sync::Arc, | ||
thread, | ||
time::{SystemTime, UNIX_EPOCH}, | ||
}; | ||
|
||
use chrono::Utc; | ||
use chrono::{Duration, Utc}; | ||
use log::*; | ||
use tari_common_types::types::{FixedHash, PublicKey}; | ||
use tari_crypto::tari_utilities::ByteArray; | ||
use tari_dan_common_types::{optional::Optional, services::template_provider::TemplateProvider, NodeAddressable}; | ||
use tari_dan_common_types::{ | ||
optional::Optional, | ||
services::template_provider::TemplateProvider, | ||
NodeAddressable, | ||
TemplateSyncRequest, | ||
}; | ||
use tari_dan_engine::{ | ||
flow::FlowFactory, | ||
function_definitions::FlowFunctionDefinition, | ||
|
@@ -54,6 +66,12 @@ const LOG_TARGET: &str = "tari::validator_node::template_manager"; | |
|
||
const CONCURRENT_ACCESS_LIMIT: isize = 100; | ||
|
||
#[derive(Debug, Clone)] | ||
pub enum TemplateResult { | ||
Template(Box<Template>), | ||
PendingTemplate, | ||
} | ||
|
||
#[derive(Debug)] | ||
pub struct TemplateManager<TAddr> { | ||
global_db: GlobalDb<SqliteGlobalDbAdapter<TAddr>>, | ||
|
@@ -120,26 +138,31 @@ impl<TAddr: NodeAddressable> TemplateManager<TAddr> { | |
name: name.to_string(), | ||
address, | ||
binary_sha, | ||
author_public_key: Default::default(), | ||
}, | ||
executable: TemplateExecutable::CompiledWasm(compiled_code), | ||
} | ||
} | ||
|
||
pub fn template_exists(&self, address: &TemplateAddress) -> Result<bool, TemplateManagerError> { | ||
pub fn template_exists( | ||
&self, | ||
address: &TemplateAddress, | ||
status: TemplateStatus, | ||
) -> Result<bool, TemplateManagerError> { | ||
if self.builtin_templates.contains_key(address) { | ||
return Ok(true); | ||
} | ||
let mut tx = self.global_db.create_transaction()?; | ||
self.global_db | ||
.templates(&mut tx) | ||
.template_exists(address) | ||
.template_exists(address, status) | ||
.map_err(|_| TemplateManagerError::TemplateNotFound { address: *address }) | ||
} | ||
|
||
pub fn fetch_template(&self, address: &TemplateAddress) -> Result<Template, TemplateManagerError> { | ||
pub fn fetch_template(&self, address: &TemplateAddress) -> Result<TemplateResult, TemplateManagerError> { | ||
// first of all, check if the address is for a bulitin template | ||
if let Some(template) = self.builtin_templates.get(address) { | ||
return Ok(template.to_owned()); | ||
return Ok(TemplateResult::Template(Box::new(template.to_owned()))); | ||
} | ||
|
||
let mut tx = self.global_db.create_transaction()?; | ||
|
@@ -149,6 +172,11 @@ impl<TAddr: NodeAddressable> TemplateManager<TAddr> { | |
.get_template(address)? | ||
.ok_or(TemplateManagerError::TemplateNotFound { address: *address })?; | ||
|
||
// notify the caller that the template is under sync, so not yet ready | ||
if matches!(template.status, TemplateStatus::Pending) { | ||
return Ok(TemplateResult::PendingTemplate); | ||
} | ||
|
||
if !matches!(template.status, TemplateStatus::Active | TemplateStatus::Deprecated) { | ||
return Err(TemplateManagerError::TemplateUnavailable); | ||
} | ||
|
@@ -167,9 +195,9 @@ impl<TAddr: NodeAddressable> TemplateManager<TAddr> { | |
_ => return Err(TemplateManagerError::TemplateUnavailable), | ||
} | ||
|
||
Ok(result) | ||
Ok(TemplateResult::Template(Box::new(result))) | ||
} else { | ||
Ok(template.into()) | ||
Ok(TemplateResult::Template(Box::new(template.into()))) | ||
} | ||
} | ||
|
||
|
@@ -185,6 +213,27 @@ impl<TAddr: NodeAddressable> TemplateManager<TAddr> { | |
Ok(templates) | ||
} | ||
|
||
pub(super) fn add_pending_template( | ||
&self, | ||
template_address: tari_engine_types::TemplateAddress, | ||
) -> Result<(), TemplateManagerError> { | ||
let template = DbTemplate::empty_pending(template_address); | ||
|
||
let mut tx = self.global_db.create_transaction()?; | ||
let mut templates_db = self.global_db.templates(&mut tx); | ||
match templates_db.get_template(&template.template_address)? { | ||
Some(_) => templates_db.update_template( | ||
&template.template_address, | ||
DbTemplateUpdate::status(TemplateStatus::Pending), | ||
)?, | ||
None => templates_db.insert_template(template)?, | ||
} | ||
|
||
tx.commit()?; | ||
|
||
Ok(()) | ||
} | ||
|
||
pub(super) fn add_template( | ||
&self, | ||
author_public_key: PublicKey, | ||
|
@@ -301,10 +350,34 @@ impl<TAddr: NodeAddressable + Send + Sync + 'static> TemplateProvider for Templa | |
return Ok(Some(template)); | ||
} | ||
|
||
let Some(template) = self.fetch_template(address).optional()? else { | ||
let Some(template_result) = self.fetch_template(address).optional()? else { | ||
return Ok(None); | ||
}; | ||
|
||
debug!(target: LOG_TARGET, "CACHE MISS: Template {}", address); | ||
|
||
// getting template | ||
let template = match template_result { | ||
TemplateResult::Template(template) => template, | ||
TemplateResult::PendingTemplate => { | ||
let start = SystemTime::now(); | ||
let mut template = None; | ||
while template.is_none() { | ||
let elapsed = start.duration_since(UNIX_EPOCH).expect("Time went backwards"); | ||
if elapsed.gt(&self.config.pending_templates_wait_timeout()) { | ||
break; | ||
} | ||
if let Some(TemplateResult::Template(fetched_template)) = self.fetch_template(address).optional()? { | ||
template = Some(fetched_template); | ||
} | ||
// sleeping here to not overload the local database while waiting for the template to be ready | ||
thread::sleep(std::time::Duration::from_millis(100)); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not ideal. Will think about how we should handle pending templates in a transaction. |
||
} | ||
debug!(target: LOG_TARGET, "Failed to fetch template {} within {:?}", address, self.config.pending_templates_wait_timeout()); | ||
template.ok_or(Self::Error::TemplateUnavailable)? | ||
}, | ||
}; | ||
|
||
let loaded = match template.executable { | ||
TemplateExecutable::CompiledWasm(wasm) => { | ||
let module = WasmModule::from_code(wasm); | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
break here?