Pass in last_id explicity

[Swoorup]

Currently it seems the only reason for SequenceProperties to be mutable is that it needs to be aware of last_timestamp, and sequence when generating new ids.

If you need to have a multi-threaded sequence generator this would require to have an Arc<Mutex<SequenceProperties>> due to it.

Would suggest to make changes or expose a new api so that perhaps, you can pass the last_id seperately as immutable, so perhaps AtomicU64 could be used instead of the Mutex?

[drconopoima]

Hi @Swoorup. Thanks for the suggestion. I'm wanting a new interface for this anyways, but I think we could also take advantage of the pattern for interior mutability. I'm making some progress in branch

Issue1_interior_mutability_last_current_datetime_partial_cached

https://github.com/drconopoima/sequence-generator-rust/tree/Issue1_interior_mutability_last_current_datetime_partial_cached

[Swoorup]

@drconopoima

Sounds good , I am picking rust again for a project of mine, and this is like the first things I need. 😄
I think it better to remove any mutable fields from SequenceProperties and instead keep the struct as immutable.

Also doesn't seem much reason to keep the following properties?

sequence,
current_timestamp 
last_timestamp
partial_cached_id

So generate id would look like

pub fn generate_id(
    properties: &SequenceProperties,
    last_id: u64
) -> Result<u64, SequenceGeneratorSystemTimeError> 
{
	// decode the last_timestamp and sequence from last_id ?
}

This way, it's up to the user to how to handle multithreading, atomics(maybe unavailablee on some platforms) or mutex.

[drconopoima]

@Swoorup
From version 0.3.1, the SequenceProperties struct doesn't need to be mutable. The private parameters are still present, for last_timestamp, current_timestamp, sequence, partial_cached_id, but the implementation details do not leak as they are handled automagically with interior mutability.

There is an issue with generating an ID from a pre-existing ID, we would take quite a leap of faith to understand the ID that was passed contains the same bit numbers for all sections, the timestamp, node_id, sequence_id, and the maximum value for the sequence. Even if this applies, the clock could change between the generated previous ID and the moment a new one is calculated. The private fields are keeping track of those corner cases.

I wanted to change the interface more in the sense of allowing the creation of empty properties struct without any bits setup, and allowing properties to be set up by methods independently, counting degrees of liberty until we have enough properties to derive all others.

I don't know if the changes in 0.3.1 are already enough for your case, or we could work something out. But I think passing a previous ID could be confusing and error-prone as an interface due to the fully configurable bit parameters (in other Snowflake implementations that have bit sizes hardcoded it would be quite convenient)

[Swoorup]

Fair point, although
you could also have a seperate struct which contains the properties + mutable data, but either or.

In that case, wouldn't you need to use either Atomics, Arc mutex instead of Refcell? I don't think sure if Send and Sync are derived using Cell, Rc.

[drconopoima]

The usage I need out of it would ensure different worker threads use independent copies with their own node IDs, so I can identify the worker by the sequence ID. If you can provide a sample code requiring Send+Sync I can consider migrating to Arc/Atomic.

[Swoorup]

I don't have a simple example to share, but I am using it to id as well as timestamp messages coming from different sources using tokio.

fn println_id(new_id: i64) {
  let thread_id = thread::current().id().as_u64();
  let stdout_ref = &std::io::stdout();
  let mut lck = stdout_ref.lock();
  writeln!(&mut lck, "Thread number {}, Got Id: {}", thread_id, new_id).unwrap();
}

#[tokio::main]
async fn main() {
  let gen = dist_sequencial_id::Generator::new();
  thread::scope(|s| {
    (1..1000)
      .map(|_| {
        s.spawn(|| {
          let new_id = gen.generate_i64();
          println_id(new_id);
        })
      })
      .for_each(|j| j.join().unwrap());
  });
}

And wrapping the generator as

  const CUSTOM_EPOCH_SINCE_UNIX: std::time::Duration =
    std::time::Duration::from_millis(1577836800000u64);
  // Current worker/node ID
  const NODE_ID: u16 = 500;
  /// Operate in milliseconds (10^3 microseconds)
  const MICROS_TEN_POWER: u8 = 1;
  /// 10-bit node/worker ID
  const NODE_ID_BITS: u8 = 8;
  /// 12-bit sequence
  const SEQUENCE_BITS: u8 = 12;
  /// unused (sign) bits at the start of the ID. 1 or 0 generally
  const UNUSED_BITS: u8 = 1;
  /// initial time in nanoseconds for exponential backoff wait after sequence is exhausted
  const COOLDOWN_NS: u64 = 1500;
  
  pub struct Generator {
    properties: RwLock<SequenceProperties>,
    /// 0 if uninitialized
    last_id: AtomicI64,
  }

  impl Generator {
    pub fn new() -> Self {
      let properties = SequenceProperties::new(
        std::time::UNIX_EPOCH + CUSTOM_EPOCH_SINCE_UNIX,
        NODE_ID_BITS,
        NODE_ID,
        SEQUENCE_BITS,
        MICROS_TEN_POWER,
        UNUSED_BITS,
        COOLDOWN_NS,
      );

      Self {
        properties: RwLock::new(properties),
        last_id: AtomicI64::new(0i64),
      }
    }

    pub fn last_id(&self) -> i64 {
      self.last_id.load(Ordering::Relaxed)
    }

    pub fn decode_timestamp(&self, id: i64) -> Timestamp {
      let properties = &self.properties.read();
      let timestamp_micros = decode_id_unix_epoch_micros(id as u64, properties);
      Timestamp::from_micros(timestamp_micros as i64)
    }

    pub fn decode_sequence(&self, id: i64) -> u16 {
      let properties = &self.properties.read();
      decode_sequence_id(id as u64, properties)
    }

    pub fn last_sequence_and_timestamp(&self) -> (Timestamp, u16) {
      let last_id = self.last_id() as u64;
      let properties = &self.properties.read();
      let timestamp_micros = decode_id_unix_epoch_micros(last_id, properties);
      let sequence = decode_sequence_id(last_id, properties);
      let timestamp = Timestamp::from_micros(timestamp_micros as i64);
      (timestamp, sequence)
    }

    pub fn generate_i64(&self) -> i64 {
      let properties: &mut SequenceProperties = &mut self.properties.write();
      let id = generate_id(properties).unwrap() as i64;
      self.last_id.swap(id, Ordering::Relaxed);
      id as i64
    }
  }