Data Product Catalog prototype #2667

timcanham · 2024-04-08T19:55:32Z


*Related Issue(s)*	#1713
*Has Unit Tests (y/n)*	y
*Documentation Included (y/n)*	y

Change Description

This is a prototype demonstration of Svc/DpCatalog It was used to demonstrate an early release of data products in conjunction with FPP updates and DpManager/DpWriter

Svc/DpCatalog is not complete yet - this version is to capture the demo and allow users to play with it.

Rationale

Get a consolidated set of data product features int he baseline.

Testing/Review Recommendations

The demo can be run and experimented with.

Future Work

Svc/DpCatalog work will be continued with further releases.

timcanham · 2024-04-08T19:56:33Z

Requested @bocchino for changes made to Svc/DpWriter.

Svc/DpCatalog/DpCatalog.cpp

+    void DpCatalog ::
+        fileDone_handler(


Svc/DpCatalog/DpCatalog.cpp

+    void DpCatalog ::
+        START_XMIT_CATALOG_cmdHandler(


Svc/DpCatalog/DpCatalog.cpp

+        this->m_initialized = true;
+    }
+
+    Fw::CmdResponse DpCatalog::doCatalogBuild() {


Svc/DpCatalog/DpCatalog.cpp

+
+    }
+
+    bool DpCatalog::checkInit() {


Svc/DpCatalog/DpCatalog.cpp

+
+    }
+
+    void DpCatalog::sendNextEntry() {


Svc/DpCatalog/DpCatalog.cpp

+    DpCatalog ::
+        DpCatalog(const char* const compName) :


Fw/Dp/DpContainer.hpp

@@ -98,6 +98,9 @@
    //! \return The time tag
    Fw::Time getTimeTag() const { return this->m_timeTag; }

+    //! Get the product state
+    Fw::DpState getState() const { return this->m_dpState; }


Svc/DpCatalog/DpCatalog.hpp

+        // Private data structures
+        // ----------------------------------
+
+        struct DpStateEntry {


Svc/DpCatalog/DpCatalog.cpp

+        this->m_initialized = true;
+    }
+
+    Fw::CmdResponse DpCatalog::doCatalogBuild() {


github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Joshua-Anderson · 2024-04-09T00:02:05Z

Svc/DpCatalog/DpCatalog.cpp

+                    this->m_directories[dir].toChar(),
+                    this->m_fileList[file].toChar()
+                );
+                this->log_ACTIVITY_LO_ProcessingFile(fullFile);


Might be worth lowering the severity of this EVR to diagnostic, sending a event per data product file processed could add up.

Joshua-Anderson · 2024-04-09T00:15:02Z

Is storing transmit state (UNTRANSMITTED/PARTIAL/TRANSMITTED) in the data product header the long term term plan?

It seems like this would force us to re-write data products once we start transmitting them, which could be a bit of a circular dependency. It also could get a bit confusing because the onboard DP file checksum would change after you start transmitting it.

Did we consider having DpCatalog own data product transmit state instead? (I.E. have a catalog file with a list of data products and their transmit states)

This could also provide a bit more flexibility because different projects may want to track data product transmit state differently (I.E. track not just that a data product is partial but how much of the file was sent, or which portions of the file need to be retransmitted).

timcanham · 2024-04-09T14:41:06Z

@Joshua-Anderson The plan is to modify the header as we process the data products, i.e. DpWriter writes it as UNTRANSMITTED, then as DpCatalog starts downlinking it will change to PARTIAL, then TRANSMITTED when done. I'm not sure I see the circular dependency part, so elaborate on that for me.

You are correct that having a separate DP state file (or other variants on the idea) is an option and @bocchino and I discussed it, but I've always preferred files to be self-contained in their state so that the DpCatalog can be simpler in that it discovers state in the files and doesn't have to keep track of things in two places. Just a design preference.

config/DpCfg.hpp

Svc/DpWriter/DpWriter.hpp

bocchino · 2024-04-09T15:28:03Z

The changes to DpWriter look good to me! I just had a couple of comments.

Svc/DpCatalog/DpCatalog.cpp

+        ~DpCatalog()
+    {}
+
+    void DpCatalog::configure(


Svc/DpCatalog/DpCatalog.cpp

+        ~DpCatalog()
+    {}
+
+    void DpCatalog::configure(


Svc/DpCatalog/DpCatalog.cpp

+
+    DpCatalog ::
+        ~DpCatalog()
+    {}


Svc/DpWriter/DpWriter.cpp

@@ -18,10 +18,14 @@
 // Construction, initialization, and destruction
 // ----------------------------------------------------------------------

-DpWriter::DpWriter(const char* const compName) : DpWriterComponentBase(compName) {}
+DpWriter::DpWriter(const char* const compName) : DpWriterComponentBase(compName), m_dpFileNamePrefix() {}


LeStarch

@bocchino would you mark a final review on this demo component? Just clarify your comments were resolved.

LeStarch · 2024-04-25T21:50:52Z

Ref/SignalGen/SignalGen.fpp

+    product get port productGetOut
+
+    @ Data product request port
+    product request port productRequestOut


Why is there bothj get and request on this component and one is left in TODO state?

LeStarch · 2024-04-25T21:51:53Z

Ref/SignalGen/SignalGen.cpp

+            Fw::Success::T status
+        )
+    {
+        // TODO


This should be removed or demonstrated.

I was going to add that update to SignalGen after the demo was merged.

Hold off for a day or so - decided to implement the missing function.

@timcanham this was merged yesterday.

Ah... Okay, I'll do another PR for the update.

bocchino

My comments have been addressed.

timcanham added 28 commits January 19, 2024 13:11

Starting on FDD for DpCatalog

26c8765

More DpCatalog SDD edits

041a450

More refined DpCatalog requirements

c4f6294

Starting prototype code

91a363e

Working on DpCatalog build command

3e7d121

More DpCatalog work

e1a5783

More DpCatalog work

fe96b1a

Merge from devel

666e77b

More work on reading DP headers

f99d851

More work...

4394336

Continued work

f3af664

Spelling fixes

c17652f

More spelling fixes

06c3fa6

First run of init unit test

c80d844

One DP works

2848241

Refactor unit test helpers

7b9be73

Working on multiple DPs

e51a07e

More work on multiple DPs

da0700e

Fixed filesRead bug

0a1176d

Merge from fprime devel

7425848

More merge fixes

1e3452f

Adding DPs to SignalGen

cdd6b71

Merge from F Prime devel

1f0d4ec

Adding DP components

a0748fe

Added DP directory to DpWriter

7753fdd

First working version of Data Products!

977a6a6

Added missing FileDownlink connection

d4a3a0e

Added early proto warning to SDD.

3797067

timcanham added the enhancement label Apr 8, 2024

timcanham requested a review from LeStarch April 8, 2024 19:55

timcanham requested a review from bocchino April 8, 2024 19:56

github-advanced-security bot found potential problems Apr 8, 2024

View reviewed changes

Joshua-Anderson reviewed Apr 9, 2024

View reviewed changes

bocchino reviewed Apr 9, 2024

View reviewed changes

config/DpCfg.hpp Outdated Show resolved Hide resolved

bocchino reviewed Apr 9, 2024

View reviewed changes

Svc/DpWriter/DpWriter.hpp Outdated Show resolved Hide resolved

Switched to Fw::FileNameString

dc5dd7b

github-advanced-security bot found potential problems Apr 10, 2024

View reviewed changes

Svc/DpCatalog/DpCatalog.cpp

~DpCatalog()

{}

void DpCatalog::configure(

Check notice

Code scanning / CodeQL

No raw arrays in interfaces Note

Raw arrays should not be used in interfaces. A container class should be used instead.

timcanham and others added 12 commits April 10, 2024 14:02

Fixed unit test compile

f8ac1cc

Add missing override keyword

018f4ad

Fix unit test override

38a3fd6

Fixed format specifiers

745b19f

Reformat code

d8991d6

Revise DpWriter uts

715cc47

Revise dp writer

52168ce

Revise dp writer

e3bc890

Removed extra ;

e62571c

Merge remote-tracking branch 'upstream/devel' into work/DpCatalog

623653a

Fix markdown broken links

da1065b

Updated SDD to force CI to run again

3c17315

github-advanced-security bot found potential problems Apr 25, 2024

View reviewed changes

LeStarch reviewed Apr 25, 2024

View reviewed changes

bocchino requested review from LeStarch and bocchino April 25, 2024 22:07

bocchino approved these changes Apr 25, 2024

View reviewed changes

LeStarch merged commit 4557066 into nasa:devel Apr 25, 2024
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data Product Catalog prototype #2667

Data Product Catalog prototype #2667

timcanham commented Apr 8, 2024 •

edited

Loading

timcanham commented Apr 8, 2024

github-advanced-security bot left a comment

Joshua-Anderson Apr 9, 2024 •

edited

Loading

Joshua-Anderson commented Apr 9, 2024 •

edited

Loading

timcanham commented Apr 9, 2024

bocchino commented Apr 9, 2024

LeStarch left a comment

LeStarch Apr 25, 2024

LeStarch Apr 25, 2024

timcanham Apr 25, 2024

timcanham Apr 26, 2024

LeStarch Apr 26, 2024

timcanham Apr 26, 2024

bocchino left a comment

Data Product Catalog prototype #2667

Data Product Catalog prototype #2667

Conversation

timcanham commented Apr 8, 2024 • edited Loading

Change Description

Rationale

Testing/Review Recommendations

Future Work

timcanham commented Apr 8, 2024

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Joshua-Anderson Apr 9, 2024 • edited Loading

Choose a reason for hiding this comment

Joshua-Anderson commented Apr 9, 2024 • edited Loading

timcanham commented Apr 9, 2024

bocchino commented Apr 9, 2024

LeStarch left a comment

Choose a reason for hiding this comment

LeStarch Apr 25, 2024

Choose a reason for hiding this comment

LeStarch Apr 25, 2024

Choose a reason for hiding this comment

timcanham Apr 25, 2024

Choose a reason for hiding this comment

timcanham Apr 26, 2024

Choose a reason for hiding this comment

LeStarch Apr 26, 2024

Choose a reason for hiding this comment

timcanham Apr 26, 2024

Choose a reason for hiding this comment

bocchino left a comment

Choose a reason for hiding this comment

timcanham commented Apr 8, 2024 •

edited

Loading

Joshua-Anderson Apr 9, 2024 •

edited

Loading

Joshua-Anderson commented Apr 9, 2024 •

edited

Loading