Archive engine: array archiving

[joshikk]

Hello,
I am trying to archive the synapps sscan.D01DA field in the archiver. The data from the sscan record is archived to the database when the scan is restarted. ie it saves the present data only on start of new scan. I guess this may be due to the sscan timestamp issue, I will try working around this with using another record. I am listing below the webserver output
This problem occures only when ATIME field is not zero.

This is not the problem with other PV's or when ATIME is zero.

Can archiver be configured to archive only on the scan complete DBE_LOG event generated by sscan? can I have epics record with filter for DBE_LOG event?

I need to reduce the data points to less than the MPTS as by default all MPTS are posted.
Can the archiver store arrays with variable size? ie output of subArray record?

Where can I find detailed documentation on Phoebus Archiver?

Regards,
Kuldeep

The truncated webserver output is given below

On completion of the first run of the sscan

Channel Info
Channel scan1.D01DA
Connected Connected
Internal State Connected
Mechanism on change [00:00:10]
Current Value 2021-09-16 09:52:30.171392945 VFloatArray[[-7.151067E-5, -1.5252829E-4, -1.14656985E-4, -1.00940466E-4, -1.0354817E-4, -1.064837E-4, -1.3849884E-4, -1.0485202E-4, -1.0060519E-4, -1.3162941E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, ...], size [1000], NONE(NONE) - NO_ALARM, 2021-09-16T04:22:30.171392945Z] [NONE/NONE]
Last Archived Value 2021-09-16 09:52:30.171392945 VFloatArray[[9.4771385E-6, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, 2.6836991E-5, ...], size [1000], NONE(NONE) - NO_ALARM, 2021-09-16T04:22:30.171392945Z] [NONE/NONE]
Enablement Passive
State Enabled
Queue Len. 0
Queue Avg. 0.0
Queue Max. 2
Capacity 6
Overruns 0

On initiation of the second sscan

Channel Info
Channel scan1.D01DA
Connected Connected
Internal State Connected
Mechanism on change [00:00:10]
Current Value 2021-09-16 10:06:04.438161773 VFloatArray[[-7.151067E-5, -1.5252829E-4, -1.14656985E-4, -1.00940466E-4, -1.0354817E-4, -1.064837E-4, -1.3849884E-4, -1.0485202E-4, -1.0060519E-4, -1.3162941E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, ...] [NONE/NONE]
Last Archived Value 2021-09-16 10:06:04.438161773 VFloatArray[[-7.151067E-5, -1.5252829E-4, -1.14656985E-4, -1.00940466E-4, -1.0354817E-4, -1.064837E-4, -1.3849884E-4, -1.0485202E-4, -1.0060519E-4, -1.3162941E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, -1.2283027E-4, ...] [NONE/NONE]
Enablement Passive
State Enabled
Queue Len. 1
Queue Avg. 0.0
Queue Max. 2
Capacity 6
Overruns 0

[georgweiss]

Strictly speaking the Archiver Appliance is not a CS Studio/Phoebus component/module. For details on the service please see https://slacmshankar.github.io/epicsarchiver_docs/index.html

Phoebus does however include a client for the Archiver Appliance, which in turn depends on the "external" EPICS Java module (org.epics:pbrawclient). Note however that Phoebus may be configured to use other types of archiver data sources, see
https://github.com/ControlSystemStudio/phoebus/blob/master/app/databrowser/src/main/resources/databrowser_preferences.properties, line 59 and onwards.

[joshikk]

Hello, I am not using archiver appliance. I am using this https://control-system-studio.readthedocs.io/en/latest/services/archive-engine/doc/index.html with postgresql database. I guess the issue may be with the timestamp of sscan record. I was wondering of i can find a workaround by explicitly logging on DBE_LOG event Kuldeep

On Thursday, September 16, 2021, Georg Weiss ***@***.***> wrote: Strictly speaking the Archiver Appliance is not a CS Studio/Phoebus

component/module. For details on the service please see https://slacmshankar.github.io/epicsarchiver_docs/index.html

Phoebus does however include a client for the Archiver Appliance, which

in turn depends on the "external" EPICS Java module (org.epics:pbrawclient). Note however that Phoebus may be configured to use some other type of archiver data source, see

https://github.com/ControlSystemStudio/phoebus/blob/master/app/databrowser/src/main/resources/databrowser_preferences.properties, line 59 and onwards.

…

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[georgweiss]

OK, please disregard my comment.

[kasemir]

Both the RDB-based archive engine and the archive appliance can store arrays, but they're meant for keeping the long-term history of for example temperatures.
They're not meant to store experimental data where you "start" an experiment, take some data, "stop" the experiment, and then later want to analyze the data for just that experiment.
For experimental data storage, look at for example the area detector toolset or https://www.aps.anl.gov/BCDA/Data-Acquisition.
As you noted, the generic archivers simply store array samples based on their time stamp. They have no idea if a sample is part of an experiment, there's no support for fetching the data for one specific experiment, there's nothing to reduce the data, since that's not their purpose.