Validating NeuroKit functionality

[peterhcharlton]

I'd like to ask about the validation of NeuroKit's functionality. By validation, I am referring to the performance of algorithms and their implementations.

I wondered:

• What evidence is there that NeuroKit performs well / as intended? e.g.

  • The toolbox includes a series of tests, which I expect largely test that the code executes as expected, rather than that the results are accurate (although please do correct me if I'm mistaken).
  • There is a benchmarking study of R-peak detection algorithms in the NeuroKit2 documentation; and the performance of the NeuroKit ECG R-peak detector has subsequently been independently assessed here (and found to perform very well).
  • The performance of other algorithms has often been assessed in their original (or subsequent) publications. For instance, the 'Bishop' PPG beat detection algorithm was found to perform well in a recent benchmarking study. However, one should note that just because an algorithm has previously been found to perform well, this does not necessarily indicate that its implementation in NeuroKit performs well. In this example the benchmarking study used a Matlab implementation, which I translated to Python for NeuroKit, so in reality I cannot be entirely confident that the performance of the NeuroKit version is equivalent to that reported in the benchmarking study.

• How important is it to validate NeuroKit's functionality? The NeuroKit benchmarking study of R-peak detection algorithms illustrates a desire amongst NeuroKit contributors to validate NeuroKit's functionality. Furthermore, other toolboxes have been accompanied by validation studies (e.g. here), perhaps indicating the value placed on the validation of algorithms by the wider community.

• What work is being done to validate NeuroKit's functionality? It might be helpful to discuss planned/ongoing work to avoid any duplication, and potentially to harness expertise and resources from across the community to work towards a common goal of validating NeuroKit's functionality.

Many thanks.

[DominiqueMakowski]

Thanks for opening this thead.

Regarding the "benchmarking study" in the documentation, there are ongoing plans with @danibene to update it and improve it. It's tied to a project led by Dani so I'll let her elaborate more if she can (given the scope of the open-source contribution is not yet fully known).

@JanCBrammer was the author of the original peak detection function, so perhaps he's aware of other developments

In general yes, even though NK is made to be a "toolbox" that offers the user a variety of tools (rather than a software that implements / pushes a specific method), it would be indeed nice to have more validation and benchmarking data so that users can make better informed decisions. This would also further reinforce the credibility of NK for bio/health applications

[danibene]

Thanks for opening this thread @peterhcharlton and for adding me to it @DominiqueMakowski. Yes, we have an ongoing project updating the existing benchmarking of peak detection algorithms on ECG data, aiming to identify a well-performing algorithm to be optimized to run in real time. We are planning to publish open-source eventually, but I can’t give an exact timeline for that. Happy to discuss further though and see if it would be possible to combine efforts.
In any case, since the scope of that project is limited to ECG, I think it would be very useful to have similar benchmarking for other signals e.g. with open-access PPG datasets.

[peterhcharlton]

Hi @danibene , that's good to hear - very happy to discuss the ECG benchmarking, do get in touch if it would be helpful.

Thanks for the note about the PPG benchmarking. I'm currently working on development and validation of a refined PPG beat detection algorithm which I hope to contribute shortly. The benchmarking is being done in Matlab (although the algorithm will be in Python), so it's not quite equivalent to the ECG benchmarking study.

More generally, I'd be interested in whether there is a 'wish-list' / 'roadmap' (or similar) for validation in NeuroKit. Whilst I acknowledge @DominiqueMakowski 's comment that NK is a toolbox rather implementations of specific methods, I personally think validation / benchmarking nonetheless is probably an important component, which may warrant a discussion to identify the most pressing aspects to be validated, and to prompt the community to consider contributing to these tasks.

[JanCBrammer]

@peterhcharlton, I've benchmarked the ECG R-peak detector in https://joss.theoj.org/papers/10.21105/joss.02621, using https://github.com/JanCBrammer/biopeaks/blob/main/biopeaks/benchmarks/benchmark_ECG_stream.py.

The ECG R-peak detector was originally implemented here: https://github.com/JanCBrammer/biopeaks/blob/4340fea58161aa7fca6221db975034067589e7bc/biopeaks/heart.py#L14

and subsequently used for NeuroKit as well:

NeuroKit/neurokit2/ecg/ecg_findpeaks.py

Line 239 in 1aa8dee

def _ecg_findpeaks_neurokit(

.

[peterhcharlton]

Great - thanks for the information. I've now proposed a slight change to the ecg_peaks.py documentation in this PR, mentioning the JOSS benchmarking paper. Personally I think it could be helpful given the previous questions about this method. In particular, @JanCBrammer , please do have a look and see if you are happy with this. Thanks very much.