2024-10-14 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, Month DD, YYYY - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Karl Moll	FINOS	@karlmoll
Alvin Shih	Morgan Stanley	@alvin-c-shih
Peter Gyorok	KX Systems	@gyorokpeter
Rian Ó Cuinneagáin	KX Systems	@rianoc
Gary Davies	Data Intellect	@drgdavies
Aaron Davies	Morgan Stanley	@adavies42

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

• (5 min) Convene, roll call, welcome new people.

• Display FINOS Antitrust Policy summary slide

• Review Meeting Notices (see above)

• (5 min) Approve previous meeting minutes.

  • Approved by acclamation.

• (10 min) Any interesting takeaways from KX Meetup?

  • See 2024-10-14 - KDB Project Call Meeting Minutes #112 (comment) for links to Meetup videos.

  • A First Look at PyKX 3.0 (54 minutes)

    • Towards the end, talks about how it makesq more learnable for people coming from pure Python.
    • Python objects with chained calls.
    • Launch the tick architecture from Jupyter using Python objects!

  • Detecting anomalies and patterns with kdb+ (12 minutes)

    • Can define shapes to look for.
      • eg. Technical analysis head and shoulders shape.
    • Doesn't require pre-training / indexing.

  • Native code tips and tricks with kdb+ (7 minutes)

    • Citadel qcpp.
    • Serialize a variety of C++ data structures to q wire format without first populating struct k0.
    • Speed gains by using compound keys in C.

• (5 min) STAC Research events this week

  • AI STAC dev on Wedesday, October 16th

    • https://stacresearch.com/aistacdevNYC
    • Hands-on workshops related to AI.
    • Features KDB.AI workshop from 1-5pm.

  • STAC Summit on Thursday, October 17th

    • https://stacresearch.com/fall2024nyc
    • Presentations on latency, throughput, and AI.

• (5 min) Anybody familiar with Weka?

  • https://www.weka.io/company/partners/technology-alliance-partners/nvidia/
  • https://stacresearch.com/news/KDB210507
  • Curious if Weka is suitable for transactional workloads.

• (5 min) .kxi.packages.load[package;version;kwargs]

  • https://code.kx.com/insights/1.9/api/packaging/q/packages.html
  • Worth pushing as a standard for package loading?
    • Specific to Insights.
    • Specific entry points for Storage Manager, etc.
  • pip for q packages?
    • Anaconda supports pre-built binaries.
    • See "Platform Wheels"
      • https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#platform-wheels

• (10 min) Does kdb Insights use of hidden code slow adoption?

  • https://kx.com/blog/kdb-q-insights-scripting-with-q/#:~:text=Copy-,Hiding%20Source%20code,-We%20may%20wish

  • Protects IP, but does KX offer responsive customization and do they want to raise the barrier to entry?

  • No way to submit a fix. (Many examples of improvements to q.k from the community.)

  • PyKX respects PYTHONPATH and able to find related packages that way.

• (5 min) Merged HDBs?

  • .Q.lo

    • Load a database without changing directory and/or loading scripts in the database (since 4.1t 2023.03.01).
    • A stepping stone to merged HDBs?
    • More for loading from object stores since you can't cd(1) into them.

  • .Q.li / .Q.bvi

    • Incremental HDB reloads.
    • Available from September release of 4.1.
    • Doesn't work for linked columns.

• (5 min) Any word on whether KX is trying this out?

  • https://github.blog/ai-and-ml/llms/customizing-and-fine-tuning-llms-what-you-need-to-know/

  • Anyone curating a list of repos that are suitably licensed to be used as training data?

    • https://github.com/search?q=language%3Aq&type=Repositories&ref=advsearch&l=q&l=

  • Linux Foundation still working on licenses that grant permission to train LLMs on source code.

• (5 min) Try some q/k tasks with Sourcegraph Cody powered by Claude 2.5 Sonnet.

  • Seems better on parsing space-delimited file.
  • Still fails on query that returns trades with price higher than average trade price.
    • Should use fby.

• (5 min) AOB, Q&A, Adjourn.

Join Zoom Meeting

• https://zoom.us/j/94523415326?pwd=NkJDaUtUUlZTcHVwcUJHdG0yRFJUdz09
• Meeting ID: 945 2341 5326
• Passcode: 361990
• Find your local number: https://zoom.us/u/abTtYRZ0B6

[rianoc-kx]

Rian Ó Cuinneagáin / KX

[rdffy]

Ross Duffy / Morgan Stanley

[drgdavies]

Gary Davies / Data Intellect

[gyorokpeter]

Péter Györök / KX

[karlmoll]

Karl Moll / FINOS

[adavies42]

Aaron Davies/Morgan Stanley

[alvin-c-shih]

Alvin Shih / Morgan Stanley

[rianoc-kx]

Meetup videos:

• Detecting anomalies and patterns with kdb+ https://www.youtube.com/watch?v=hhjioixyk5c
• Pivoting into KX Dashboards https://www.youtube.com/watch?v=ikMfvRGrY6g
• KX: past, present, and future https://www.youtube.com/watch?v=NcakVLOAkMU
• Native code tips and tricks with kdb+ https://www.youtube.com/watch?v=eEAClMWWmdY

Other:

• A First Look at PyKX 3.0 https://www.youtube.com/watch?v=8pqpQdCpJMg
• ML Toolkit 4.0.0 supports PyKX and EmbedPy https://github.com/KxSystems/ml/releases/tag/4.0.0