Optimize searching for positioning ATX (#5952)

Searching for positioning ATX is slow because:
- the SQL query is slow
- it usually happens at the same time as many ATXs are being inserted into the DB (the poet CG)

.github/workflows/ci.yml

@@ -34,7 +34,7 @@ concurrency:
 jobs:
   ## stage 0: check which files were changed
   filter-changes:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     outputs:
       nondocchanges: ${{ steps.filter.outputs.nondoc }}
     steps:
@@ -55,7 +55,7 @@ jobs:
   ## these run on all pushes to all pull requests, all branches
   ## note that secrets may not be accessible in this phase
   quicktests:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: filter-changes
     if: ${{ needs.filter-changes.outputs.nondocchanges == 'true' }}
     # should not take more than 2-3 mins
@@ -87,7 +87,7 @@ jobs:
           make test-generate
 
   lint:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: filter-changes
     if: ${{ needs.filter-changes.outputs.nondocchanges == 'true' }}
     # should not take more than 15-16 mins
@@ -126,17 +126,17 @@ jobs:
       fail-fast: true
       matrix:
         os:
-          - ubuntu-latest
+          - ubuntu-22.04
           - [self-hosted, linux, arm64]
           - macos-13
           - [self-hosted, macOS, ARM64, go-spacemesh]
-          - windows-latest
+          - windows-2022
     steps:
       - name: Add OpenCL support - Ubuntu
-        if: ${{ matrix.os == 'ubuntu-latest' }}
+        if: ${{ matrix.os == 'ubuntu-22.04' }}
         run: sudo apt-get update -q && sudo apt-get install -qy ocl-icd-opencl-dev libpocl2
       - name: disable Windows Defender - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           Set-MpPreference -DisableRealtimeMonitoring $true
       - name: checkout
@@ -172,17 +172,17 @@ jobs:
       fail-fast: true
       matrix:
         os:
-          - ubuntu-latest
+          - ubuntu-22.04
           - [self-hosted, linux, arm64]
           - macos-13
           - [self-hosted, macOS, ARM64, go-spacemesh]
-          - windows-latest
+          - windows-2022
     steps:
       - name: Add OpenCL support - Ubuntu
-        if: ${{ matrix.os == 'ubuntu-latest' }}
+        if: ${{ matrix.os == 'ubuntu-22.04' }}
         run: sudo apt-get update -q && sudo apt-get install -qy ocl-icd-opencl-dev libpocl2
       - name: disable Windows Defender - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           Set-MpPreference -DisableRealtimeMonitoring $true
       - name: checkout
@@ -216,16 +216,16 @@ jobs:
       fail-fast: true
       matrix:
         os:
-          - ubuntu-latest
+          - ubuntu-22.04
           - [self-hosted, linux, arm64]
           - macos-13
           - [self-hosted, macOS, ARM64, go-spacemesh]
-          - windows-latest
+          - windows-2022
     steps:
       # as we use some request to localhost, sometimes it gives us flaky tests. try to disable tcp offloading for fix it
       # https://github.com/actions/virtual-environments/issues/1187
       - name: disable TCP/UDP offload - Ubuntu
-        if: ${{ matrix.os == 'ubuntu-latest' }}
+        if: ${{ matrix.os == 'ubuntu-22.04' }}
         run: |
           sudo ethtool -K eth0 tx off
           sudo ethtool -K eth0 rx off
@@ -236,11 +236,11 @@ jobs:
           sudo sysctl -w net.link.generic.system.hwcksum_tx=0
           sudo sysctl -w net.link.generic.system.hwcksum_rx=0
       - name: disable TCP/UDP offload - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           Disable-NetAdapterChecksumOffload -Name * -TcpIPv4 -UdpIPv4 -TcpIPv6 -UdpIPv6
       - name: disable Windows Defender - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           Set-MpPreference -DisableRealtimeMonitoring $true
       - name: checkout
@@ -262,10 +262,10 @@ jobs:
           go-version: ${{ env.go-version }}
           cache: ${{ runner.arch != 'arm64' }}
       - name: Add OpenCL support - Ubuntu
-        if: ${{ matrix.os == 'ubuntu-latest' }}
+        if: ${{ matrix.os == 'ubuntu-22.04' }}
         run: sudo apt-get update -q && sudo apt-get install -qy ocl-icd-opencl-dev libpocl2
       - name: Add OpenCL support - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: choco install opencl-intel-cpu-runtime
       - name: setup env
         run: make install
@@ -295,7 +295,7 @@ jobs:
       - build-tools
       - build
       - unittests
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     env:
       # short-circuit success if no non-doc files were modified
       # this is the easiest way to access success/failure state of previous jobs in this workflow

.github/workflows/coverage.yml

@@ -10,7 +10,7 @@ on:
 
 jobs:
   filter-changes:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     outputs:
       nondocchanges: ${{ steps.filter.outputs.nondoc }}
     steps:
@@ -28,7 +28,7 @@ jobs:
               - '!**/*.md'
 
   coverage:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: filter-changes
     if: ${{ needs.filter-changes.outputs.nondocchanges == 'true' }}
     steps:

.github/workflows/dockerhub.yml

@@ -20,12 +20,20 @@ on:
         default: ''
 jobs:
   dockerpush:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
     - name: checkout
       uses: actions/checkout@v4
       with:
         ssh-key: ${{ secrets.GH_ACTION_PRIVATE_KEY }}
+
+    - uses: extractions/netrc@v2
+      with:
+        machine: github.com
+        username: ${{ secrets.GH_ACTION_TOKEN_USER }}
+        password: ${{ secrets.GH_ACTION_TOKEN }}
+      if: vars.GOPRIVATE
+
     - name: Build docker images
       run: |
         export VERSION="${{ github.ref_name }}"

.github/workflows/release.yml

@@ -13,30 +13,30 @@ jobs:
     strategy:
       matrix:
         include:
-          - os: ubuntu-latest
+          - os: ubuntu-22.04
             outname_sufix: "linux-amd64"
           - os: [self-hosted, linux, arm64]
             outname_sufix: "linux-arm64"
           - os: macos-13
             outname_sufix: "mac-amd64"
           - os: [self-hosted, macOS, ARM64, go-spacemesh]
             outname_sufix: "mac-arm64"
-          - os: windows-latest
+          - os: windows-2022
             outname_sufix: "win-amd64"
     steps:
       - shell: bash
         run: echo "OUTNAME=go-spacemesh-${{ github.ref_name }}-${{ matrix.outname_sufix }}" >> $GITHUB_ENV
 
       - name: Install dependencies in windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: choco install make wget zip
 
       - name: Add OpenCL support - Ubuntu
-        if: ${{ matrix.os == 'ubuntu-latest' }}
+        if: ${{ matrix.os == 'ubuntu-22.04' }}
         run: sudo apt-get update -q && sudo apt-get install -qy ocl-icd-opencl-dev libpocl2
 
       - name: disable Windows Defender - Windows
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           Set-MpPreference -DisableRealtimeMonitoring $true
 
@@ -99,13 +99,13 @@ jobs:
         run: brew install coreutils
 
       - name: Calculate the hashsum of the zip file
-        if: ${{ matrix.os != 'windows-latest' }}
+        if: ${{ matrix.os != 'windows-2022' }}
         shell: bash
         run: |
           sha256sum ${{ env.OUTNAME }}.zip | awk '{ print $1 }' > sha256-${{ matrix.outname_sufix }}.txt
 
       - name: Calculate the hashsum of the zip file (Windows)
-        if: ${{ matrix.os == 'windows-latest' }}
+        if: ${{ matrix.os == 'windows-2022' }}
         run: |
           (Get-FileHash ${{ env.OUTNAME }}.zip -Algorithm SHA256).Hash > sha256-${{ matrix.outname_sufix }}.txt
 
@@ -117,7 +117,7 @@ jobs:
           retention-days: 5
 
   release:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: build-and-upload
     steps:
       - name: Download the artifacts

.github/workflows/systest.yml

@@ -40,7 +40,7 @@ concurrency:
 
 jobs:
   filter-changes:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     outputs:
       nondocchanges: ${{ steps.filter.outputs.nondoc }}
     steps:
@@ -53,7 +53,7 @@ jobs:
               - '!**/*.md'
 
   systest:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     if: ${{ needs.filter-changes.outputs.nondocchanges == 'true' }}
     needs:
       - filter-changes
@@ -92,6 +92,13 @@ jobs:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
+      - uses: extractions/netrc@v2
+        with:
+          machine: github.com
+          username: ${{ secrets.GH_ACTION_TOKEN_USER }}
+          password: ${{ secrets.GH_ACTION_TOKEN }}
+        if: vars.GOPRIVATE
+
       - name: Push go-spacemesh build to docker hub
         run: make dockerpush
 
@@ -103,6 +110,13 @@ jobs:
         shell: bash
         run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
 
+      - uses: extractions/netrc@v2
+        with:
+          machine: github.com
+          username: ${{ secrets.GH_ACTION_TOKEN_USER }}
+          password: ${{ secrets.GH_ACTION_TOKEN }}
+        if: vars.GOPRIVATE
+
       - name: Build tests docker image
         run: make -C systest docker
 
@@ -140,7 +154,7 @@ jobs:
     needs:
       - filter-changes
       - systest
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     env:
       # short-circuit success if no non-doc files were modified
       status: ${{ (needs.filter-changes.outputs.nondocchanges == 'false' || needs.systest.result == 'success') && 'success' || 'failure' }}

CHANGELOG.md

@@ -9,6 +9,12 @@ See [RELEASE](./RELEASE.md) for workflow instructions.
 With this release the minimum supported version for Intel Macs is now macOS 13 (Ventura) and for Apple Silicon Macs it
 is macOS 14 (Sonoma) or later ([#5879](https://github.com/spacemeshos/go-spacemesh/pull/5879)).
 
+This update removes migration code for go-spacemesh databases created with versions before v1.5.0.
+Upgrading to this version requires going through v1.5.x first. Removed migrations for:
+
+* legacy keys in the post data directory ([#5907](https://github.com/spacemeshos/go-spacemesh/pull/5907)).
+* ATX blob separation and always populating nonce column in atxs ([#5942](https://github.com/spacemeshos/go-spacemesh/pull/5942))
+
 ### Highlights
 
 ### Features
@@ -22,7 +28,79 @@ is macOS 14 (Sonoma) or later ([#5879](https://github.com/spacemeshos/go-spaceme
 
 * [#5877](https://github.com/spacemeshos/go-spacemesh/pull/5877) Fix verifying ATX chain after checkpoint.
 
-## (v1.5.0)
+* [#5882](https://github.com/spacemeshos/go-spacemesh/pull/5882) Use backoff during routing discovery.
+  This should reduce network load from the peer discovery mechanism when it's enabled.
+
+* [#5888](https://github.com/spacemeshos/go-spacemesh/pull/5888) Handle DHT discovery startup errors properly
+
+* [#5932](https://github.com/spacemeshos/go-spacemesh/pull/5932) Fix caching malfeasance when processing new proofs
+
+* [#5943](https://github.com/spacemeshos/go-spacemesh/pull/5943) Fix timing out querying proof in 1:N in a presence of
+  a broken Poet.
+
+  Previously, every identity waited for the full timeout time (~20 minutes) before giving up.
+
+* [#5958](https://github.com/spacemeshos/go-spacemesh/pull/5958) Fix node incorrectly detecting a remote smeshing setup
+  as supervised.
+
+  Ensure that your key file in `data/identities` is named `local.key` if you run a supervised node or with the change
+  the node will not start.
+
+* [#5965](https://github.com/spacemeshos/go-spacemesh/pull/5965) Start considering ballots that still contain the
+  deprecated inlined activeset as invalid. go-spacemesh references the active set via hash since v1.3.0, and has been
+  pruning the data of old ballots since then as well.
+
+* [#5927](https://github.com/spacemeshos/go-spacemesh/pull/5927) Fixed vulnerability in the way a node handles incoming
+  ATXs. This vulnerability allows an attacker to claim rewards for a full tick amount although they should not be
+  eligible for them.
+
+## Release v1.5.4
+
+### Improvements
+
+* [#5963](https://github.com/spacemeshos/go-spacemesh/pull/5963) Increase supported number of ATXs to 5.5 Mio.
+
+* [#5952](https://github.com/spacemeshos/go-spacemesh/pull/5952) Optimized searching for a positioning ATX.
+  Instead of a slow database query the ATX builder now uses the in-memory ATX store to pick a positioning ATX.
+
+## Release v1.5.3
+
+### Improvements
+
+* [#5929](https://github.com/spacemeshos/go-spacemesh/pull/5929) Fix "no nonce" error when persisting malicious
+  (initial) ATXs.
+
+* [#5930](https://github.com/spacemeshos/go-spacemesh/pull/5930) Check if identity for a given malfeasance proof
+  exists when validating it.
+
+* [#5923](https://github.com/spacemeshos/go-spacemesh/pull/5923) Fix high memory consumption and performance issues
+  in the proposal handler.
+
+## Release v1.5.2-hotfix1
+
+This release includes our first CVE fix. A vulnerability was found in the way a node handles incoming ATXs. We urge all
+node operators to update to this version as soon as possible.
+
+### Improvements
+
+* Fixed a vulnerability in the way a node handles incoming ATXs. This vulnerability allows an attacker to claim rewards
+  for a full tick amount although they should not be eligible for them.
+
+## Release v1.5.2
+
+### Improvements
+
+* [#5904](https://github.com/spacemeshos/go-spacemesh/pull/5904) Avoid repeated searching for positioning ATX in 1:N
+
+* [#5911](https://github.com/spacemeshos/go-spacemesh/pull/5911) Avoid pulling poet proof multiple times in 1:N setups
+
+## Release v1.5.1
+
+### Improvements
+
+* [#5896](https://github.com/spacemeshos/go-spacemesh/pull/5896) Increase supported number of ATXs to 4.5 Mio.
+
+## Release v1.5.0
 
 ### Upgrade information
 
@@ -39,8 +117,6 @@ coins. Fixes an oversight in the genesis VM implementation.
 * [#5791](https://github.com/spacemeshos/go-spacemesh/pull/5791) Speed up ATX queries.
   This also fixes ambiguity of nonces for equivocating identities.
 
-* [#5856](https://github.com/spacemeshos/go-spacemesh/pull/5856) Bump github.com/spacemeshos/api/release/go to v1.37.0.
-
 ## Release v1.4.6
 
 ### Improvements

Dockerfile

@@ -44,7 +44,7 @@ RUN make get-libs
 COPY go.mod .
 COPY go.sum .
 
-RUN go mod download
+RUN --mount=type=secret,id=mynetrc,dst=/root/.netrc go mod download
 
 # Here we copy the rest of the source code
 COPY . .