Things to test or do before next apio release (2024)

[zapta]

@Obijuan, @cavearr, this issue lists functionality of apio that may have changed but was not tested, for example because lack of boards or platforms. Please make sure to revisit and test it before the next release.

I will add comments to this thread as we go.

[zapta]

'apio upload' command with a tinyprog board.

To Test

Run an apio upload command with a tinyprog board. Verify that the progress percentage report stays on a single line instead of scrolling down.

Notes

The scons filter was changed to preserve formatting and empty lines and this may cause the percentage report to scroll down. If this is the case, the likely fix would be to add erasure of a second line in the tinyprog filter.

apio/apio/managers/scons_filter.py

Line 288 in 5c47620

print(CURSOR_UP + ERASE_LINE, end="", flush=True)

print(CURSOR_UP + ERASE_LINE + CURSOR_UP + ERASE_LINE, end="", flush=True)

[zapta]

'apio upload' command with a fumu board.

To Test

Run an apio upload command with a fumo board. Verify that the progress percentage report stays on a single line instead of scrolling down.

Notes

The scons filter was changed to preserve formatting and empty lines and this may cause the percentage report to scroll down. If this is the case, the likely fix would be to add erasure of a second line in the fumu filter.

apio/apio/managers/scons_filter.py

Line 260 in 5c47620

print(CURSOR_UP + ERASE_LINE, end="", flush=True)

print(CURSOR_UP + ERASE_LINE + CURSOR_UP + ERASE_LINE, end="", flush=True)

[cavearr]

Ok @zapta ! good work! i'm try to prepare a rig with some fpgaboards to automtize the checks, i think could be very interesting

[zapta]

Test all commands with a Gowin project

Currently we have test projects, with testbenches, for ICE40 and ECP5 (see below) but not for Gowin such that some of the development we are doing is with no testing, even not manual.

https://github.com/FPGAwars/apio/tree/develop/test-examples/TB/Alhambra-II/icestudio/ledon

https://github.com/FPGAwars/apio/tree/develop/test-examples/TB/ColorLight-5A-75B-V8/icestudio/button

[cavearr]

I'm working on gowin fpgas support i'm push very soon.

[zapta]

Address the FTDI driver change on windows

Windows that upgraded to the new release may need to change the ftdi driver they installed using zadig. We should test this upgrade path, streamline it with code if possible (e.g. provide a warning if we see an old driver) and include that in the release notes.

[cavearr]

Hi @zapta! with the developer version (apio upstream) zadig does't launch , has you changed the apio command to enable the driver?

[zapta]

@cavearr, it does launch on my windows 10. For testing, try also 'apio raw zadig --env'.

Are you sure that windows doesn't ask you for permission to elevate zadig to high privileges? Look at the task bar at the bottom of the screen, it may be blinking.

[cavearr]

i'm updating icestudio, apio works well, thanks!

[cavearr]

Hi @zapta the command that fails from Icestudio is "apio install drivers" , now, what is the equivalent?

Thanks!

[zapta]

@cavearr, apio install drivers installs the drivers package which is available only on windows.

The new equivalent is apio packages --install drivers. Another option is to run apio packages --install to install all the packages that are applicable to this platform which is more cross platform from user's perspective.

apio/apio/resources/packages.json

Line 82 in f0aca51

"windows_amd64"

[zapta]

Test migration to the new version

Some of the changes in this version are

• apio.ini is required.
• Packages added/removed/changed.
• Windows requires a different driver (WinUsb vs libusbk).
• Artifact files are now in _build instead of the src directory.
• Commands added/removed/changed.
• Other changes we can't recall.

This entry is for the testing of the transition to the new apio version. The starting point can be, the user has a 'non-clean' working project (without apio.ini? using a custom board? using icestudio?), and release notes, and needs to migrate to the new version. This should be tested and tweaked to achieve a high success rate.

If needed, we can add additional code to simplify the migration. So far we have:

• apio clean also cleans old artifact files in the src directory.
• apio packages --fix followed by apio packages --install --force should take care of the packages changes.

[cavearr]

Hi @zapta ! i'm working in icestudio migration in a couple of days (along the weekend as most) i'll can test it.

I think all the tests should be formed in apio alone, and icestudio, independently.

The first test i'll do with alhambra-ii board but once all works i'll test with some ecp5 boards , gowin boards (por apicula support), and custom boards to test raw commands.

[zapta]

Hi @cavearr, you are right, apio should be tested independently.

Your plan sounds great. On what platforms can you test?

Also, I suggest to test only projects with apio.ini (that is, no --board, --fpga, --size, etc flags) since this will be the coming release.

If you encounter any problem with apio or have some wish, please let me know and I will fix it.

[cavearr]

Hi @zapta, the plan is once the development is finished, I will test it on Linux, Linux Arm64 and Osx Arm. @Democrito will test it on Windows with the Alhambra-ii

I will try the Alhambra-ii, colorlight-i5, radiona ulx3s, tinyfpga, icebreaker, fomu and for apicula on the tang nano 1k and 9k

Look forward to some fun days XD

[zapta]

Hi @cavearr, that's exciting!

You don't need to test the deprecated commands since they will go away anyway.

Deprecated commands:
  modify     [Depreciated] Modify the apio.ini project file.
  time       [Depreciated] Report design timing.
  verify     [Depreciated] Verify the verilog code.
  install    [Depreciated] Install apio packages.
  uninstall  [Depreciated] Uninstall apio packages.

[zapta]

@cavearr, also no need to test command lines flags that are marked as DEPRECATED since they will go away soon. You can find them in the command --help text. For example, the build command has 6 deprecated options:

$ apio build --help
Usage: apio build [OPTIONS]

...

Options:
  -p, --project-dir path  Set the root directory for the project.
  -v, --verbose           Show detailed output.
  --verbose-yosys         Show detailed yosys output.
  --verbose-pnr           Show detailed pnr output.
  -t, --top-module name   [DEPRECATED] Set the top level module name.
  -b, --board str         [DEPRECATED] Set the board.
  --fpga str              [DEPRECATED] Set the FPGA.
  --size str              [DEPRECATED] Set the FPGA size (1k/8k).
  --type str              [DEPRECATED] Set the FPGA type (hx/lp).
  --pack str              [DEPRECATED] Set the FPGA package.
  -h, --help              Show this message and exit.

[zapta]

Verify packages.json and distribution.json

During development I sometimes point a package to a dev version or to my own temp repo.

This item is for verifying the package sources and versions in packages.json() and distribution.json().

[cavearr]

Hi @zapta ! i'm going step by step with the tests, i'm posting you here doubts ok? if you prefer that i open new issues tell me how do you want to manage it.

• I have doubts if you remove this parameteR:
  apio examples -d Alhambra-II/Blinky

in the new way i'm trying:

apio packages --install examples

This works as spected and download the full example dir in .apio dir but i ask you if there is a way to install examples in particular like the old way.

In other hand i'm asking you for a command to launch zadig in windows. In some tests , i am founding that in some windows is necesary launch zadig and install winusb from there, and now apio not launch zadig or i don't know how (i don't find a pattern, could be depending the driver for the usb port, for this is important to have an option to launch zadig).

Thanks!

[zapta]

Hi @cavearr, posting here is ok.

1. apio examples -d Alhambra-II/Blinky copies an example from the installed examples package to the user's directory and I think it works the same way as it used to be. That is, it requires the entire 'examples' package to be installed, as it used to be.

2. apio packages --install examples installs the examples packages and it replaces the old apio install examples' The functionality is the same as the old way, just the command changed.

3. Regarding zadig, when you run apio drivers --ftdi-install on windows, does it start zadig? If not, do you see some blinking task at the bottom of the screen? Also, try apio raw zadig. Does it start zadig (again, look for blinking at the bottom of the screen).

[zapta]

When I start zadig from apio I get this blinking at the bottom of the screen (play the video below). When I click on it windows asks me for permission to run zadig. This is because zadig needs administrator privileges. This is on windows 10.

zadig.mp4

[cavearr]

Hi @zapta ! thanks for your sooner reply! is strange ecause i know the "blinking" popup for permissions, but this doesn't work, could be a problem in my setup, i'll review it, don't worry.

Thanks for your detailed reponse!

[zapta]

Try to see if you have the zadig.exe in a package installed under ~/.apio/packges. If you find it, try to run it from a python script, similar to the ay apio does in the link below. ``` import os os.system("<path to zadig.exe>") ``` here https://github.com/FPGAwars/apio/blob/1b28d89205ad8627b2107d3a1df198d169bd8e63/apio/managers/drivers.py#L476

…

On Sat, Nov 30, 2024 at 12:08 PM Carlos Venegas Arrabé < ***@***.***> wrote: Hi @zapta <https://github.com/zapta> ! thanks for your sooner reply! is strange ecause i know the "blinking" popup for permissions, but this doesn't work, could be a problem in my setup, i'll review it, don't worry. Thanks for your detailed reponse! — Reply to this email directly, view it on GitHub <#444 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQVMQLGEFXXNW6XUP2KAPL2DILKVAVCNFSM6AAAAABQHQ2PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGE4DMNJRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cavearr]

Don't worry problem in my system, i'm on it

[zapta]

It's interesting, on my windows computer, I get the blinking tab only when I run apio from git bash. When I run apio from windows cmd, zadig starts up with no blinking. That's good because most windows users, I presume, use windows cmd.

…

On Sat, Nov 30, 2024 at 1:54 PM Carlos Venegas Arrabé < ***@***.***> wrote: Don't worry problem in my system, i'm on it — Reply to this email directly, view it on GitHub <#444 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQVMQIMPAU52RIBYL75FLD2DIXXVAVCNFSM6AAAAABQHQ2PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGM4TMMJXGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zapta]

Inform the user about non obvious changes

• It's now recommended not to use VCD_OUTPUT and $dumpfile in testbenches.

[cavearr]

@zapta ,Why do you don't recomend it? if we modify the documentation could be interesting explain why and what is the preferred format.

[zapta]

@cavearr, I created here an issue for this feature request #481

Using DUMPSTR and VCD_OUTPUT is confusing and and unnecessary and with this new feature, hardcoding $dumpfile("my_tb.vcd") may lead to inconsistency because it may conflict with where apio is expects the file. For example, we moved all the artifact files from the project dir to _build subdirectory and the hard coded $dumpfile() stayed at the old place.

[zapta]

Create new packages releases and update distribution.json

The goal is to make sure that the packages repos and their build script are not broken and that the releases included the latest stuff.

1. For each packages in packages.json create a new release.
2. For each package in packages.json, set it's minimum version in distribution.json to the new release.

[zapta]

Test 'apio raw' on all platforms, using apio generated command

This release changed 'apio raw' with the goal of supporting all apio generated commands. For example the apio build generated command nextpnr-ice40 --hx8k --package tq144:4k --json hardware.json --asc hardware.asc --pcf main.pcf -q fail when run using apio raw on windows.

To test, on each platform, run the apio clean and then the commands apio build, apio lint, and apio sim, copy the commands they generate and run them using apio raw.

Note that with the new apio raw, quoting the entire command is not necessary.

[zapta]

Document apio shell auto completion

As a click application, apio supports shell auto completion as documented here https://click.palletsprojects.com/en/stable/shell-completion/ . Item is to document or at least mention it somewhere in the documentation such that advance user can take advantage on it.

On my MacOSX, I use bash as the default shell and add to upgrade (using homebrew) from bash version 3 to 5 for the autocompletion to work. All I add to do is adding this to my .bashrc

eval "$(_APIO_COMPLETE=bash_source apio)"

That sets in the shell a function _apio_completion which you can view by typing the commands

$ export _APIO_COMPLETE=bash_source
$ apio

_apio_completion() {
    local IFS=$'\n'
    local response

    response=$(env COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD _APIO_COMPLETE=bash_complete $1)

    for completion in $response; do
        IFS=',' read type value <<< "$completion"

        if [[ $type == 'dir' ]]; then
            COMPREPLY=()
            compopt -o dirnames
        elif [[ $type == 'file' ]]; then
            COMPREPLY=()
            compopt -o default
        elif [[ $type == 'plain' ]]; then
            COMPREPLY+=($value)
        fi
    done

    return 0
}

_apio_completion_setup() {
    complete -o nosort -F _apio_completion apio
}

_apio_completion_setup;```

[zapta]

@Obijuan, @cavearr, I am done with the apio changes I planned and it should be in a good shape now.

If you know people that may want to give it a try and provide feedback, you can share this link https://github.com/FPGAwars/apio/wiki/Apio-development-environment#just-trying-the-latest-apio-dev-version

Features that I suggest to leave to next release are 1) support for nested directories and 2) support for multiple build envs. System verilog is already implemented but is experimental and not mentioned in the documentation (just use .sv files instead of .v).

[zapta]

I'm working on gowin fpgas support i'm push very soon.

@cavearr, do you have a gowin board? If so, which one?

I may get the same one, as our canonical gowin board.

[cavearr]

Hi @zapta, i'm working on it, i plan release gowin icestudio support in a couple of weeks, i'm preparing examples for seepid tang nano 9k and 20k.

[zapta]

Ok, I just ordered the sipeed tang nano 9k. Will use it as the canonical gowin board for testing.

…

On Sat, Dec 28, 2024 at 11:58 PM Carlos Venegas Arrabé < ***@***.***> wrote: Hi @zapta <https://github.com/zapta>, i'm working on it, i plan release gowin icestudio support in a couple of weeks, i'm preparing examples for seepid tang nano 9k and 20k. — Reply to this email directly, view it on GitHub <#444 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQVMQLQX4HMQ6ZC4DYXMND2H6TRVAVCNFSM6AAAAABQHQ2PJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRUGY2DIMZVGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zapta]

Hi @cavearr, I got the tang nano 9k and it works well with apio dev and this example https://github.com/FPGAwars/apio-examples/tree/master/examples/sipeed-tang-nano-9k/blinky

[cavearr]

Hi @zapta ! i'm continuing testing and integrating the new Apio.

With TinyFPGA-BX i have been problems.

If i use only with Apio (outside of Icestudio):

Error: [Internal] 'prog' argument does not contain the '$SOURCE' marker. [tinyprog --libusb --program]

I'll try to find in Apio source but i can't find in a first view where is that.

Tinyprog probably not work today because has problems but i want to know if that error is managed. If i use tinyprog directly not work but the output is:

Remember that tinyprog should be installed with pip if you try to run apio upload inside tinyfpga-bx board you could view the pip install tip.

I'm research a little and i could find that apio not use tinyprog from oss-cad-suite (Do you know why?). It's true that oss-cad-suite tinyprog not work (python exceptions). I helped other users this summer to patch its local tinyprog programmers to works , If you think is a good idea i could make our own package for tinyprog (aparently is abandoned) that works, what do you think?

[zapta]

Hi @cavearr, I was able to reproduce it. Will take a look later today.

[zapta]

@cavearr , are you running on macosx? Try to add $SOURCE to the tinyprog command here

apio/apio/managers/scons.py

Line 288 in 48007bf

return "tinyprog --libusb --program"

return "tinyprog --libusb --program $SOURCE"

Let me know if this works and I will submit a fix with other changes I have here.

[cavearr]

This fix is needed!, with this tinyprog works but fails (but this is the current tinyprog error not for Apio), now i'm trying to prepare our own version of tinyprog that works well.

`apio upload -p ~/Downloads/ice-build/01.\ One\ LED
Setting the envinronment.
[Thu Jan 9 20:51:01 2025] Processing tinyfpga-bx

tinyprog --libusb --program _build/hardware.bin
Traceback (most recent call last):
File "/Users/carl/.apio/packages/oss-cad-suite/libexec/tinyprog", line 31, in
sys.exit(load_entry_point('tinyprog==1.0.23', 'console_scripts', 'tinyprog')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/carl/.apio/packages/oss-cad-suite/libexec/tinyprog", line 23, in importlib_load_entry_point
return next(matches).load()
^^^^^^^^^^^^^^^^^^^^
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/importlib/metadata/init.py", line 202, in load
module = import_module(match.group('module'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1126, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/tinyprog-1.0.23-py3.11.egg/tinyprog/init.py", line 10, in
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/jsonmerge-1.9.2-py3.11.egg/jsonmerge/init.py", line 4, in
from jsonmerge.resolver import LocalRefResolver
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/jsonmerge-1.9.2-py3.11.egg/jsonmerge/resolver.py", line 2, in
from jsonschema.validators import RefResolver, urldefrag, urljoin
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/jsonschema-4.23.0-py3.11.egg/jsonschema/init.py", line 13, in
from jsonschema._format import FormatChecker
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/jsonschema-4.23.0-py3.11.egg/jsonschema/_format.py", line 11, in
from jsonschema.exceptions import FormatError
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/jsonschema-4.23.0-py3.11.egg/jsonschema/exceptions.py", line 15, in
from referencing.exceptions import Unresolvable as _Unresolvable
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/referencing-0.35.1-py3.11.egg/referencing/init.py", line 5, in
from referencing._core import Anchor, Registry, Resource, Specification
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/referencing-0.35.1-py3.11.egg/referencing/_core.py", line 9, in
from rpds import HashTrieMap, HashTrieSet, List
File "/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/rpds/init.py", line 1, in
from .rpds import *
ImportError: dlopen(/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so, 0x0002): tried: '/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (no such file), '/Users/carl/.apio/packages/oss-cad-suite/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (not a mach-o file)
scons: *** [upload] Error 1
============================================================================================================ [ ERROR ] Took 0.31 seconds ===============================`

[cavearr]

I have been found that rpds.cpython-311-darwin.so in oss-cad-suite for osx (ARM) is wrongly, this file is for darwin x86-64.

I'll try to find how to fix it or replace when we build our system-tools.

I'll been research around it

[zapta]

Hi @cavearr, is this a problem with the upstream oss-cad-suite from the yosys team? If so, I suggest to file there a bug.

I am not an expert but I think that OSX ARM can use x86 files via emulation which is transparent to the user.

[cavearr]

i'm working on it, in general you could activate "rosetta" support to emulate transparently x86 files, but in this case pyhton is arm native and this library is x86 and loaded dynamically and i understand that rosetta can't mix a native binary with a shared library emulated.

I'm trying to compile nativetly this library and if it works we could replace in our package.

[cavearr]

@zapta, yosys uses different packages:

[zapta]

Can you reproduce the problem with the upstream yosys package release? https://github.com/YosysHQ/oss-cad-suite-build

After you uncompress the yosys package you need to run source activate to setup the env.

[cavearr]

Hello @zapta! yes, the first thing I tried was to download and play with the latest version of oss-cad-suite and the results are the same, I frequently use the oss cad suite directly, I am used to using it and installing it without Apio.

Now I have tinyprog working.

I compiled natively on OSX under oss-cad-suite python target and replaced the original with my own and it works perfectly (scheduler crashes due to tinyprog issues which we will fix with some patches, we will also have to maintain our own version, because the project is stagnant and without apparent maintenance).

Since I already have it working, tomorrow I'll test it on Windows and Linux to see if it's stable natively (I think it will be) and if that's okay with you, I'll put together a package to use in our system-tools to replace this library on the osx package.

[zapta]

@cavearr, that's very good!

If the problem is with the upstream oss-cad-suite from yosys, I suggest to file there an issue, for the record.

[cavearr]

Yep! i'll open the issue tomorrow