Java: Include all XML files with codeql-cli

[RickGY]

I've trying to use the codeql command line interface to analyze a project locally. I would like the extractor to include all the XML files, but it seems that the "codeql database create command" will only include pom.xml.

• I noticed that by including a lgtm.yml file in the repo and write the following, LGTM can include all xml files:

  extraction:
  java:
  index:
  xml_mode: all

I wonder if it's possible to do something similar while using codeql-cli?

[RasmusWL]

Hi @RickGY, yes, instead of doing

codeql database create --source-root <src> --language java --command <java command> <db>

You can split up the database creation process to manually include the XML files:

codeql database init --source-root=<src> --language java <db>
codeql database trace-command --working-dir=<src> <db> <java command>
codeql database index-files --language xml --include-extension .xml --working-dir=<src> <db>
codeql database finalize <db>

If you're just using the current directory as your source, you can skip the --working-dir <src> parts.

[dilanbhalla]

Hi @RasmusWL , I actually had a similar question. I am attempting to include xml files for a python CodeQL database I am building through the CLI, but am running into some issues. What should I include as the component in step 2 (but if I am working in python)? I am a bit unsure about what that command does, and if I run all of the commands except that step, my database includes the xml files I would like but nothing else (so none of the source code at all). Would you mind providing some guidance?

[RasmusWL]

Hi @dilanbhalla. I agree that this part is a bit tricky, and requires internal knowledge of how to do things. I was a bit confused about how to do it as well for #3911 (we're having internal talks about making it easier).

You can use this trick to trigger the same behavior as with codeql database create 👍

codeql database trace-command --working-dir=<src> <db>  $(codeql resolve extractor --language=python)/tools/autobuild.sh

[RickGY]

Thanks a lot! @RasmusWL

[RasmusWL]

No problem 😊

Gonna close this now then

[RasmusWL]

Hi Rasmus, apologies in advance for the long follow up, but I have a question unrelated to this query and am unsure how else to contact you (since our discussion thread from earlier is now closed). This may seem a little silly, but the trace-command you showed my for python won't work due to the simple error that the '$' is not recognized. My end goal is to simply use the CLI to build a python database that includes some custom xml files I wrote, so I believe your method would work for python (init, index the xml files, trace-command, finalize). Does your PR need to be merged before this trace-command will work? Or is it something simple that I may be doing wrong with regards to the expression starting with '$'? And lastly, if including the xml is not at all possible, would you happen to know any other method to include custom data (maybe through something like a csv) and reference it within a python ql file? Thank you so much!
- #3878 (comment)

Hi @dilanbhalla, you can comment on closed issues, no problem. I think the most appropriate place to ask questions about the CLI in the future is on https://github.com/github/codeql-cli-binaries/issues. You can always use https://github.com/github/securitylab/discussions as well for CodeQL questions 😊

I guess the part you're having trouble with is $(codeql resolve extractor --language=python)/tools/autobuild.sh. That is just using command substitution in bash. Basically it means, run codeql resolve extractor --language=python and insert the output of that command in place of $(codeql resolve extractor --language=python). So if you're running a different shell where command substitution doesn't work, you can do the following:

$ codeql resolve extractor --language=python
/foo/bar/codeql/python
$ codeql database trace-command --working-dir=<src> <db>  /foo/bar/codeql/python/tools/autobuild.sh

if you're on windows, you should probably use autobuild.cmd instead of autobuild.sh 😊

[dilanbhalla]

Hi @RasmusWL! This worked like a charm. Sorry I did not realize I can continue to comment on a closed issue and thank you so much for the advice. Everything is running perfectly for me now, with the .xml files I was attempting to include earlier. Thanks again!