Repo Merge & Switch to tycho. #2135
Comments
|
ps: most repos contain a years old ade-newbuild branch with the experiments @ArneDeutsch did back then |
|
the maven plugins can be part of the same repo without problems; I think they should. |
|
Hi. We did some bigger repository merge (4 into one) some time ago (actually containing a set of Xtext DSLs).
To make things reproducible (and we did need some test iterations...) we wrote scripts to automate the whole process. I can provide them if you like. |
|
@LorenzoBettini i added some npm script to package json |
@cdietrich I cherry picked your commit on my branch and added the frontend-maven-plugin (I think it's a mainstream one) LorenzoBettini/xtext-web@a8c964e I don't know much about nodejs but I think I specified the goals correctly in the POM By running |
|
@LorenzoBettini this looks good. in your jar they are not packaged |
|
I've been working on that, and now they are: |
|
I've basically updated all the projects (with tests as well). The one that remains to be done is That one is not meant to be deployed to Maven Central, is it? |
|
yes and no. its not deployed. but it does two things. |
Let's see what I can do with that project. I see that also Orion has to be downloaded... |
|
yes there is unfortunately no webjar for it |
|
So, I converted also that project and configured the POM to download and unzip Orion. If I run the example I get this, which looks fine, right?
|
|
yes. and the same should work for ace and codemirror |
|
ace and codemirror were already working because they are part of our sources; Orion instead had to be downloaded and unzipped |
|
Great work, @LorenzoBettini ! I'm amazed how you powered through this involving change of the build infra. I'm really looking forward to merging this after the release and then tackle the repo merge. |
|
again my question: can we already downport parts like manifest/build.properties changes and renamings? |
|
Starting with the class renames would make sense. Having these consistent prior to the subsequent changes, allows for easier diffing the results. |
|
@cdietrich, I forgot to answer your past question: I'm not sure what you meant. If you mean merging the PRs, I'd first need a few small adjustments and, most of all, harmonize @szarnekow thank you for the kind words :) For sure, concerning I'd also need some additional information concerning the By looking at the build.gradle I seem to understand that the contents of test-files directory must be zipped and put into the output folder inside "libs"? If you already have a WAR file generated using the current configuration, could you please show me the expected layout of the WAR? This is the current layout (without test-libs): |
+1 on that @LorenzoBettini no i dont mean merge the prs. i mean merge parts of the changes that can be safely done on master. |
|
about the jetty war. i have no idea about the testlibs. this also might be a bug. |
|
mh... there's no trace of test-libs. If you run the example with jetty, you should get an error when trying to select one of the resources:
|
|
this trys to load file from disk. i dont know to which path this points to in your example |
|
@cdietrich can you please confirm that if you run jetty (with gradle) from that project it finds the test input files or it doesn't? The Jenkinsfile doesn't tell me much, because it only seems to confirm that the test files are not in the WAR. |
|
it is. and they are picked separately from filesystem. so that they also can be written/saved
|
Of course I'm not saying to ban shell scripts at all ;) |
|
@cdietrich I'm still not able to solve the problem of milestone in my project, without applying the solution I proposed above (which is the one currently part of Xtext monorepo) consisting in passing the additional property for milestone. Note that this means that, if we ever switch to a single command for releasing Maven artifacts and p2 site, we won't still be able to do that for milestones. I also found this eclipse-mwe/mwe#112 which confirms you had the same problem... I still don't understand your solution though. I seem to understand that you use the shell script only to replace versions. I think I come up with the same version replacement without the shell script. When running the Maven build I don't understand why it works for you and not for my simple project. |
|
@cdietrich I found your hack ;) https://github.com/eclipse/mwe/blob/master/set_version.sh#L123 you manually replace .qualifier with the timestamp before the build in the source MANIFEST! That's why it works for you and that's why the resulting When the Maven/Tycho build runs there's no .qualifier at all in manifests, just the timestamps. I'll see whether this can be achieved in a cleaner way (same result but without shell hacks) |
|
i see. |
|
@cdietrich So I managed to make it work in my simple example. More technical details later. In the meantime, here are the results (note: they're shell scripts but they only invoke Maven command): STANDARD BUILD mvn clean package RELEASE ./prepare-for-release.sh
mvn clean package -Pp2-releasediff --git a/org.example.feature/feature.xml b/org.example.feature/feature.xml
index 0f67cda..b3b3f50 100644
--- a/org.example.feature/feature.xml
+++ b/org.example.feature/feature.xml
@@ -2,7 +2,7 @@
<feature
id="org.example.feature"
label="Example Feature"
- version="1.0.0.qualifier"
+ version="1.0.0.v20230318-1827"
provider-name="Lorenzo Bettini">
<description url="http://www.example.com/description">
diff --git a/org.example.feature/pom.xml b/org.example.feature/pom.xml
index 7337a44..8a0d05f 100644
--- a/org.example.feature/pom.xml
+++ b/org.example.feature/pom.xml
@@ -8,7 +8,7 @@
<relativePath>../org.example.parent/pom.xml</relativePath>
<groupId>org.example</groupId>
<artifactId>org.example.parent</artifactId>
- <version>1.0.0-SNAPSHOT</version>
+ <version>1.0.0</version>MILESTONE ./prepare-for-milestone.sh M1
mvn clean package -Pp2-releasediff --git a/org.example.feature/feature.xml b/org.example.feature/feature.xml
index 0f67cda..b183616 100644
--- a/org.example.feature/feature.xml
+++ b/org.example.feature/feature.xml
@@ -2,7 +2,7 @@
<feature
id="org.example.feature"
label="Example Feature"
- version="1.0.0.qualifier"
+ version="1.0.0.v20230318-1830"
provider-name="Lorenzo Bettini">
<description url="http://www.example.com/description">
diff --git a/org.example.feature/pom.xml b/org.example.feature/pom.xml
index 7337a44..71708fc 100644
--- a/org.example.feature/pom.xml
+++ b/org.example.feature/pom.xml
@@ -8,7 +8,7 @@
<relativePath>../org.example.parent/pom.xml</relativePath>
<groupId>org.example</groupId>
<artifactId>org.example.parent</artifactId>
- <version>1.0.0-SNAPSHOT</version>
+ <version>1.0.0.M1</version> |
|
@cdietrich some more technical details about the solution. I've tested in my simple project https://github.com/LorenzoBettini/tycho-archive-example where I also have GitHub Actions jobs to simulate snapshot, release, and milestone; a Tycho integration test verifying the expected bundle and maven versions in the manifest in the 3 different scenarios; the jobs produce p2 sites that can be manually inspected for the 3 scenarios (e.g., https://github.com/LorenzoBettini/tycho-archive-example/actions/runs/4467446821). The modification of MANIFEST have already been explained previously and already been applied to Xtext. For releases and milestones, the idea is to first use Tycho versions plugin to change The parent POM contains the property: <timestamp-pattern>'v'yyyyMMdd-HHmm</timestamp-pattern>used consistently in the rest of the POM. If for any reason, we want a different format, we need to change a single property. For example, <plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-packaging-plugin</artifactId>
<version>${tycho-version}</version>
<configuration>
<format>${timestamp-pattern}</format>
<skipPomGeneration>true</skipPomGeneration>
</configuration>
</plugin>There's a profile (just to make sure the goal is not called unintentionally), using the build helper plugin to compute the timestamp according to the same format and set it in the property <profile>
<id>replace-qualifier-with-timestamp</id>
<build>
<plugins>
<!-- meant to be called explicitly via the goal syntax:
build-helper:timestamp-property@timestamp -->
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>timestamp</id>
<goals>
<goal>timestamp-property</goal>
</goals>
<phase>none</phase>
<configuration>
<name>computedTimestamp</name>
<pattern>${timestamp-pattern}</pattern>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>Now we have shells scripts to call Maven goals. For example, to turn qualifiers/snapshots into timestamps ( #!/bin/sh
mvn \
-Preplace-qualifier-with-timestamp \
build-helper:parse-version \
build-helper:timestamp-property@timestamp \
org.eclipse.tycho:tycho-versions-plugin:set-version \
-DnewVersion=\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}.\${computedTimestamp} \
-DgenerateBackupPoms=false
Another shell script #!/bin/sh
mvn \
build-helper:parse-version \
versions:set \
-DgenerateBackupPoms=false \
-DnewVersion=\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}Another shell script #!/bin/sh
if [ $# -eq 0 ]; then
echo "No arguments provided: pass the milestone, e.g., M1"
exit 1
fi
mvn \
build-helper:parse-version \
versions:set \
-DgenerateBackupPoms=false \
-DnewVersion=\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}.$1Then, there are the two main scripts that call appropriately the previous ones. One for preparing for a release #!/bin/sh
script_dir="$(dirname "${BASH_SOURCE:-$0}")"
${script_dir}/replace-qualifier-with-timestamp.sh && \
${script_dir}/remove-snapshots-poms.shOne for preparing for a milestone #!/bin/sh
if [ $# -eq 0 ]; then
echo "No arguments provided: pass the milestone, e.g., M1"
exit 1
fi
script_dir="$(dirname "${BASH_SOURCE:-$0}")"
${script_dir}/replace-qualifier-with-timestamp.sh && \
${script_dir}/set-milestone-poms.sh $1Of course, when then performing the build with the modified versions, we must tell Tycho to ignore the differences between OSGI and POMs; there's a profile to be activated (it's the same for release and milestone): <profile>
<id>p2-release</id>
<build>
<plugins>
<plugin>
<groupId>org.eclipse.tycho</groupId>
<artifactId>tycho-packaging-plugin</artifactId>
<configuration>
<strictVersions>false</strictVersions>
</configuration>
</plugin>
</plugins>
</build>
</profile>It's simpler than it may sound ;) |
|
@LorenzoBettini will this also work in future tycho versions or do we have to adapt the the tycho versions plugin is more coupled to the maven versions plugin |
|
@cdietrich I've just tested my project with Tycho 3.0.4 and it works as before |
|
@LorenzoBettini i assume we can close this one as done and handle remainders with separate issues |
Great 🎉 |
|
The main focus was on keeping the history and retain the existing committers knowledge about the code structure. Listing a few dozen modules is no real burden compared to the need for additional code research in the daily work of triage or code archeology. |
|
If git detects that a file was moved (i.e. a added file is similar enough to a deleted file) then the history from before the move is retained and the git blame contains the same information as before. So this does not have an influence on the per line git-blame, unless you also change some lines while renaming the file. The rename is also considered in the history view of EGit where it shows commits touching the previous file name when you select to show only changes of the selected resource. |
@cdietrich Yes, I guess this one can be closed. @HannesWell I'm not a big fun of POMless build ;) , but if others think about restructuring I'm OK with that. |
|
yes will close. |
May I ask you why that's the case? |
|
@HannesWell i dont trust the autogenerated poms to publish to maven central at all |
|
@HannesWell I had a few problems in the past with pomless builds (in other projects). Moreover, as said by @cdietrich , the poms generated by Tycho give a few problems: they are not consumable by Maven plugins (at least, that's my experience in the past; I tried several times and it didn't work). With a well-curated parent POM, the poms in bundle and feature projects are minimal. In test projects, we typically still have to write a few additional configurations. |
Understand. Have you considered a pom-first approach then where you basically make the projects Maven projects and generate the OSGi Manifest using the maven-bundle-plugin or bnd-maven-plugin? You would then save the duplicated dependency handling and the generated Manifest is often more accurate and 'better'. And m2e has a connector built in so that those projects can participate in the regular PDE workspace build with the generated manifest. |
|
yes. but also the bnd tools produces funny stuff. had also not found out how to properly build/test eclpse ui stuff with it |
OK. Yes If one wants to have a non default configuration (at least what the bnd-devs think is the best default) it requires some set up to produce the desired results. |
@HannesWell that sounds much more complicated than the current approach ;) |
That probably depends on the details, but if you are happy with it how it is, that's of course fine :) |
|
commiter activity count will be done manually for now |
following the discussions in eclipse/xtext-core#2052 and #2133
we consider Repo Merge & Switch to tycho again
the following things we need to clarify
cc @LorenzoBettini @szarnekow @kthoms
Thinks not to forget
The text was updated successfully, but these errors were encountered: