Support for custom Annotator classes

[joelittlejohn]

Original author: [email protected] (January 28, 2013 19:05:34)

Hi,
It seems current logic only handles "properties" array to generate POJOs. Is there any programmatic way to extend this logic to additionally support link-relationships representation in POJOs?

---

Specifically, jersey-server-linking defines @Ref/@binding annotations that allows to specify resource and template parameter binding.

Sample that comes with the library is a representation bean (POJO) as follows:

public class ItemRepresentation {

@XmlElement
private String name;

@Ref(
    resource=ItemResource.class,
    style = Style.ABSOLUTE,
    bindings=@Binding(name="id", value="${resource.id}")
)
@XmlElement
URI self;

@Ref(
    resource=ItemResource.class,
    style = Style.ABSOLUTE,
    condition="${resource.next}",
    bindings=@Binding(name="id", value="${resource.nextId}")
)
@XmlElement
URI next;

@Ref(
    resource=ItemResource.class,
    style = Style.ABSOLUTE,
    condition="${resource.prev}",
    bindings=@Binding(name="id", value="${resource.prevId}")
)
@XmlElement
URI prev;

public ItemRepresentation() {
    this.name = "";
}

public ItemRepresentation(String name) {
    this.name = name;
}

}

Right now, the following schema:
{
"type":"object",
"properties": {
"foo": {
"type": "string"
},
"bar": {
"type": "integer"
},
"baz": {
"type": "boolean"
}
},
"links": [
{
"rel": "self",
"href": "clients/{id}"
}]
}

... generates same POJO as following schema (without links):
{
"type":"object",
"properties": {
"foo": {
"type": "string"
},
"bar": {
"type": "integer"
},
"baz": {
"type": "boolean"
}
}
}

And since POJO generation is dynamic, I would like to be able to either decorate the generated POJOS dynamically, or to influence the code generating the classes in the first place, to add additional annotations in the generated code.

Beside the obvious decoupling from Jersey, do you see why links are not generated (at all) in the POJOs? Should I specifically add a bag of properties to have the defined links listed?

Thanks for your help.

Original issue: http://code.google.com/p/jsonschema2pojo/issues/detail?id=86

[joelittlejohn]

From [email protected] on January 28, 2013 20:33:41
Hi Philippe, thanks for taking the time to raise this. Nice someone including links in their representation and creating a self-describing API!

Yes the tool ignores the "links" schema property right now. At the moment I'm finding it difficult to see how jsonschema2pojo could support this in a general way.

I think the best option for you right now might be to use jsonschema2pojo to generate the types as step 1, but make further manual changes to the sources produced (i.e. and keep the sources under normal version control from then on). You can create a property of type "string" with format "uri" to create a URI type property. You'll have to annotate the properties yourself.

Another option would be to create your own clone and implement this as a custom change.

If you have any neat ideas about how this might be included in jsonschema2pojo feel free to propose a solution. I wonder if the only general way to do this would be to add some way of providing your own custom Annotator, which is invokes whilst sources are generated and has the opportunity to add custom annotations. Would this work for you?

[joelittlejohn]

From [email protected] on January 28, 2013 20:50:10
My biggest concern right now is the lack of tooling to use json schema as the only API contract. Right now, JAX-RS in Java has some level of expressing contract using ResourceBeans making reference to resource representation (POJOs) and links, which itself could generate json-schemas. But that'd mean code-first contract, which is risky to validate against backward compatibility over time. I'd prefer the other way around, defining schemas as binding contracts and generating API stubs on top of it. So if one change the contract, e.g. adding an optional property or link relationship, the beans and API stubs get updated in an automated way.

Beside forking the complete project, I was hoping for any listener/extension class I could extend to instruct jsonschema2pojo to generate beans differently. Modifying beans manually is error prone and expose risk of getting schemas and beans out of sync. Big deal in an enterprise environment...

Exposing properties as URIs is different to me than exposing relationships. But I will try this and see if this brings me anywhere.

Regards,
Phil

[joelittlejohn]

From [email protected] on February 01, 2013 22:52:52
Joe, are you open to contributors in the code source?

My team (@oraclecorp) is actually interrested at participating to generate JAXB compliant annotation out of the jsonschema in the generated POJOs.

Let me know and contact me directly if you are.

[joelittlejohn]

From [email protected] on February 01, 2013 23:11:35
Philippe, yes absolutely. Feel free to create a clone and make some changes, if you're successful then I will certainly consider applying this contribution to the core. Of course, we need your contributions will be licensed under Apache 2.0 for this to happen.

If I understand you correctly, you'll be implementing this existing feature request:

http://code.google.com/p/jsonschema2pojo/issues/detail?id=74

If you need any help navigating the code or advice on implementation then please feel free to send me some questions. Cheers!

[joelittlejohn]

From [email protected] on February 05, 2013 17:42:57
By reading your comments twice:

I wonder if the only general way to do this would be to add some way of providing your own custom Annotator, which is invokes whilst sources are generated and has the opportunity to add custom annotations. Would this work for you?

I like this idea very much: allowing custom Annotater be registered through configuration and then invoking it during generating process would keep jsonschema decoupled and yet extensible for any technology specific use cases. E.g. supporting JAXB type of annotation, or @rel jax-rs type of annotation for each links.

While generating classes, the configuration should allow to specify to pass the current file (java class being generated) or another file. Why another file? So it could actually generate an API class additional to the DTO class. Now that would be neat :))) >> having an API solely described by the json-schema, including (@get or @post) operation that returns DTOs (as defined per links schema semantic), so changing it ripples down to update API and DTOs in the code... no out-of-sync between the Java API and the json-schema definition... that would be great!

Do you have a concrete idea how to allow customized annotation while parsing the schema (including properties AND link definitions)? Could we review together your ideas?

[joelittlejohn]

From [email protected] on February 05, 2013 17:53:52
By API class, I think to an interface in the form of JAX-RS resource beans.

so
customTypeName.jsonschema:
{
"type":"object",
"properties":
{
"javaType" : "com.other.package.CustomTypeName",
...
}
"links": [
{
"rel": "self",
"href": "clients/{id}",
"method": "GET",
"schema":
{
"type": "object",
"properties":
{
"expand":
{
"description": "Returns all version details (by default, only version identifier and links are returned)",
"type": "boolean"
}
}
},
"mediaType": "application/json"
}]
}

becomes

@path("clients/{id}")
@produces({ MediaType.APPLICATION_JSON })
public interface CustomTypeNameResource {

    @GET
CustomTypeName getCustomTypeName(@Context UriInfo uriInfo, @QueryParam ("expand") boolean expand);

}

Neat... really neat...

[joelittlejohn]

From [email protected] on February 05, 2013 20:35:36
I've written a tool very similar to what you've described. It uses a combination of WADL and JSON Schema to create JAX-RS annotated types and POJOs for binding. Unfortunately this is unlikely to be open sourced. It also requires a few conventions in the input that are specific to the context in which it's used.

Because of the way CodeModel[1] works, it would be trivial for a custom Annotator to add new types (as well as, of course, modify the existing ones). I expect the difficulty would come in trying to pass enough contextual information to the Annotator for it to sanely build the additional set of classes (I felt these pains when implementing the tool I've described above). No harm in trying though.

[1] http://codemodel.java.net/

[joelittlejohn]

From [email protected] on February 05, 2013 20:38:39
Philippe, I think it make sense for us to change the focus of this enhancement request and generalise it based on what we've discussed.

I'll tentatively peg this for 0.3.6. I reserve the right to change this though if something else becomes more urgent :)

[joelittlejohn]

From [email protected] on February 20, 2013 15:58:51
Hi Joe,
any update on this one? We actually look forward adding annotation for jackson-xml databinding (see https://github.com/FasterXML/jackson-dataformat-xml#additional-annotations), since currently, only JSON jackson annotation are generated.

@JacksonXmlElementWrapper allows specifying XML element to use for wrapping List and Map properties
@JacksonXmlProperty allows specifying XML namespace and local name for a property; as well as whether property is to be written as an XML element or attribute.
@JacksonXmlRootElement allows specifying XML element to use for wrapping the root element (default uses 'simple name' of the value class)
@JacksonXmlText allows specifying that value of one property is to be serialized as "unwrapped" text, and not in an element.

Should I enter a separate enhancement so this becomes part of the default build? (jsonschema2pojo already has jackson dependencies, it should be logical to support such annotations as well, e.g. by allowing some specific jsonschema property, similar to javaType you already have).

Thanks!

[joelittlejohn]

From [email protected] on February 22, 2013 20:44:46
It would be very helpful if you could put some thought into what exact contextual information you would need in order to add these Jackson XML annotations. Here's the current annotator interface:

http://code.google.com/p/jsonschema2pojo/source/browse/jsonschema2pojo-core/src/main/java/com/googlecode/jsonschema2pojo/Annotator.java?name=jsonschema2pojo-0.3.5

This is quite specific to Jackson's JSON annotations - I think we may need to generalize this interface further so that it has the opportunity to operate on any generated construct.

Could you try experimenting with your own implementation of the Annotator interface to see if you have enough power to apply the Jackson XML annotations where you need them? And when the Annotator is invoked, does it currently receive enough information to decide the correct XML annotation?

[joelittlejohn]

From [email protected] on February 22, 2013 23:10:27
Attached is a PDF with a proposal of how it could look like.

I think below 2 methods would be key:

• void propertyInclusion(JDefinedClass clazz)
• void propertyField(JFieldVar field, String propertyName)
  and sufficient for the needs.

As you will see, I added 2 new special properties in the schema
(xmlItemName and isXmlProperty). The other values needed for XML
serialization can be extrapolated from the existing properties of JSON
schema. Bear with me the naming of these 2 properties and rename at your
best convenience.

The "id" which semantically represents the namespace of the JSON schema
would be used as XML namespace. The class name (javaType or default) would
be used for the xml root elements annotation (both parent and child classes
in case of $refs/sub types).

For collections (array) there is special handling:

• when the xmlItemName is found in the schema, this becomes the value for
  the JacksonXmlProperty localname (namespace could reuse value of class'
  parent namespace or left out). If the property is not found in schema, no
  JacksonXmlProperty is generated for that "array" property.
• in any case, the JacksonXmlElementWrapper "localname" attribute is
  populated from the property name (of "array" type).

For all other properties, the logic is as follows:

• when the "isXmlProperty" attribute is found beneath a given property the
  annotation JacksonXmlProperty is created with ("isAttribute=true). Else no
  JackonXmlProperty is generated (it will take the property field name by
  default as XML element name.

You would likely enhance the AnnotatorStyle with an additional JACKSON2+XML
enum item. And adapt the config tasks to support that new Annotator. This
one should generate BOTH JSON and XML annotations (most ppl want to support
both).

The advantage would be that compared to now, you would have the flexibility
to indicate how to serialize the POJO generated in XML (using XML attribute
or element), have the ability to specify the root elements of lists and
list items, and finally associate an XML element to a given namespace. Note
that the namespace stuff is nice to have. I care more about the list and
attribute stuff.

Let me know what you think and whether this would sound doable in your
mind.

[joelittlejohn]

From [email protected] on February 22, 2013 23:15:33
Hmm, I don't think you can attach files to issues by attaching a file to the email. Could you attach the file to a comment via:

https://code.google.com/p/jsonschema2pojo/issues/detail?id=86

?

[joelittlejohn]

From [email protected] on February 23, 2013 00:17:00
Nope, still nothing. I guess maybe Google has disable

Feel free to email it to me, my gmail account name is joelittlejohn.

[joelittlejohn]

From [email protected] on February 23, 2013 01:00:23
See attached the PDF.

[joelittlejohn]

From [email protected] on February 23, 2013 21:49:26
Hey Joe,
I cloned and started implementing the Annotator to handle Jackson XML annotations.
This is very early dev, I committed still but no guarantee it is safe to use without further testing.

https://code.google.com/r/marsteau-jsonschema2pojo/source/detail?r=eae568eedffb4d1c78ee97ddef49a871c8bf9fbf

You will see I did have to change/add a few operations to the Annotator interface as well as a few other minor changes.

Please review and let me know what you think.

One known limitation right now is that array having "$ref" to other schema are not working right now. But the proposal above is merely implemented.

I still need to test the actual XML serialization using the generated POJOs. I only tested that the POJOs were looking how I wanted them to look like.

This is work in progress, let me know what you think before I invest too much time in this.

[joelittlejohn]

From [email protected] on February 23, 2013 22:11:20
Good stuff Philippe. I'll take a detailed look soon, but for now I have one initial comment:

I think it would be more flexible to keep the XML annotations in their own annotator (not mix with/extend the json annotator), but allow users to specify more than one annotator in their configuration (probably as a comma-separated list). This makes the annotators much more composable. So if you need jackson2 json annotations, and jackson2 xml annotations, you specify:

jackson2,jackson2xml

(or equivalent for Ant and CLI).

We could even support custom annotator classes by allowing a these values to be fqcn's, e.g.

jackson2,jackson2xml,com.mycompany.CustomAnnotator

Please don't think too much about this - I'm happy to apply this change (switch from 'one' to 'one or more' annotators) and I think you should concentrate on your XML production. Once you have written a functioning annotator we can integrate it into the project any way we see fit.

Oh and just for the record, could you comment here indicating that you are happy for contributions to be released, as part of this project, under Apache 2.0.

[joelittlejohn]

From [email protected] on February 23, 2013 22:40:54
I agree with separating the annotations. However, XML databinding is on top of Jackson2 (not compatible with Jackson1). I think in this case it might be better to have all jackson2 stuff (JSON and XML) in one Annotator, and instead of a XML specific Annotator have a "generate XML annotation" flag (similar to JSR303) to indicate whether only JSON or both JSON and XML annotation should be generated.

But for other types of annotations (e.g. JAXB which is not Jackson-xml) or for external coded annotator it does make sense.

Could you help me understand why JSON properties that references other schemas are not dereferenced BEFORE starting the generation process? Right now, the property of an array contain the "$ref" reference, but not its content. My XML annotator needs the dereferenced content. Is there any utility I could call to get the dereferenced JsonNode instead of the one containing the reference?

As I said, this is not final. I added an integrated test case for the simple properties and the root element (incl. namespace), and another for array which is lacking a few assertions in it (I validated visually in the generated class the generated code was good, but I need to add the validation in the JUnits).

[joelittlejohn]

From [email protected] on February 23, 2013 23:00:29
Okay, maybe I have misunderstood how jackson-databind-xml works. Does it depend on both Jackson JSON annotations and Jackson XML annotations to produce XML? If so, I prefer how you have structured this already (don't want to introduce another, annotator-specific flag here).

Re deref'ing refs before starting, unfortunately the code is just not structured in this way. The rules simply navigate the schema as they find it. Depending on your situation, you may be able to solve your problem like:

https://code.google.com/p/jsonschema2pojo/source/browse/jsonschema2pojo-core/src/main/java/com/googlecode/jsonschema2pojo/rules/JsonSchemaRule.java?name=jsonschema2pojo-0.3.5#59

But I warn you that there is no limit to how deep the referencing may go, including (in some cases) circular references.

Maybe you could explain the scenario in more detail or give an example? When you say 'the property of an array contain the "$ref" reference' are you talking about the 'items' property that is included as part of the array definition? I don't think I understand why you need to access this. Are you trying to jump down from an array property into its items schema and check the content of the properties on the array elements?

[joelittlejohn]

From [email protected] on February 23, 2013 23:29:38
I just added a new revision including a test case + code that will
show you what's the intent here.

In a nutshell the namespace of child items (element wrapper
annotation) should use the "id" property value of the item
schema/object (instead of the container schema "id").

I'll look into your sample code tomorrow.

Thanks for the quick reply.

[joelittlejohn]

From [email protected] on February 24, 2013 16:32:43
Committed few more changes to my cloned repository, including support for dereferencing "items" or "array" nodes.

I have added integrated tests, including the serialization. What would miss would be proper documenting which property in the JSON schema would generate which annotation (along with an example of the serialized XML).

Below is an overview of the addition to jsonschema2pojo that I hope you would integrate in the core project:

• Generation of namespace-aware POJOs for Jackson-XML datatbinding (using "id" property value)
• Ability to specify whether a schema property should be serialized as XML attribute vs. XML element (set "isXmlProperty" property to "true" to make it an XML attribute; if not set or false, will make an XML element)
• Ability to control the XML element name for children of schema array properties. Jackson-XML by default enables wrapping of List or Set fields (that is, schema array properties). The XML wrapper element will be set to the schema array property name. In addition, the individual child element name can be specified in the JSON schema (set "xmlItemName" in the schema array property definition).

Let me know what you think and feel free to refactor/adapt as appropriate, as long as the integrated test cases continue to pass.

[joelittlejohn]

From [email protected] on February 24, 2013 16:34:25
just realized "isXmlProperty" is badly named, and should be renamed for "isXmlAttribute". My bad.

[joelittlejohn]

From [email protected] on March 05, 2013 21:42:10
Hi Joe, any chance/ETA when you would incorporate this enhancement?

[joelittlejohn]

From [email protected] on March 13, 2013 00:03:17
Philippe, I've just merged some commits into master that allow a custom annotator to be specified in config.

Could you try building a snapshot from the current HEAD and let me know how you get on?

[joelittlejohn]

From [email protected] on March 15, 2013 20:51:07
I'm going to tentatively close this ER as complete. Hopefully, Philippe, you will get a chance to try the new customAnnotator config property before 0.3.6 is released.

[joelittlejohn]

From [email protected] on April 04, 2013 11:52:20
Hi Joe. I haven't yet been able to test, but this is great addition. I see you adapted Annotator interface so the proposed implementation I had submitted can be done. I will update when I get time to get back on this.

[joelittlejohn]

From [email protected] on April 04, 2013 11:56:32
Great. Yes I updated the Annotator interface to provide more details about the context in which it's being called.

iirc you added a new method for one of your annotations but I tried to include this in one of the existing, class-level annotation methods instead (because I think your case was a bit xml-specific).

[mgudivada]

Guys, Can you please let me know how do I generate POJO for my json with annotation "@JacksonXmlProperty(isAttribute=true)"?