Array of Map

[ismail261]

My schema has an array of map. And I believe that feature does not exist. Am I correct on this? Or Am I missing something?

[kszosze]

Hi Ismail,
That is an interesting question. We support Arrays of almost whatever type you can define in AVRO. As far I saw in the code, array of maps should be covered as well, but we didn't test this specific case.
Did you find any error or exception? If so, please let us now in order to fix it.
Cheers!!

[ismail261]

As soon as I tried to load my schema, it failed on line number: 153 in SchemaExtractor class with index out of bound exception.
internalFields.get(0).setFieldName(innerField.name());
This happened because extractArrayInternalFields(innerField); returned empty array.

It returned empty Array because the above method didn't had typeset of Map defined in SchemaExtractor.

private final Set<Type> typesSet = EnumSet.of(Type.INT, Type.DOUBLE, Type.FLOAT, Type.BOOLEAN, Type.STRING, Type.LONG, Type.BYTES, Type.FIXED);

After I added Type.MAP in the above line, Schema Extractor worked fine.

Now I don't know what format will it accept for values section.
I tried adding [{a:1234}] or [a:1234] but it failed with below message:

org.apache.kafka.common.errors.SerializationException: Error serializing Avro message Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.util.Collection (java.lang.String and java.util.Collection are in module java.base of loader 'bootstrap')

And I think it does not know to convert string into map inside array type. If parent type is Map or Array then it can handle things correctly. But if parent type is Array and subtype is Map, then code might not exist to handle that. But I might be wrong.

I do see 2 changes that needs to be done:

• Schema Extractor line: 153. Else condition should check if array size == 1

• Schema Extractor: Add MAP type in typeset

[kszosze]

Thanks for the feedback!
Those look easy changes but we´ll take a look into this.
Now we are in the middle of a nice rework, so support lists of maps will be a bit tricky.
But we will handle it asap.
Cheers.

[ismail261]

Thank you for replying fast. One more thing.

I have printed out in console the message it is trying to write and it looks like:
message: {"name": "abcd", "value": "map-array"}
which does not match what is configured.

[kszosze]

Hi there,
Can you please provide an avsc file with you schema?
Just to add this use case to our test.
Cheers

[ismail261]

{ "protocol" : "TestDTO", "namespace" : "com.abcd", "types" : [ { "type" : "record", "name" : "Test", "fields" : [ { "name" : "name", "type" : "string" }, { "name" : "values", "type" : { "type" : "array", "items" : { "type" : "map", "values" : "string" } } } ] } ], "messages" : { } }

Would you be able to give any timeline on when this feature can be available?

[kszosze]

Hi Ismail,
Thanks for the Schema.
There is a branch with a candidate solution for this feature (#39).
If you feel ok to try it, that will accelerate the process.
Cheers!

[ismail261]

Hey. My Java request fails and it fails from master too. When i try to open new java request in jmeter, i get following error: Uncaught Exception java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol. See log file for details.

[ismail261]

I checked out commit from 11th june with 1.5.1 version and it worked.

[kszosze]

Uh, that's strange. I test it before commit. Did you delete previous plugin versions?

[kszosze]

Latest release jar name had change. Since we complet the standalone version now there is a "-plugin-" In the name. This is the right one to use.

[ismail261]

This is the jar name "kloadgen-1.6.1.jar" that it gets builds and install in local repository. I do not see a "plugin" in name in jar version.

[kszosze]

Can you please check the target folder where you are building the plugin. This version should generate a kloadgen-plugin-1. 6.1.jar file . About 30Mb. The one you are using doesn't have all the required dependencies.

[kszosze]

Please verify you are using the Branch #39_Map_array

[ismail261]

Thanks for that it worked. I was copying jar file that got installed to local maven repository and I didn't realize the process had changed.
And yes I am using branch #39_Map_array. Will test it and let you know if it worked for me.

[ismail261]

Can you help with the format for array map in field value list?

[kszosze]

Sure, details are in the readme. But for add values to generate testing data you should write pair key:value comma separated.
Key1:value1, key2:value2,....
From that the system will get those values to generate random maps. Keep in mind that actual implementation will use those values so if you define a map size bigger than testing set you only can get those values.
Now I'm not at home.
Later I'll check how is going

[ismail261]

No worries.

This is what i put in value: [n:1, a:1]

Some of the values that it generates on topics are:
{"name":"abcd","value":[{"a":"1","n":"1"}]} - array with single map object containing 2 parameters
{"name":"abcd","value":[{"n":"1"}]} - array with single map object containing 1 parameters

But it does not generate array with multiple map objects.

My specific requirement is to be able to generate 10 array object with each object containing map of 2 values in it.

[ismail261]

This is my configuration:

It is not generating array of size 5 and it is also not generating string for 20 characters.

{"name":"abcd","value":[{"a":"PUrzT","b":"PIYBO","c":"uNTuA"}]}
{"name":"abcd","value":[{"a":"YdAID","b":"fxBSa","c":"NyBJR"}]}
{"name":"abcd","value":[{"a":"HHLvy","b":"eZIHb"}]}
{"name":"abcd","value":[{"a":"aMXLJ","b":"UCuJr","c":"xzTrk"}]}
{"name":"abcd","value":[{"a":"YmKHS","b":"MSvgi","c":"ksfNu"}]}
{"name":"abcd","value":[{"a":"LDYmV","c":"yJmoO"}]}
{"name":"abcd","value":[{"b":"UQuTq","c":"chOyH"}]}
{"name":"abcd","value":[{"a":"OdqHy","b":"jMbKK","c":"kmdMj"}]}
{"name":"abcd","value":[{"a":"ZAKzN","b":"WqMXa","c":"UNWui"}]}
{"name":"abcd","value":[{"a":"CORZl","b":"DkSNm","c":"zaTub"}]}

[ismail261]

With this configuration:

It still generated array of size 1 but there were 5 parameters in a map every time. I think it is using an array number to generate number of elements in map.

There should be a way to specify 2 numbers now. One for size of array and one for number of parameters required in map.

{"name":"abcd","value":[{"HfwYu":"kDSKm","grwqw":"CZodT","sWWBe":"qUFHq","aLAHA":"sCjLy","NjcjV":"Oawyg"}]}
{"name":"abcd","value":[{"hWJOV":"sLmVG","DofXp":"dRJZh","bHYgJ":"bCwna","GOuPF":"dbTbo","pBIed":"iLElO"}]}
{"name":"abcd","value":[{"xKUSW":"qgpaT","BZQcy":"hOqhu","nZgaT":"KANNi","GbCvv":"ycJJk","GsiKw":"pbAiB"}]}
{"name":"abcd","value":[{"tDCcB":"rIOuV","VVdKS":"zatDt","KvyxZ":"dSsFq","cSHHj":"vxdiO","qyTfe":"lwbEp"}]}
{"name":"abcd","value":[{"AYGNL":"iBeCn","DukeO":"kjQae","qxLjh":"HiWBx","FTlAl":"oFSaw","wnOaw":"SjpYl"}]}
{"name":"abcd","value":[{"QnnvN":"tfIXt","aitKI":"HnRzW","RkXSU":"NQtbQ","OEEtZ":"JJYnv","hORaF":"aqtuj"}]}
{"name":"abcd","value":[{"ddKEv":"FkOzZ","kjmEu":"FAycj","hPoFJ":"EWggq","PPiog":"blgdN","DEeJo":"uxkvB"}]}

[kszosze]

Thank you for all the feedback. At the moment array of map is still something under development.
Only the [5] is used to define the size of the inner collection.
I'll figure out a way to specify the second size. Probably I'll use the 3 column for that.
Next week probably I'll get something. It's ok?

[ismail261]

That's fine. I appreciate so much for you getting this done asap. Thank you very much.
Let me know whenever you are ready with changes and I can help to test.

[kszosze]

Hi Ismail,
Just got some improvements related to the double size request.
Now number between [] will set the array size, in the meantime Field Length will set up the map size.
Keep in mind it you want to generate a map of 20 fixed values, you should set up them in the Field Value List or if you can survive with random values, just leave the Field value list empty but set a value in the Field Length.
Cheers!

[ismail261]

Thanks Jose. Let me give it a try.

What if value inside the map needs to be of specific size too? As in generate a map of 20 fixed values with 45 characters in length.

[ismail261]

Hi Jose,

With this configuration:

It was supposed to produce an array of size 3 with map size of 2. But it did reverse. It produced an array of size 2 with map size of 3

{"name":"abcd","value":[{"BN":"iG","Mr":"MF","gE":"NJ"},{"NV":"Sv","Nf":"dW","ji":"Ty"}]}
{"name":"abcd","value":[{"ib":"OE","Ig":"wN","Pk":"Gc"},{"FB":"fo","ev":"hg","MJ":"Ml"}]}
{"name":"abcd","value":[{"pl":"RF","Kd":"jL","AT":"pW"},{"ue":"no","cL":"Hp","DW":"Gm"}]}
{"name":"abcd","value":[{"Qe":"yY","lN":"Om","ki":"af"},{"PR":"Fv","zp":"Hw","Mk":"BF"}]}
{"name":"abcd","value":[{"KS":"es","Am":"IB","mR":"Ag"},{"FR":"YG","MC":"BO","sf":"wG"}]}
{"name":"abcd","value":[{"Vc":"fy","YC":"nH","qX":"wg"},{"Ln":"dd","VE":"CA","Nm":"MY"}]}
{"name":"abcd","value":[{"ZK":"Fk","Uf":"hS","Ny":"UX"},{"Js":"gJ","PZ":"zX","qo":"rL"}]}

[ismail261]

Hi Jose,

Is there a way to to generate random string for one of the value in the map?

If Field Values list can contain something like this:
[a:1,b:randomString(45),c:randomInt(4)]

[kszosze]

Mmmm not at the moment, maps we generate are same type value.. All the values should have the same time. However, we support variable replacement, that is, you can write [${mygeneratedmap}] and it will replace by the value of the variable with the same name. You can use a Pre-processor to do that, like a Groove, bean shell.... Or values from a csv file...

[kszosze]

"It was supposed to produce an array of size 3 with map size of 2. But it did reverse. It produced an array of size 2 with map size of 3"
I think this is already fixed.

[ismail261]

Hi Jose,

I tested and yes array issue has been fixed. I see correct sizes of array and map written to topic. Also I was able to use pre processor to generate random map values. So that worked too.

I think all looks good at this moment.

[kszosze]

Good to know, will merge that branch them.
Cheers!!

[ismail261]

Thank you Jose for getting this done so soon 👍 Appreciate a lot