Support nested JSON objects

[devinus]

E.g.:

{
  name: {
    first: 'Devin',
    last: 'Torres'
  }
}

Thoughts:

1. name: DS.attr('object')
2. firstName: DS.attr('string', { key: 'name.first' })

Justification:

Nested JSON objects are too prevalent to require transforming them every time they're used. Models should model the data, not some transformed representation you morph in adapter, and hopefully not through hundreds of transformers.

[tchak]

Even if I was arguing with @devinus on the subject I am actually +1 on this :)
If the idea is approved I could submit a patch

[ghempton]

I'm +1 on this as well. This would be especially nice for people using non-relational databases like couch.

[devinus]

Or ElasticSearch in my case.

[devinus]

Another idea:

name: DS.attr({
  first: DS.attr('string'),
  last: DS.attr('string')
})

[tchak]

Hmm, the problem with this future is potential overhead on write. Read
is juste getPath, but on write, we would have to walk throu the whole
data hash, creating missing nodes...

On 19 janv. 2012, at 20:02, Devin Torres
[email protected]
wrote:

Another idea:

name: DS.attr({
first: DS.attr('string'),
last: DS.attr('string')
})

---

Reply to this email directly or view it on GitHub:
#53 (comment)

[devinus]

Could I have a quick comment on this from either @tomdale or @wycats so I could possibly move forward with a patch?

[tomdale]

Is this for read-only cases or do you expect to write them back to the server as well? How do you detect when the top-level object is dirtied?

You should be able to create multiple embedded models. Given the JSON data:

{
  name: "Steve",

  address: {
    street: "1 Infinite Loop",
    country: {
      short_name: "USA",
      long_name: "United States"
    }
  }
}

You should be able to model this with the following models:

Country = DS.Model.extend({
  shortName: DS.attr('string', { key: 'short_name' }),
  longName: DS.attr('string', { key: 'long_name' })
});

Address = DS.Model.extend({
  street: DS.attr('string'),
  country: DS.hasOne(Country, { embedded: true })
});

Person = DS.Model.extend({
  name: DS.attr('string'),
  address: DS.hasOne(Address, { embedded: true })
});

Are you arguing that this is too much ceremony? It seems like you actually do want each of these conceptually to be separate models, so you can track dirty states separately.

[devinus]

I'm arguing that it can end up being too much ceremony, yes. For example, sometimes I may get back nested JSON that doesn't map well to hasOne attributes. For example:

{
  name: {
    first: 'Devin',
    middle: 'Alexander',
    last: 'Torres'
  }
}

Am I supposed to create a Name model for something like this? These things also aren't coming back with unique id's by the way, how does hasOne handle that?

[tomdale]

Since you are dealing with crazy, non-conventional JSON anyhow, couldn't you normalize the data in your adapter before you send it to the store?

[devinus]

This kind of nested JSON is all over the place: CouchDB, ElasticSearch, Neo4j... I'd argue that there's nothing crazy or unconventional about it. I'd also argue normalizing data in an adapter isn't what an adapter should be for either. I've had thoughts of maybe introducing some type of Transformer, but really--shouldn't the model model your data and not the other way around? Your thoughts?

[NickFranceschina]

+1 ... agree with @devinus . sometimes nested json properties are actually part of the "top-level object"... so then if any property inside of them changes, it would "dirty" the whole thing. and yes, for our app some of these nested-json sections are indeed being serialized and sent back to server

currently, can't use hasOne because nested models dont have "primary keys". could fake it in adapter, as mentioned... but that's extra work to make two unrelated concepts fit together (these aren't separate "database objects", which is really what hasOne/hasMany is doing for us.... so I'd be working around the framework)

will find another workaround for the timebeing...

[wycats]

Are you willing to write out the entire object when it changes, as opposed to being able to modify the object directly? Something like:

Person = DS.Model.extend({
  attrs: DS.attr('object')
});

var person = Person.find(1);

var attrs = person.modify('attrs', function(attrs) {
  attrs.age++;
});

// this would be illegal
person.get('attrs').age++;

The reason for this is that if you modify a conceptually whole attribute by mutating it directly, we don't know that the attribute has changed. We don't want to observe changes to the hash directly, because that may mutate the object in ways that harm its ability to be serialized back (for example, observers add on ES5 accessors to warn users who try to access them directly that they should be going through set/get; even without that, the object needs to be annotated with some extra observability information, which changes it from being a raw, easily serializable object to an extended object).

With that said, would you be comfortable with a .modify API (or something like it; open for suggestions here) for object-like attributes? If so, I think we could arrange it. Unfortunately, because we don't want to directly observe the object, it won't be easy for us to detect mistakes where people try to directly modify it :/ (we wouldn't want to make geting illegal, of course). One thing we could possibly do would be to use ES5 seal where available to try to trap mistakes, but the error messages won't necessarily be easy to figure out (because they'd come from the browser, not Ember).

Suboptimal all around.

[wycats]

Another idea is to return a proxy when you request the object (with [set]UnknownProperty), which would return the underlying value (or another proxy if it was nested data), and also mark the record as dirty when anything was set on it.

This would have the benefit of being mostly transparent, but @tomdale is still skeptical. I think it can work ;)

[devinus]

@wycats You're assuming we're only working with JSON that's nested one level deep.

[NickFranceschina]

I understand what you're saying... and agree that it is suboptimal. I think hasOne() actually works pretty closely enough... it would give me the control to build a Model for my nested JSON, which allows me to determine what sub-properties are being observed and how... if I want to do that. but currently that pattern:

• requires IDs for each model (I get errors if I dont fake the IDs... even with {embedded:true})
• I dont see a way for a nested-model to notify its parent that it has changed... no way to link the parent's "isDirty" to the child's "isDirty". I think maybe DS.belongsTo() might have done provided something like this??... but it doesn't seem to be in the 0.9.5 code

feels like there would be a way to change the association code a bit (hasOne, hasMany)... so then you're not inventing a DS.attr('object') where you dont exactly know what the developer wants you to do with it. we can make an "object" transform now... and do the work there on full set/get... but it feels like the tools are already here for the developer to be explicit with the model schema using DS.Model and build up a nested schema as complicated as they want

[NickFranceschina]

looks like a duplicate: #100

[wycats]

@devinus nope. note that I said "or another proxy if it was nested data".

@NickFranceschina I also thought about reusing the hasOne code, but note that hasOne doesn't mark the parent object as dirty. Additionally, it wouldn't work for nested objects without even more shenanigans (you would want changes, however deep, to mark the parent as dirty). I think the proxy solution is the best solution we have so far.

[devinus]

@wycats Ah, my bad.

[garth]

I think that there is still a case for treating primitive arrays attributes seperate from object attributes. We can use syntax like DS.attr('date[]') and easily use transforms and manage dirty state. #96

[jfgirard]

+1 I'm about to start a project using couchdb. It makes a lot of sense to have object attribute type to support nested json objects.

[analytically]

+1

[wycats]

We haven't forgotten about this issue :D

[stefanpenner]

bump to restart discussion...

[leehambley]

What's the current status of this issue, looks like it's effectively dead?

If the issue won't be fixed, what is the recommended workaround?

[elwayman02]

https://github.com/lytics/ember-data.model-fragments was the way to go for us, @lolmaus

[lolmaus]

@elliterate, model-fragments claims to be incompatible with the current version of Ember Data. I also find it risky to depend on a third party extension that hacks Ember internals: even after becoming compatible, it might break with next update of Ember Data.

I managed to come up with a solution using native Ember features: http://emberjs.jsbin.com/vapama/1/edit?html,js,output

Highlights:

• The model has a JSON-like attribute thedata.
• The template renders checkboxes for every item contained by the attribute. Each checkbox has checked set to the corresponding item.
• Checkboxes are customized Ember.Checkbox that send a stain action to the controller on change.
• When the action is triggered, the controller marks the record as dirty.
• For the controller to know which record to stain, i have to pass the record from the template to the checkbox, and the checkbox has to pass the record to the triggered action.

The latter will require a lot of utility code, so don't hesitate to suggest a more concise approach if you have something in mind.

[lolmaus]

Here's a slightly more compact version: http://emberjs.jsbin.com/vapama/3/edit?html,js,output

Instead of having the checkbox ask the controller to stain the record, the checkbox stains the record itself.

[lolmaus]

An even shorter version, now reusing the input helper rather than subclassing it: http://emberjs.jsbin.com/vapama/4/edit?html,js,output

Can you guys please review this and tell your opinion on whether it's an appropriate solution?

UPD: http://emberjs.jsbin.com/vapama/5/edit?html,js,output

• Used an observer of the checked property rather than a callback assigned to the change attribute. Not sure which way is better, though.
• I discovered that staining would happen before the change would be applied to the object. Resolved this issue by wrapping the staining into an Ember.run.later.

UPD2: http://emberjs.jsbin.com/vapama/8/edit?html,js,output

• I've realized that staining the record does not announce the change in the JSON-like property, so i've decided to replace staining the record with announcing the change in the property. But i've discovered that announcing the change in the property does not result in staining the record, so i had to have both staining the model and announcing property change.
• Extracted most of the code from Ember.Checkbox to Ember.View, so that it can be reused in other helpers.

[lolmaus]

Okay, this works in a JSBin, but in real-life scenario this approach requires so much utility code that it becomes simpler, more error-proof and definitly more Ember-way to use nested models.

[lolmaus]

Yay, Model Fragments have been updated to support Ember Data beta 14! :D

[KTKate]

I'm also trying to do a DS.attr('json') type thing to send back to the Rails backend. My situation is very similar to that of @alvincrespo where the user needs to be able to specify the key and the value. In my case we need the user to be able to specify multiple key/value pairs if needed. I'm hesitant to try ModelFragments because I'm wary of adding more dependencies.

[slindberg]

@JKGisMe I have the same need in my app for which model fragments works well. I understand your reluctance to add more dependencies, but model fragments was born out of your exact need. Ember Data may one day support nested id-less records like this, but that day is far in the future.

If you are interested, this is the rough pattern I'm using:

app/models/thing.js

export default DS.Model.extend({
  config: DS.hasManyFragments('config_field')
});

app/models/config-field.js

export default DS.ModelFragment.extend({
  name: DS.attr('string'),
  value: DS.attr('string')
});

app/templates/thing.hbs

{{each config as |config|}}
  <div class="form-control">
    <label>{{config.name}}</label>
    {{input value=config.value}}
  </div>
{{/each}}

This will serialize thing records with a config property that is an array of objects with name and value properties. If your API expects an object, you can override the thing serializer:

app/serializers/thing.js

export default DS.RESTSerializer.extend({
  serializeAttribute: function(snapshot, json, key) {
    if (key === 'config') {
      json.config = snapshot.attr('config').reduce(function(config, fieldSnapshot) {
        config[fieldSnapshot.attr('name')] = fieldSnapshot.attr('value');
        return config;
      }, {});
    } else {
      this._super.apply(this, arguments);
    }
  }
});

[igorT]

👍 for model fragments

[delapuente]

Hi people. I'm following this approach. In my case I want to store a info object for a channel entity I'm working on:

var Channel = DS.Model.extend({
  name: DS.attr('string'),
  info: DS.attr({ defaultValue: () => Info.create() })
});

var Info = Ember.Object.extend({
  version: DS.attr('string'),
  description: DS.attr('string')
});

What is bad about this approach?

Thank you.

[igorT]

Well, I have no idea what happens when you put a DS.attr inside something that is not a DS.Model it definitely isn't a supported use case. You could have Info be a normal object, but you don't get dirtyness/rollback and some of the normalization/serialization niceties, but it definitely works for simpler use cases

[KTKate]

Thank you @slindberg I didn't see this earlier!. That looks very clean and probably would have worked great. The specs for our project changed a little and I ended up just making a whole new model & accompanying component in Rails & Ember which I then convert to customized JSON in the end from Rails.

[billpull]

Is ember-data-model-fragments now incompatible with ember-data 1.13.2

[igorT]

It will be compatible soon

[boehlke]

Hello guys, I followed this discussion because I have the same use case (nested documents).
After some thinking, I decided the solve the problem by creating a server-side component which flattens the data so that it can be used with ember-data.
My input to this discussion: Probably you do not want embedded records in the first place. Think about the complexity you introduce the your ember app. It is probably better handled on the server.

Having said that, I vote against having model-fragments in ember-data.
👎

[lolmaus]

@boehlke You assume that any data can be flattened, but that's absolutely not the case.

[boehlke]

@lolmaus Sorry, I cannot think of an example JSON object which cannot be flattened. Can you provide one? Probably that's easier than for me to prove that every JSON can be flattened. Thanks!

[wildchild]

This thread is turning into masturbation. Embedded objects are parts of JSON documents. If ember won't support it, fuck it. This framework is too "idiomatic" to be usable.

[lolmaus]

@boehlke Imagine a hash that has arrays which contain hashes which contain arrays of hashes.

You can of course undestructively flatten it by producing really elaborate key names. But working with such a flat structure from Ember will be a nightmare. You won't be able iterate over specific sub-arrays, etc. That's just not a working solution.

[stefanpenner]

The reality is, depending on the scenario people want both flattened and nested structures are ideal.
As with a DBMS, if one wants to keep data consistent, normalized structures are ideal. Many times though, this constraint isn't important and more natural graphs become easier to deal with.

Compromises exists, but as ED's goal was consistency, that explains its current state (not its final form).

If ember won't support it, fuck it.

Ember works fine with deeply nested data-structures out of the box.

Ember-data also works with deeply nested documents.

I believe the pain people are noticing. Is they want there nested blobs to behave like ember-data models, in both a read/write/serializer/deserialize fashion. Ember-data today, is largely the following units (on top of ember), and models have nice patterns, such as:

1. an identity map (to ensure you are always dealing with a single canonical entity)
2. a set of adapters (to normalize foreign endpoints/api)
3. a set of serializers (meant to contain all logic related to transformation of data to/from the server)
4. a mechanism of relationship syncing
5. a mechanism to detect changes, and the ability for a developer to commit said changes back a foreign data-store

Expanding these patterns to the nested id'less entities, is something we are interested in exploring. the model-fragments project is showing promise, and is a very intriguing interpretation of the problem. Unfortunately, it relies on private API and monkey patching to accomplish its goal. One of our goals moving forward is, to ensure a stable public API for model-fragments. This will prevent the upgrade turmoil currently introduced.

Now as for model-fragments becoming part of ember-data itself, it's too soon to know for sure. It is great to push this logic into user-space, as it will come with some caveats (and the problem space needs some more fleshing out) The important part today is for both model-fragments and ember-data to work better in tandem. This will allow the flexibility for authors to experiment, but also the stability to ship real projects with it.

[fivetanley]

@wildchild I would advise you to read the community guidelines. I do not mean to belittle your frustration but this kind of behavior is not acceptable in the ember community. http://emberjs.com/guidelines/

[devinus]

@stefanpenner

Ember works fine with deeply nested data-structures out of the box.

I just ran into this last night:

{{upload-field inputId="desktop-hero" accept="image/*" signURL=signURL uploadedURL=model.templateSettings.context.hero.src}}

Because model.templateSettings = {} and model.templateSettings.context.hero doesn't exist yet, the value is never updated.

I am forced to do this as a workaround in afterModel:

    if (isNone(get(model, 'templateSettings.context.hero'))) {
      set(model, 'templateSettings.context.hero', {});
    }

[stefanpenner]

@devinus a better place would be to add that to your serialization code.

Also, those questions are better asked on SO.

[devinus]

@stefanpenner What question?

(Also, this is before serialization even happens. I was just providing a counter-example to your claim that "Ember works fine with deeply nested data-structures out of the box." In my experience, it does not.)

[aivanov93]

@lolmaus I'm having trouble figuring out how to nicely flatten a JSON.
I have a similar problem to this SO question http://stackoverflow.com/questions/25220442/json-api-and-translations-with-ember-data and I really like the first answer with nesting everything under the translations field.
Sure, I could flatten everything to title_en, title_fr, but this way on a change of locale, I would have to manually change each field.
Any nice solution to this case?

[lolmaus]

@aivanov93 ember-data-model-fragments seem to fit nicely into your use case.

But I would attempt a completely different solution. I would split the model into two parts, e. g. post and post-translation. A single post would look like this then:

{
  id: '1'
  type: 'post',
  relationships: {
    translations: [
      {id: '1', type: 'post-translation'},
      {id: '2', type: 'post-translation'}
    ]
  }
}

{
  id: '1'
  type: 'post-translation',
  relationships: {
    post: {id: '1', type: 'post'}
  },
  attributes: {
    lang: 'en',
    title: 'Hello world!',
    body: "Wonderful weather we have today, don't we?"
  }
}

{
  id: '2'
  type: 'post-translation',
  relationships: {
    post: {id: '1', type: 'post'}
  },
  attributes: {
    lang: 'ru',
    title: 'Здравствуй, мир!',
    body: 'Чудесная погодка сегодня, не так ли?'
  }
}

This approach will also let you collect language-specific comments to posts.