more generic composition

[waynedpj]

ahoy,

first, thanks for the refactor and all this work on Assemble. this helper was the thing that caught my eye and made me give Assemble a try. and your push this morning fixed a bug i was encountering last night when i finally got a chance to try this helper. thanks again ;)

so far the compose helper seems to work as advertised, but i am having a hard time using it in a layout and making in generic enough to work with whatever uses that layout.

here is a simplified version of my setup:

├── releases
│   ├── releases.yaml
│   ├── A
│   │   ├── tracks
│   │   │   ├── 1
│   │   │   │   ├── index.html
│   │   │   │   └── info.yaml
│   │   │   └── 2
│   │   │       ├── index.html
│   │   │       └── info.yaml
│   │   └── A.yaml
│   └── B
│       ├── tracks
│       │   ├── 1
│       │   │   ├── index.html
│       │   │   └── info.yaml
│       │   ├── 2
│       │   │   ├── index.html
│       │   │   └── info.yaml
│       │   ├── 3
│       │   │   ├── index.html
│       │   │   └── info.yaml
│       │   └── 4
│       │       ├── index.html
│       │       └── info.yaml
│       └── B.yaml
└── pages
    ├── about.html
    └── news.html

i am trying to use Assemble to view the collections that are already modeled by the directory structure i.e. the releases (AKA albums) collection has 2 elements A and B, each release collection element has a sub collection tracks which contain the tracks of the release, and pages has simple content pages.

i am trying to use this compose helper in a layout for a release to pull in the track contents in the subdirectory tracks of the release collection. so far i have been unable to set the cwd option for each release folder differently unless i specify each release separately in the global Assemble options.

what i am trying overall to achieve is to have the directory structure be the configuration, allowing new releases to simply be created by making a new directory and adding content in the correct place, instead of explicitly naming everything in the global Assemble config.

i am new to Assemble and perhaps there is another way to do this, or this is not a supported use case at this time. any advice would be appreciated.

thanks again.

[jonschlinkert]

It think this is doable, but it would be a lot easier to explain just by changing some code. Is this repo publicly available, or is there any chance you could put it up somewhere (or even a reduced test case) so we can collaborate on getting it working for you?

[jonschlinkert]

@doowb do you have any thoughts on how we can accomplish this? I like the use case

[waynedpj]

my project is a mess right now as i have been hacking all kinds of things together trying to get this "directory structure as collection configuration" setup working. however, i will either clean it or make a simplified test case repo and get that posted soon.

thanks again.

[jonschlinkert]

Actually, you know what might work... we could allow cwd to be a function, so you could pass in the dirname of each collection as cwd or something like that. would still be good to have something to test against, but this might be a good route to consider

[doowb]

I can take a look. I don't have any ideas off the top of my head, but a reduced test case repo would be helpful.
Also, I just want to clarify something about the collections. When you say collections, you added those to the assemble.options.collections option in the gruntfile and have yaml frontmatter in each "page" that would tell what collection it's part of, correct?

[waynedpj]

@doowb @jonschlinkert working on a simplified test case repo as we speak .. my project is a bit in tatters and i am very new to git/etc.. thanks for your patience.

@doowb yes, i added the collections to assemble.options.collections but i did not add YFM indicating the collection to each page. i assumed that the collection's file directive (i.e. the target inside the assemble task) would indicate the collection's content. e.g. (this is from a YAML file that is loaded into the assemble config object:

releases:
  options:
    compose:
      cwd: <%= project.src %>/root/releases
  files:
    - expand: true
      cwd: <%= project.src %>/root/releases
      src:
        - "*/*.html"
      dest: <%= project.stage %>/releases

thus this would indicate that the releases collection contains any */*.html files in <%= project.src %>/root/releases? part of this i imagine is that i am understanding collections in assemble.

however, what i am going for is the ability to define collections by the directory structure (and possibly YAML/YFM data) instead of adding an explicit directive for each collection inside the Gruntfile. this would allow one to add more items (in this case a new release) to a collection simply by creating the folder and content in the appropriate location (i.e. as a sub directory of releases). maybe i need to create a plugin or helper to accomplish this?

will get that test repo up soon.

thanks.

[doowb]

Okay, looking at that releases config, you'll be picking up all of the html files in the tracks folders.

From the main post, it looks like you're trying to build a list of "index" pages for each release and each "index" page will contain the content of the tracks/#/index.html. I'm I correct in this?

Are you also wanting to render out each individual tracks/#/index.html page? I have some ideas around this and I think it would require adding some pages to the releases/{letter} folder with the collection name in them.

[waynedpj]

From the main post, it looks like you're trying to build a list of "index" pages for each release and each "index" page will contain the content of the tracks/#/index.html. I'm I correct in this?

bingo. a release (AKA album) is basically an index page for the tracks in its release-name/tracks sub directory. i was attempting to use this handlebars-helper-compose to do the pulling in of each track's index.html page, along with any data files, listed in each track sub directory. then i am using that data+content to build the index or track list in the release's index.html.

Are you also wanting to render out each individual tracks/#/index.html page? I have some ideas around this and I think it would require adding some pages to the releases/{letter} folder with the collection name in them.

yes, this is the other issue: each track's index.html page will need to be rendered as well. basically the tracks sub directory of a release is another collection or sub collection. i would like to define a generic layout for any tracks sub collection (much like i can do with releases) and have that applied to any tracks sub collection of any release. i figure this may take some helper/plugin help.

thanks.

[waynedpj]

OK, thanks for your patience and help so far. i created a simple repo to hopefully demonstrate what i am trying to do:

https://github.com/waynedpj/Assemble-releases

all the config and data is in the Gruntfile.js to keep things simple. i am unsure about to specify the tracks sub collection and left that out of the Gruntfile.js for now.

we can consider A and B each to be a release in the releases collection. for each release, the tracks sub directory holds the tracks sub collection that represents the track items that make up this release.

my goals are:

1. define a releases collection; each release may have a tracks sub directory; the content of each track in a tracks sub directory can be pulled into the release's index.html on demand and in a certain order (e.g. via handlebars-helper-compose)
2. keep the tracks ignorant of which release they are a part of; that is decided by which release they are a sub directory of (i.e. avoid hard coding the release inside the track's index.html)
3. if possible, allow the directory hierarchy to specify the configuration/relationships of collections and sub collections to avoid having to list specific collections in Gruntfile.js. not sure if this is possible at all in current Assemble.

thanks again.

[doowb]

K. I think it looks close and we should just have to add some YFM to the index.html files in each release and update the compose src to use the YFM value.

[doowb]

Made the changes to the show rendering the tracks inside the releases' index.html. Rendering the tracks themselves should be just running assemble normally.

[doowb]

Hope this helps

[bludrop]

Hi @waynedpj (and everyone else),

First, I recognized your use case and setup from a ruhoh discussion. Just curious- what made you look at/move to assemble? (this may be a discussion for a different venue)

Second, my setup is a little similar to yours, so I thought I'd share. I'm trying to create a magazine archive consisting of issues and articles. Here is the directory structure I am using:

├── index.hbs
├── issues
│   ├── 1
│   │   ├── article-title.md
│   │   ├── index.hbs
│   │   └── intro.md
│   └── 2
│       ├── article-title.md
│       ├── index.hbs
│       └── intro.md

/index.hbs is the list of issues. The metadata is mostly in the YFM. You can view the repo here: https://github.com/bludrop/ptmag-archive

To display the correct articles on the issue's index page, I am using {{#is ../../../issue issue }}. This is matching the issue number in the YFM of issue's index.hbs to the issue number in the YFM of the article.md. I guess it could be better if the {{compose}} helper was aware that they were in the same folder...

Here is the gruntfile:

      pages: {
        options: {
          layout: 'page.hbs',
        },
        expand: true,
        cwd: '<%= build.src %>/',
        src: ['*.hbs','!_*/**','!index.hbs'],
        dest: '<%= build.out %>/'
      },
       issues: {
        options: {
          layout: 'issue.hbs',
        },
        expand: true,
        cwd: '<%= build.src %>/issues/',
        src: ['../index.hbs','**/index.hbs','!_*/**'], //only do index.hbs
        dest: '<%= build.out %>/issues/'
      },
        articles: {
        options: {
          layout: 'article.hbs'
        },
        expand: true,
        cwd: '<%= build.src %>/issues/',
        src: ['**/*.md','!**/intro.md'],
        dest: '<%= build.out %>/issues/'
      }

I am also new to this—open to comments, suggestions, etc...

[waynedpj]

Hope this helps

definitely helps @doowb thanks so much. i made some comments on the commit but i realize that i should have put them here. copying+pasting ..

thank you very much for the quick help. it works as expected with both releases.

i played with this in my project and was able to get it a little more generic by placing a release.yaml file in the releases folder above A and B and in that file i placed:

# setup default properties for releases
tracksPath: <%= title %>/tracks/**/*.html

which sets the tracksPath property to the current release's title + the standard tracks glob which works. one odd thing: i tried to use dirname instead of title but dirname was always undefined?

so in my layout for releases i now have {{compose src=release.tracksPath}} and for each release it pulls in the correct subdirectory. basically this release.yaml acts like a default value for tracksPath that all releases can use. come to think of it, i guess i could also put tracksPath in the collection config? that does not seem to work either.

anyway, i have a few follow ups that i will post in the original thread once i get my head around this a bit more. thanks again to you and @jonschlinkert for the help. your solution in particular inspired some "a ha!" moments in me once i figured out how it worked.

peace

[waynedpj]

Hi @waynedpj (and everyone else),

ahoy @bludrop .. thanks for taking the time to post your setup, i appreciate it.

First, I recognized your use case and setup from a ruhoh discussion. Just curious- what made you look at/move to assemble? (this may be a discussion for a different venue)

sure, no worries. i really like Ruhoh and Ruby in general, but Ruhoh seems like it is going through a major overhaul and i got stuck in a few places with the current version, particularly with sub collections as sub directories. of course, if i was a Ruby programmer i could have coded my way out of it, but unfortunately i am not. i also have a deadline for getting this site done and so here i am trying out Assemble which is very nice. also, due to its Grunt integration, i found lots of automated tasks which turned out to be useful as this site has lots of graphics that need to be tamed. as always, different tools for different tasks. however, the new Silly stuff that the next Ruhoh is based on looks pretty neat-o.

I am also new to this—open to comments, suggestions, etc...

thanks again for sharing your setup. it looks like you are ahead of me with understanding Assemble in general, and particularly collections. they are pretty powerful and i am just starting to get the hang of them. i am always getting stuck on thinking that the directory hierarchy has all the info for defining collections and i forget about the Gruntfile.js config. however, now my config is starting to look like yours now, especially after this issue.

coming from Ruhoh/Mustache, i also forget about all these powerful helpers. in particular, i am interested how you are using {{compose}} with {{is}} .. though it may take me some time to sort it out.

thanks again.

[waynedpj]

ahoy @doowb @jonschlinkert

moving along thanks to the changes you proposed but had a few comments/questions about this setup:

1. as i mentioned earlier, in an effort to make things a bit more generic and reliant on directory structure, i added tracksPath: <%= title %>/tracks/**/*.html to the context in a release.yaml file to allow the {{compose src=tracksPath}} helper in the release layout to find the tracks for each release and thereby avoiding having to add something like tracks: B/tracks/**/*.html for every release. i realized this evening that this could be simplified even more by simply using dirname and going back 2 levels to get the name of the release from the directory there (really, everything could be named from the directory in this setup). i have been struggling with writing a helper to do this after not finding an already existing method. i will post the helper tomorrow in a separate issue but just in case, does anything easier come to mind to traverse up 2 levels from dirname and use that in {{compose}}?
2. currently i have a tracks collection setup as follows (this is YAML fed into the Gruntfile.js):

### collections of tracks per release
tracks:
  options:
    layout: track.html
    data:
      - <%= project.src %>/root/releases/*/tracks/**/*.{json,yml,yaml}
  files:
    - expand: true
      cwd: <%= project.src %>/root/releases
      src:
        - "*/tracks/**/*.html"
      dest: <%= project.stage %>/releases

with this setup (and referring to the dir layout at the top of this issue) all tracks from all releases get placed in the same tracks collection. this is a problem for various reasons, like pagination through tracks of a particular release. while i realize that i could define a unique tracks collection for each release (e.g. tracks-A, tracks-B), that defeats the purpose of making the configuration reliant on the directory structure and thus more generic/flexible. my searching/testing seems to indicate that this is not easily done with Assemble, i.e. defining unique collections of the same type purely on directory location. does that sound correct?

OK, thanks again and good night.

[jonschlinkert]

I'm glad you're making progress!

all tracks from all releases get placed in the same tracks collection. this is a problem for various reasons

This is exactly what I was thinking when I mentioned making cwd a function. I'll have to look into your use case more, there might be other ways.

my searching/testing seems to indicate that this is not easily done with Assemble, i.e. defining unique collections of the same type purely on directory location. does that sound correct?

I've never encountered this use case, so I'm not sure if it's correct but it's unlikely that Assemble can't resolve it if we try to work through this. Oftentimes the solution ends up being simpler than it initially seems.

To put it another way, since Assemble gives you all of the configuration information as variables (src, dest etc) throughout the build, I think we just need to figure out how to provide the right context to the src or cwd of the compose helper at any given time.

How about using YFM with a custom property in each index that defines either the cwd and/or the src for that collection? e.g. (did you try this already? I might have missed that somewhere):

---
cwd: a/b/c/
---

<!-- post -->
{{#compose cwd=cwd src="*.html"}}
  {{{@content}}}
{{/compose}}

@doowb, I think this is a good use case for the Strings lib. I could see it being handy if paths for the entire target were built dynamically using those replacement patterns

[bludrop]

Is it possible to retain all tracks in the tracks collection, but filter them using {{#is}} or something similar when you need to do pagination or whatever? That would seem to prevent the need to put them in separate collections (although I suppose there could be other benefits to that).

I think this is what I am doing with {{#is}}. The code is below. Basically, I am using {{#compose}} to fetch all articles. Ignoring the section part, I am using {{#is}} to check whether the issue number on the context (from the YFM) of the issue page matches the issue number of the article, also defined in the YFM. I think I tried to dynamically insert the issue number in the {{#compose src}} section (src/issues/<ISSUENUMBER>/*.md), but I couldn't figure out how to do it. So I tried the {{#is}} method and it seemed to work...

{{#compose 'src/issues/**/*.md' }}
    {{#is ../. section}}
    {{#is ../../../issue issue }}

    <li class="media list-group-item">
    <a class="pull-left" href="{{../filename}}.html">
      <img class="media-object" src="{{article_image}}" alt="article thumbnail" style="width:150px;height:100px;">
    </a>
    <div class="media-body">
      <h4 class="media-heading"><a href="{{slugify title}}.html">{{title}}</a></h4>
      <h5>{{subhead}}</h5>
      <p class="author"><a href="/magazine/tova-serkin">{{author}}</a></p>
    </div>
  </li>
{{/is}}
    {{/is}}
{{/compose}}

[jonschlinkert]

I think that should work, but the {{is}} helpers should be on the outside of the block, since you're filtering what's available to the compose helper, not the other way around.

[jonschlinkert]

Okay, this is just to get some ideas flowing. take a look at https://github.com/helpers/handlebars-helper-compose/tree/master/test/fixtures/dynamic. In each dir, a and b, there is a file named book.hbs with the following content:

{{#compose cwd="<%= dir %>" src="posts/*.md"}}
  <p class="content">{{{@content}}}</p>
{{/compose}}

This is just an example of what we can do, but I added a dir variable to the context, which is the dirname for the src file (or rather, the dirname of the file containing the compose helper). Lo-dash templates can be used in the src or cwd hash properties, but only for certain variables. Until I can document that you might have to do some trial and error.

[bludrop]

The reason I put the {{is}} on the inside is because {{compose}} is giving me access to the issue variable (which comes from the content that {{compose}} is bringing into the page. If I put them on the outside, I wouldn't have access to the issue variable as it pertains to the article- right?

[bludrop]

Also, this dynamic dir/lo-dash idea looks great. I will try it out, too.

[waynedpj]

How about using YFM with a custom property in each index that defines either the cwd and/or the src for that collection? e.g. (did you try this already? I might have missed that somewhere):

@jonschlinkert thanks. if i am following correctly, that is what @doowb initially suggested here and what you meant way up at the start of this thread by using cwd.

it works but the problem (IMHO) is that we are duplicating data already available in the directory structure itself: we know that the only tracks we want to compose/import are those in the [current release directory]/tracks/*.html. i am trying to use the directory structure as the configuration and not rely on having to add this path info to each release. this is what brought me to looking for a relative path helper which it seems you are way ahead of me on ;)

[waynedpj]

Is it possible to retain all tracks in the tracks collection, but filter them using {{#is}} or something similar when you need to do pagination or whatever? That would seem to prevent the need to put them in separate collections (although I suppose there could be other benefits to that).

@bludrop i was thinking the same thing that night and i believe using your method with the {{is}} helper will work to compose sub collections, but if i am following correctly it would have to be duplicated on every template that wanted to get tracks from a particular release (e.g. release index page with tracks, pagination of tracks, etc.). since this information is already in the directory structure i am hoping to rely on it alone.

also, having all the tracks in the same collection was similar to the problem i ran into with Ruhoh that eventually got me stuck. there, all the tracks were being pulled into the release collection (since even sub directories under a collection directory ends up in that collection), so again pagination, sorting and other things became problematic.

also, it seems like the Assemble wizards have already started on a more dynamic solution ;)

thanks again.

[waynedpj]

Okay, this is just to get some ideas flowing. take a look at https://github.com/helpers/handlebars-helper-compose/tree/master/test/fixtures/dynamic.

@jonschlinkert thanks, this works great and removes the need to set release.tracksPath as i was discussing above.

obviously i have a lack of understanding about how Assemble works, but why could we not simply use dirname where you used dir? i tried various combinations last night but never could get it to resolve.

also, why is it not working to use dir directly if it is already in the context? e.g.:

{{#compose cwd=dir src="tracks/**/*.html"}}
    dir = {{dir}}
    <li class="track" id="track-{{slugify @title}}" title="{{@title}}">
        <a href="{{slugify @title}}">{{@title}}</a>
    </li>
{{/compose}}

the {{dir}} template inside the compose block works (of course not in this example since this compose never finds anything), but not in the compose line as a value for cwd?

thanks again.

[jonschlinkert]

why could we not simply use dirname where you used dir

dirname is for the dest file path, but we want to get to the src files (unless I'm missing something, which isn't all that unlikely :-)

Also, dir isn't necessarily on the page context at that time, but the advantage to doing it the way I did is that you can append the template with another path string: e.g. cwd="<%= dir %>/foo" :-)

[waynedpj]

dirname is for the dest file path, but we want to get to the src files (unless I'm missing something, which isn't all that unlikely :-)

duh! sorry for missing the obvious answer to my own question.

Also, dir isn't necessarily on the page context at that time

OK. thanks again. may i say that the difference about what is available via {{}} and <%= %> templates can get a bit confusing for this beginner ;)

[jonschlinkert]

templates can get a bit confusing for this beginner ;)

don't be too hard on yourself! it can get confusing for anyone.

As a general rule think of handlebars as what you should use for rendering HTML and Lo-Dash is what you should use for config. So, for example, let's say we have a title variable on the page context:

---
title: Home
---
<h1>{{title}}</h1>
// => renders to <h1>Home</h1>

we can change the context for {{title}} to be whatever we want without actually changing the template in the page. For example, if we wanted to add site.title to the page, we could do this in a couple of different ways:

<h1>{{site.title}}</h1>

which is probably best. Or, if there is a good reason, we could do:

---
title: <%= site.title %>
---
<h1>{{title}}</h1>

Don't get hung up on the use case, there are many other uses for this. I've been wanting to do a blog post about this specific topic for a long time

[jonschlinkert]

also, although this is probably obvious, it might be worth mentioning just as a tactic for creating a "separation of concerns" regarding usage of Lo-Dash versus Handlebars templates. As you know, with handlebars we can't just do:

{{#compose cwd=dir + "**/*.hbs" src="bar/*.hbs"}}
  {{@content}}
{{/compose}}

nor can we nest handlebars expressions like this:

{{#compose cwd="{{dir}}**/*.hbs" src="bar/*.hbs"}}
  {{@content}}
{{/compose}}

but we can do this:

{{#compose cwd="<%= dir %>/foo" src="bar/*.hbs"}}
  {{@content}}
{{/compose}}

And the only reason we can do this here is b/c I make use of grunt.config.process() on both options.hash.cwd and options.hash.src, which recursively expands lo-dash templates in the context of the grunt config.

Another thing I'll be thinking about for this is Handlebars subexpressions, which allows you to use the return value of a helper as a variable in the "parent" helper. So I could see some potential for using subexpressions to build out paths dynamically.

[waynedpj]

As a general rule think of handlebars as what you should use for rendering HTML and Lo-Dash is what you should use for config.

@jonschlinkert thanks for the overview, i appreciate and obviously need it. this rule of thumb for when to use which template system makes sense, but one thing is still unclear to me: do LoDash and Handlebars templates both share the same context/namespace? in other words does {{title}} and <%= title %> always pull from the same source of properties? it seems like they do, though Handlebar helpers are not available as functions within a LoDash template, as well as some other random properties such as page.src (only in Handlebars' context).

again, forgive me for missing these obvious things!

peace

[waynedpj]

And the only reason we can do this here is b/c I make use of grunt.config.process() on both options.hash.cwd and options.hash.src, which recursively expands lo-dash templates in the context of the grunt config.

OK, so to answer my own question from above: it depends ;) as you say, inside this compose helper you use grunt.config. but in general LoDash templates are resolved in the context of the grunt.config object, while Handlebar templates are resolved in terms of the Handlebar context (this inside a helper). never the twain shall meet? i.e. can the Handlebar context ever contain the grunt.config context or vice versa?

Another thing I'll be thinking about for this is Handlebars subexpressions

funny, i was trying to use those the other day then realized that they are only available with the latest release of Handlebars .. see handlebars-lang/handlebars.js#690 i think they would be very useful in these more dynamic setups as you say.