Intermittent results on Gatsby Build

[diondree]

Description of problem

I recently started a new gatsby project following the template repo but I did mine on a smaller scale using typescript. On my blog page I have a query to fetch all the posts I have in Sanity. For some strange when I go to build the project, sometimes I get results and other times I don't. I'm not sure I'm doing something wrong

Steps to reproduce

This is my post schema

{
      name: 'post',
      type: 'document',
      title: 'Post',
      fields: [
        {
          name: 'title',
          title: 'Title',
          type: 'string',
        },
        {
          name: 'content',
          title: 'Content',
          type: 'array',
          of: [
            { type: 'block', 
              marks: {
                annotations: [
                  {
                    name: 'internalLink',
                    type: 'object',
                    title: 'Internal link',
                    fields: [
                      {
                        name: 'reference',
                        type: 'reference',
                        title: 'Reference',
                        to: [
                          { type: 'post' },
                          // other types you may want to link to
                        ]
                      }
                    ]
                  },
                  {
                    name: 'link',
                    type: 'object',
                    title: 'External link',
                    fields: [
                      {
                        name: 'href',
                        type: 'url',
                        title: 'URL'
                      },
                      {
                        title: 'Open in new tab',
                        name: 'blank',
                        description: 'Read https://css-tricks.com/use-target_blank/',
                        type: 'boolean'
                      }
                    ]
                  },
                ]
              } 
            }
          ],
        },
        {
          name: 'excerpt',
          title: 'Excerpt',
          type: 'string',
        },
        {
          name: 'coverImage',
          title: 'Cover Image',
          type: 'coverImage',
        },
        {
          name: 'date',
          title: 'Date',
          type: 'datetime',
        },
        {
          name: 'author',
          title: 'Author',
          type: 'reference',
          to: [{ type: 'author' }],
        },
        {
          name: 'slug',
          title: 'Slug',
          type: 'slug',
        },
        {
          name: 'description',
          title: 'SEO Description',
          type: 'string',
        },
        {
          name: 'category',
          title: 'Category',
          type: 'reference',
          to: [{ type: 'category' }],
        },
      ],
      orderings: [
        {
          title: 'Publishing date new–>old',
          name: 'publishingDateAsc',
          by: [{ field: 'date', direction: 'asc' }, { field: 'title', direction: 'asc' }]
        },
        {
          title: 'Publishing date old->new',
          name: 'publishingDateDesc',
          by: [{ field: 'date', direction: 'desc' }, { field: 'title', direction: 'asc' }]
        }
      ]
}

This is my query

  query BlogPageQuery {
    posts: allSanityPost(limit: 12, sort: { fields: [date], order: DESC }) {
      edges {
        node {
          id
          coverImage {
            asset {
              _id
              fluid(maxWidth: 500) {
                ...GatsbySanityImageFluid
              }
            }
            alt
          }
          title
          slug {
            current
          }
          author {
            name
            picture {
              asset {
                fluid(maxWidth: 150) {
                  ...GatsbySanityImageFluid
                }
              }
            }
          }
        }
      }
    }
  }

Results

I place two console.log statements in my component, one that prints the entire data object and one that prints the mapped posts.
Example of successful build:

$ gatsby build
success open and validate gatsby-configs - 0.611s
success load plugins - 1.319s
success onPreInit - 0.013s
success delete html and css files from previous builds - 0.026s
success initialize cache - 0.018s
success copy gatsby files - 0.299s
warn [sanity] `overlayDrafts` is set to `true`, but no token is given
warn [sanity] Using `watchMode` when not in develop mode might prevent your build from completing
info [sanity] Fetching remote GraphQL schema
info [sanity] Transforming to Gatsby-compatible GraphQL SDL
info [sanity] Stitching GraphQL schemas from SDL
success onPreBootstrap - 0.986s
success createSchemaCustomization - 0.014s
info [sanity] Fetching export stream for dataset
info [sanity] Watch mode enabled, starting a listener
info [sanity] Done exporting!
success source and transform nodes - 0.908s
warn The type `SanityImageAsset` does not explicitly define the field `childImageSharp`.
On types with the `@dontInfer` directive, or with the `infer` extension set to `false`, automatically adding fields for children types is deprecated.
In Gatsby v3, only children fields explicitly set with the `childOf` extension will be added.

success building schema - 0.790s
info Creating blog post page: /blog/introduction/
success createPages - 0.076s
success createPagesStatefully - 0.139s
success updating schema - 0.084s
success onPreExtractQueries - 0.022s
success extract queries from components - 1.686s
success write out redirect data - 0.004s
success onPostBootstrap - 0.002s
info bootstrap finished - 12.477s
success run page queries - 0.074s - 6/6 80.78/s
success write out requires - 0.023s
success Building production JavaScript and CSS bundles - 10.388s
success Rewriting compilation hashes - 0.057s
[                            ]   0.001 s 0/7 0% Building static HTML for pages
{ posts: { edges: [ [Object] ] } }
[
  {
    id: '-e9994206-7306-561b-8ccf-afa07cf3c064',
    coverImage: { asset: [Object], alt: 'Introduction to Dewebgineer' },
    title: 'Introduction To Dewebgineer Solutions',
    slug: { current: 'introduction' },
    author: { name: 'Diondre Edwards', picture: [Object] }
  }
success Building static HTML for pages - 3.805s - 7/7 1.84/s
success onPostBuild - 0.003s
info Done building in 28.057608641 sec
✨  Done in 28.69s.

Example of build without content:

$ gatsby build
success open and validate gatsby-configs - 0.484s
success load plugins - 1.355s
success onPreInit - 0.012s
success delete html and css files from previous builds - 0.021s
success initialize cache - 0.018s
success copy gatsby files - 0.201s
warn [sanity] `overlayDrafts` is set to `true`, but no token is given
warn [sanity] Using `watchMode` when not in develop mode might prevent your build from completing
info [sanity] Fetching remote GraphQL schema
info [sanity] Transforming to Gatsby-compatible GraphQL SDL
info [sanity] Stitching GraphQL schemas from SDL
success onPreBootstrap - 0.658s
success createSchemaCustomization - 0.067s
info [sanity] Fetching export stream for dataset
info [sanity] Watch mode enabled, starting a listener
info [sanity] Done exporting!
success source and transform nodes - 0.694s
warn The type `SanityImageAsset` does not explicitly define the field `childImageSharp`.
On types with the `@dontInfer` directive, or with the `infer` extension set to `false`, automatically adding fields for children types is deprecated.
In Gatsby v3, only children fields explicitly set with the `childOf` extension will be added.

success building schema - 0.586s
info Creating blog post page: /blog/introduction/
success createPages - 0.060s
success createPagesStatefully - 0.141s
success updating schema - 0.064s
success onPreExtractQueries - 0.007s
success extract queries from components - 0.980s
success write out redirect data - 0.005s
success onPostBootstrap - 0.003s
info bootstrap finished - 10.588s
success run page queries - 0.082s - 5/5 61.23/s
success write out requires - 0.018s
success Building production JavaScript and CSS bundles - 10.396s
[                            ]   0.003 s 0/7 0% Building static HTML for pages
{ posts: { edges: [] } }
success Building static HTML for pages - 3.906s - 7/7 1.79/s
success onPostBuild - 0.004s
info Done building in 26.167894984 sec
✨  Done in 26.79s.

Let me know if you anymore information.

[aaronadamsCA]

For what it's worth I also often experience intermittent build problems with missing data.

I noticed your number of page queries differs between the successful run:

success run page queries - 0.074s - 6/6 80.78/s

And the unsuccessful one:

success run page queries - 0.082s - 5/5 61.23/s

This is always my first indicator that something is broken. If I stop, run gatsby clean, and then gatsby develop again, it generally starts working again. Unfortunately this has become a pretty standard part of my dev loop.

I've not yet been able to identify whether the race condition that's causing this is in my code, Gatsby, or the Sanity plugin. The fact you're having this problem with a Sanity starter certainly leads me to think it's the plugin.

Did Sanity ever respond via other channels? I haven't found them very responsive, and I'm starting to worry it's time to look around for another CMS.

[diondree]

@aaronadamsCA that's an interesting observation that I didn't notice originally. Yea something strange is happening and now unfortunately gatsby clean is part of my build process.

There definitely seems to be some kind of race condition but I haven't received any response on this issue. I'm not sure where is the best place to reach out to them but here or twitter didn't work.

It's really a shame cause I feel like sanity is just great.

[rexxars]

This is a very hard issue to debug because we're not able to reproduce it. I've just released v6.0.4 of the source plugin. When building with it, it should finish the retrieval of documents from Sanity with the message "Done! Exported X documents". Could you upgrade to this version and let us know what the document counts are on both successful and unsuccessful builds?

If the document count varies (more than expected) between the builds, that'll indicate some problem with the data fetching. Otherwise it might be a problem in the gatsby layer which we need to go deeper on.

[aaronadamsCA]

@rexxars, glad to hear it, I'll be sure to try some things out shortly (although note I am not OP).

Note that I have identified a confirmed concurrency bug in Gatsby core (gatsbyjs/gatsby#26056). Given you don't make any calls to context.nodeModel.runQuery, I don't believe this is affecting your plugin, however it may have been causing some of my issues. It may also point to the possibility of other concurrency issues in Gatsby core.

The "magic exports" we're required to define in gatsby-node.js can make debugging painful as well, and I know Gatsby is tentatively planning more explicit APIs in v3 (gatsbyjs/gatsby#15277 (comment)) which may help trace some of these issues.

Finally, I believe the "run page queries" count may actually count only non-cached page queries. In other words, this may be an intentional behaviour of Gatsby, with the higher count after gatsby clean merely being a side-effect of eliminating the cache.

The above are all new learnings for me in the past few weeks as I continue to learn more. I'm rebuilding my gatsby-node.js today to try to eliminate my dependencies on runQuery, and I'll let you know if I continue to experience problems (though again I am not the original reporter here).

[diondree]

@rexxars I'm not sure why but upgrading to the latest version seems to have resolved the issue.

[aaronadamsCA]

This is a very hard issue to debug because we're not able to reproduce it. I've just released v6.0.4 of the source plugin. When building with it, it should finish the retrieval of documents from Sanity with the message "Done! Exported X documents". Could you upgrade to this version and let us know what the document counts are on both successful and unsuccessful builds?

If the document count varies (more than expected) between the builds, that'll indicate some problem with the data fetching. Otherwise it might be a problem in the gatsby layer which we need to go deeper on.

FYI only, I've noticed that the document count you've added is consistent during all builds.

I'd suppose if neither of us are experiencing issues anymore, this is probably taken care of 👍 thank you!