Script rendering and indexing: Warning tales and how you can avoid them

Recently, I read Ziemek Bucko’s fascinating article, rendering queue: Google needs 9X more time to crawl JS than HTML on the Onely blog.

Bucko shared a test that showed significant delays in Googlebot’s response to links in JavaScript-reliant pages, compared with links in plain-text HTML.

Although it’s not a good idea for a website to only rely on one test, my experience is comparable. Many websites that rely too heavily on JavaScript (JS), have I seen and helped. I know I’m not the only one in this regard.

My experience shows that JavaScript-only content takes longer to be indexed than plain HTML.

I can recall many instances when frustrated clients called and emailed me asking why their items weren’t appearing in search results.

The challenge was evident in all cases except one, which were built on a JS only or mostly JS platform.

Before we move on, let me clarify that this isn’t a “hit article” about JavaScript. JS is a valuable tool.

It can be used to do tasks that other tools are unable to perform, just like any other tool. I don’t have anything against JS. It’s not against me to use it when it doesn’t make sense.

There are many reasons to use JS sparingly for everything.

These stories are based on my own experiences.

1. Text? What text? !

An old site that I helped was relaunched using a new design and heavily on JavaScript.

The new site went live on the same day that organic search traffic dropped to almost zero. This caused panic among clients.

Quick investigation revealed that the site was not only slower than expected (see next stories), but also that Google’s live pages test found the pages blank.

After a thorough evaluation, my team concluded that Google would take some time to render pages. It became evident that there was something else going on after a couple of more weeks.

To understand what was going on, I met up with the lead developer of the site. They shared their screen with me as part of our conversation to show me the back end.

This is when the “aha!” moment occurred. As the developer walked through the code line-by-line in their console I noticed that the text on each page was loading outside of the viewport. Each page was loaded using a line CSS, but was pulled into the frame by some JS.

This was done to create a fun animation effect in which the text content “slid” into focus. The page rendered slowly in the browser so the text was visible when it was finally displayed.

Users were not able to see the slide-in effect. Google didn’t seem to be able to detect the slide-in effect, so I did not see any content.

Traffic numbers began to rise after this effect was removed.

2. It is just too slow

There could be many stories, but I will summarize some of them here. JS platforms such as AngularJS or React are great for quickly developing websites and applications.

These are ideal for websites that require dynamic content. Websites with a lot dynamically driven static content are facing a challenge.

A number of pages on a website I evaluated scored low in Google’s PageSpeed Insights tool (PSI).

It took me over 1MB to find 90% of the JavaScript I had downloaded, so I used the Coverage report from Chrome’s Developer Tool across the pages.

This blockage time is almost 8 seconds if you look at it from the Core Web Vitals perspective. All the code must be downloaded and executed in the browser.

The development team pointed out that frontloading all JavaScript and CSS required on the site will speed up subsequent page visits. Since the code will be stored in browser caches, the team discussed this with them.

Although I was a former developer, the SEO within me couldn’t accept that Google’s negative view of the site’s user interface would likely degrade organic search traffic.

My experience shows that SEO is often lost to a lackluster desire to make changes once things are launched.

3. This site is the slowest ever!

Similar to the story above, I reviewed a website that scored zero on Google’s PSI. I had never before seen a zero score. There were many threes, twos, and ones, but never a zero.

I will give you three guesses as to what happened to the site’s traffic and conversions. The first two are not valid!

Sometimes it’s more that JavaScript

Excessive CSS, large images, and autoplay background videos can slow down download speeds and cause indexing problems.

These are the subjects of two articles I have written about:

• 
  Content that improves user experience
  
• 
  Bad website design for SEO, UX, and business
  

In my second tale, for example, I found that the sites were also using excessive CSS that wasn’t needed on most pages.

What can SEO do in such situations?

This problem requires close collaboration between SEO, developers, and clients or other business groups.

Building a coalition is delicate. It involves both giving and receiving. You must determine where compromises are possible and then move accordingly as an SEO practitioner.

Start at the beginning

SEO should be built into websites from the beginning. It is difficult and costly to update or change a website after it is launched.

Participate in the web development process from the beginning, when the business goals, specifications, and requirements are established.

It is a good idea to include search engine bots in user stories early on so that teams can learn their quirks and help to get content indexed efficiently.

Teach

education is an important part of this process. You need to inform developers about the importance and benefits of SEO.

Leave your ego behind and see the world from the perspective of other teams.

Assist them in understanding their needs, and determining a balance between SEO best practices.

It can be helpful to have a lunch-and learn session, and invite some food. It helps to break down walls by sharing a meal with others during discussions.

Over a few pizza slices, I have had some of the most productive conversations with developers teams.

For existing sites, get creative

If a site is already up and running, you’ll need to be more creative.

Often, developers move on to other projects. They may not have the time or the resources to “fix” the things that were working as per the requirements.

Clients and business owners may not be willing to spend more money on a website project. This is especially true for websites that were recently launched.

Server-side rendering is one possible solution. This can be a great way to speed up the process and offload client-side work.

This combination can be combined with server-side rendering caching plain-text HTML content. This is a good solution for semi-static or static content.

This saves overhead on the server side as pages are rendered only when content is changed or on a regular basis, rather than each time it is requested.

Minification and compression are two other options that may help speed problems, but not completely.

Minification reduces empty spaces between characters and makes files smaller. GZIP compression is available for CSS and JS files.

Compression and minification don’t solve blocking time problems. They can reduce the time it takes to download the files.

Google and JavaScript Indexing: What’s the deal?

Long time ago, I believed that Google’s slowness in indexing JS content could be attributed to the higher processing costs.

Based on how I heard it described, it seemed logical.

• All the plain text was captured in one pass.
• To grab, process, then render JS, a second pass was necessary.

The second step required more bandwidth and processing speed, so I assumed that it would take longer.

I asked John Mueller, Google’s Twitter account manager, if this was a reasonable assumption. He gave me an interesting response.

According to him, JS pages do not have a high cost factor. Residing pages that have not been updated is what Google considers to be expensive.

The most important thing to them was the usefulness and relevance of the content.

The post JavaScript rendering, indexing: Cautionary tales & how to avoid them originally appeared on Search Engine Land.