LiteSpeed Crawler

In this article, we will explain how to enable the LiteSpeed crawler. With the LiteSpeed crawler, you ensure that the cache is preloaded, which always provides caching for your website and makes it load much faster.

Step 1. Install the LiteSpeed caching plug-in

Read here how to install the LiteSpeed Cache Plugin.

Step 2. Open the LiteSpeed crawler page

Within the overview of the crawler page, you will see four colored bars under the heading 'Status'.

During the crawl, the colors mean the following:

• Gray: size of the sitemap / number of pages.
• Green: already cached on the respective page is skipped.
• Blue: No cache on the page, caching made via the crawler.
• Red: Unable to create cache on this page.

Step 3. Setting up the sitemap

This will let the LiteSpeed crawler know which pages are present on your website and which pages need to be cached. We will set this up under tab [6] 'Sitemap Settings'.

It is important that a valid sitemap is present, think of the Yoast SEO plugin, for example.

Step 4. Loading the sitemap

We do this on tab [2] 'Map', click on the 'Refresh Crawler Map' button on this page.

After this, you will see if there is a valid sitemap with content.

Step 5. Setting up the crawler

On tab [4] 'General Settings', we will set up the crawler, which is an important step! If this is not set up correctly, your crawler function will not work properly. On this page, there are a number of factors, including:

• how often should the crawler run?
• how long between the so-called runs?
• when should the crawler start again completely to include any new pages + products in the cache?

First, we will set the parameters, which are always scalable as we do not know beforehand how long it will take for the website to be 'fully' cached. There are multiple factors at play, including:

the number of pages involved. the server load. the efficiency of the website.

 

The settings:

We set the 'Run Duration', or how long the crawler runs each time, to 300 seconds (= 5 minutes).
We also set the 'Interval Between Runs' to 300 seconds.
Then we adjust the 'Server Load Limit' to 3. For now, we leave the rest as is.

Step 6. The first test run

We will now manually start the crawler for the first time to see how long it takes for the crawler to cache all the pages.

We do this on tab [1] 'Summary'
Click 'Manually run'

In this example, the website has 1571 pages.

Under the heading 'Watch Crawler Status', you can see live how quickly and where the crawler is located.

As soon as you press the 'Manually Run' button, you will see the crawler status.

Several things are important here, first the 'size' of the number of pages within the sitemap and the position where the crawler is currently located. The crawler will now cache the site in moments of 5 minutes every 5 minutes.

We now see that the crawler has finished, the status indicated is 'end'.

Now we will see how long it took the crawler to fully cache the website, in this case, it is 15501 seconds (260 minutes). Now that we know this, we can set how often the crawler should start completely caching again.

We will go to the [4] 'General Settings' tab for this, and we will now adjust the 'Crawl Interval'. As noted above, the crawler took 15501 seconds to fully cache, so we will add some margin and set the interval to 18000 seconds (300 minutes) in this case.

Step 7. Turning on the crawler. Now that we have figured out how often/long the crawler should/can run, we can actually turn the crawler on so that it runs automatically. We do this on the [4] 'General Settings' tab by simply turning the crawler 'On'.