Managed VPS: LiteSpeed Crawler

In this article, we explain how to configure the LiteSpeed Cache Crawler on a WordPress website. The goal of the Crawler function is to ensure that all (or at least the most important) pages of a website always have cache available and therefore always load quickly.

Step 1. Install the LiteSpeed Cache plugin for WordPress

Read here how to install the LiteSpeed Cache plugin and configure it.

Step 2. Open the LiteSpeed Crawler page

Within the overview of the crawler page, you will see four colored bars.

While the crawler is running, the colors mean the following:

• Gray: Waiting to be visited by the crawler

• Green: Cache was already present, skipped

• Blue: No cache was present on the page; cache was created via the crawler

• Red: Unable to create cache for this page (excluded)

Step 3. Link the sitemap

This tells the Crawler which pages exist and therefore which pages should be cached. We configure this under the Settings tab.

By default, WordPress already provides a sitemap (wp-sitemap.xml). Optionally, you can also use a WordPress plugin for this, such as Yoast SEO.

Note: For websites with many pages (more than a few hundred), including WooCommerce products, we do not recommend crawling all of them. This often takes too long and can actually slow down the website because the crawl task keeps running continuously. In that case, use a sitemap that includes only the most important pages.

Step 4. Load the sitemap

We do this on the Map tab. On this page, click the Refresh Crawler Map button; this loads the sitemap items. Once the crawler has run at least once, you can also see the status of each page here and whether it has cache.

Step 5. First crawler run

Now that everything is set up, it’s time to manually start the first crawl run. Go to the overview page and click the Manually run button. Then, at the bottom of the same page, click Show Crawler Status. In the visible window, you can now see the live status of the crawler.

It is important to keep track of how long this run takes, as we will adjust the crawler frequency based on this in the next step.

Step 6. Enable the crawler

Now that we have determined how long the crawler needs/can run, we can actually enable the crawler so that it runs automatically. Do this on the Settings tab by setting the Crawler option to ON.

Next, adjust the following value based on how long the run in the previous step took. For example, if the manual crawl run took 45 minutes, set this value to 3600 seconds (1 hour). Always allow a margin of a few extra minutes.