Technical SEO, without the jargon.

What technical SEO is (and what it isn't)

SEO has three broad parts: content (what your pages say), authority (who links to and trusts you), and technical (whether search engines can actually access and understand your pages). Technical SEO is the foundation the other two stand on — brilliant content on a site that can't be crawled ranks for nothing.

The good news is that technical SEO is mostly a fixable foundation, not an endless grind. Most business websites have a small number of technical faults — a blocked page, missing structured data, a broken redirect chain — that, once corrected, simply stop holding the site back. In our audit of 421 local business websites, 28.5% had no structured data at all and a substantial share had crawling or indexing issues they were completely unaware of. Fixing those is often the fastest visibility win available.

Crawling: can search engines reach your pages?

Before anything can rank, a search engine has to crawl it — send an automated bot to fetch the page. Several things control this:

robots.txt is a file at your domain root that tells bots which areas they may and may not crawl. It's powerful and dangerous in equal measure: a single stray line (Disallow: /) can hide your entire site from Google. It's one of the first things to check when a site vanishes.

Crawl budget is the finite amount of crawling a search engine will do on your site. For small sites it's rarely a constraint; for large ones, wasting it on low-value or duplicate URLs means important pages get crawled less often. Speed matters here too — a faster site lets bots crawl more pages in the same budget, which now applies to AI crawlers as well, as covered in our AI-era SEO guide.

Internal links are how bots discover pages — they follow links from page to page. A page with no internal links pointing to it (an "orphan page") may never be found. Which is why internal linking, covered below, is both a technical and a strategic concern.

Indexing: are your pages actually in Google?

Crawling is fetching; indexing is storing the page so it can appear in results. A page can be crawled but not indexed. Controls include:

The meta robots tag (and the equivalent HTTP header) can carry a noindex instruction telling search engines not to index a page. Enormously useful for thank-you pages and internal search results — and a classic self-inflicted disaster when a noindex left over from development stays on the live site. If pages aren't appearing, this is the second thing to check after robots.txt.

To see what's indexed: search site:yourdomain.co.uk in Google for a rough count, or use Search Console's Page Indexing report for the authoritative picture — which pages are indexed, which aren't, and precisely why not. Our guide on why a website isn't showing on Google works through the causes in order.

Sitemaps and robots.txt: the instructions you give Google

Two files do most of the talking between your site and search engines:

An XML sitemap is a machine-readable list of the pages you want indexed, with metadata like when each was last updated. It doesn't guarantee indexing, but it helps search engines discover pages efficiently — especially valuable for new sites with few external links, and large sites where crawling could otherwise miss pages. Submit it in Search Console.

robots.txt, as above, controls crawling and should reference your sitemap's location. The two work together: robots.txt says where not to go, the sitemap says where the good stuff is.

Canonicals and duplicate content

Duplicate content — the same or very similar content on multiple URLs — confuses search engines about which version to rank and can dilute your authority across copies. It's rarely malicious; it happens naturally through URL variations (with and without trailing slashes, with tracking parameters, HTTP vs HTTPS, www vs non-www, printer-friendly versions, product pages reachable by multiple paths).

The canonical tag is the fix: a line in a page's code that says "the master version of this content lives at this URL." It consolidates ranking signals onto the version you choose and tells search engines to treat the rest as duplicates. Every page should have a self-referencing canonical as a baseline, and genuine duplicates should point to their master. Getting canonicals right is one of the highest-leverage, least-visible parts of technical SEO.

Redirects: preserving what you've earned

When a URL changes — a page moves, a site is restructured, a domain changes — the old address needs to send visitors and search engines to the new one. That's a redirect, and getting it right is what separates a safe change from a traffic disaster.

The single most important rule of any website redesign or migration: map every old URL to its new equivalent with a 301 redirect. Skip this and you lose the rankings those pages earned — the most common and most avoidable way redesigns go wrong.

Structured data: making pages machine-readable

Structured data (usually Schema.org markup in JSON-LD format) is code that describes what a page is in terms machines understand: this is a local business, here's its address and opening hours; this is a product, here's its price and availability; this is an article, here's the author and date; these are FAQs, here are the questions and answers.

Two payoffs. In traditional search it can earn rich results — star ratings, FAQ dropdowns, prices shown directly in the listing — which lift click-through. And in the AI era it's increasingly how assistants like ChatGPT, Gemini and Google's AI Overviews reliably extract facts about your business to cite. A page without structured data forces machines to guess; a page with it hands them the answer. Given that nearly a third of the sites we audited had none, it's often a straightforward edge. The AI dimension is covered in SEO in the AI era and the practical content angle in content strategy.

Internal linking, HTTPS and mobile

Internal linking

Internal links do three jobs: they help bots discover pages, they pass authority between pages (a link from a strong page lends weight to the page it points to), and they signal relationships that build topical authority. Practical rules: link related pages to each other, make sure every important page is reachable, use descriptive link text (not "click here"), and structure content in clusters — a pillar page linked to supporting pages, as covered in content strategy.

HTTPS

HTTPS (the padlock) encrypts the connection and is both a ranking factor and a trust signal — browsers now warn visitors away from sites without it. It's non-negotiable, and thankfully free via Let's Encrypt on any decent host. A site still on plain HTTP in 2026 is losing rankings and scaring off customers simultaneously.

Mobile-friendliness

Google predominantly uses the mobile version of your site for indexing and ranking. A site that works on a laptop but breaks on a phone is being judged on the broken version. Responsive design — one site that adapts to any screen — is the standard, and it feeds directly into Core Web Vitals, which are measured primarily on mobile.

Diagnosing technical issues: crawl errors and logs

Two sources reveal what's actually happening:

Search Console is the essential free tool — it reports crawl errors, indexing status per page with reasons, Core Web Vitals, mobile usability, structured data validity, and manual actions. If you do one technical SEO thing, verify Search Console and read its reports.

Server logs record every request bots make to your site — which pages they crawl, how often, and what response they get. For large or complex sites, log analysis reveals crawl budget waste and pages bots never reach that no other tool shows. It's advanced, but for a big site it's where the real technical insight lives.

For a fast, plain-English snapshot of a site's technical health, our free website checker flags the common faults — missing HTTPS, no structured data, mobile problems, and more — in twenty seconds.

Frequently asked questions

What is technical SEO?

The work of making a website easy for search engines and AI systems to crawl, understand and index. It covers crawling and indexing controls, canonical tags, redirects, structured data, internal linking, HTTPS, mobile-friendliness and speed — the foundation that content and links depend on.

What's the difference between crawling and indexing?

Crawling is when a search engine's bot fetches your page. Indexing is when it stores that page so it can appear in results. A page can be crawled but not indexed — for example if it carries a noindex tag or is judged low-value.

What is a canonical tag?

A line of code that tells search engines which URL is the master version of a page's content, consolidating ranking signals and handling duplicate content caused by URL variations. Every page should have a self-referencing canonical as a baseline.

What is robots.txt and why does it matter?

A file at your domain root that tells search engine bots which areas they may and may not crawl. It's powerful and easy to get catastrophically wrong — a single stray 'Disallow: /' line can hide your entire site from Google.

What is an XML sitemap?

A machine-readable list of the pages you want indexed, with metadata like last-modified dates. It helps search engines discover pages efficiently, especially valuable for new sites with few links and large sites where pages might otherwise be missed. Submit it in Search Console.

What is the difference between a 301 and 302 redirect?

A 301 is permanent and passes authority to the new URL — the one you almost always want when a page moves for good. A 302 is temporary, signals the original will return, and doesn't fully pass authority. Using a 302 where you meant a 301 can cost rankings.

Why does duplicate content hurt SEO?

It confuses search engines about which version to rank and can split your ranking signals across copies, weakening all of them. It usually arises innocently from URL variations. Canonical tags fix it by naming the master version and consolidating authority onto it.

What is structured data and do I need it?

Code (usually Schema.org in JSON-LD) that describes what a page is in machine-readable terms — a business, product, article or FAQ. It can earn rich results in search and is increasingly how AI assistants extract and cite facts about your business. Nearly a third of sites have none, making it an easy edge.

How do I know if my pages are indexed by Google?

Search 'site:yourdomain.co.uk' for a rough count, or use Google Search Console's Page Indexing report for the authoritative view — which pages are indexed, which aren't, and exactly why. Search Console is free and essential.

Is HTTPS a ranking factor?

Yes — HTTPS is a confirmed Google ranking signal and a trust signal, and browsers now actively warn visitors away from sites without it. It's free to add via Let's Encrypt on any decent host, so there's no reason to be without it in 2026.

What is crawl budget and should I worry about it?

The finite amount of crawling a search engine does on your site. Small sites rarely need to worry; large sites do, because wasting it on duplicate or low-value URLs means important pages get crawled less. Faster sites get more crawled in the same budget.

What tools do I need for technical SEO?

Google Search Console is essential and free — crawl errors, indexing status, Core Web Vitals, structured data and more. PageSpeed Insights covers speed. For large sites, server log analysis reveals crawl patterns. Our free checker gives a fast plain-English snapshot of common faults.

Related: SEO in the AI era · Core Web Vitals · Website migrations · Not showing on Google? · SEO services