Technical SEO
Matthew BoyleTips and tricks around technical SEO
Check your robots.txt file
Robots.txt is a file that sits on your website and tells search engine crawlers which pages they can and cannot access. Get it wrong and you can accidentally block Google from crawling your entire site and wonder why your rankings have disappeared.
It is more common than you think. A developer makes a change, a website migration goes slightly wrong, someone copies a staging site configuration to live. Suddenly your robots.txt is telling Google and other search engines to go away and it quietly obliges.
To check yours, type your domain into a browser followed by /robots.txt — for example yourwebsite.com/robots.txt. If you see a line that reads Disallow: / under User-agent: * your whole site is blocked. That is bad. Google also has a free Robots.txt Tester inside Google Search Console. It lets you test individual URLs against your current rules and see exactly what crawlers can and cannot access. Worth checking, especially if you have recently moved platforms, redesigned your site, or had any developer work done.
Create a favicon
A favicon is the small icon that appears in a browser tab next to your page title. It sounds minor but it's a trust signal — a site without one looks unfinished. It also appears in bookmarks and on mobile when someone saves your site to their home screen.
Most website builders will have a favicon upload option in their settings. You need a square image, ideally 512x512 pixels, saved as a PNG. If you don't have a logo, even a single letter or simple icon works fine. There are free tools like favicon.io that will generate one for you in seconds and give you the right file formats.
Relevant links
Want more Technical SEO tips and ideas?
Add your website to BadgerAlerts and we'll scan it and send you personal recommendations automatically. It's a free service.
Get started freeCompress any images before publishing them
Make sure your images are an optimised reasonable file size, the right dimensions for where they're used, and proper alt text for accessibility.
Huge, unoptimised images slow your page down badly, and page speed is something Google factors into ranking. Slow-loading pages also mean visitors leave before they've even seen what you offer.
You don't need anything technical to fix this. There are free online tools (like TinyPNG or Squoosh) that compress images in seconds with barely any visible quality loss, or most website builders have a built-in compression option you can switch on. Either way, check your images aren't several megabytes each — it's one of the easiest wins available.
Compressing the image keeps the image normal to the human eye but reduces the size for machines loading the image saving time and space.
Structure your pages with proper heading tags
Think of a heading hierarchy like a book. H1 is the title, H2s are chapters, H3s are sections within those chapters. That structure helps a human skim a page, and it does the same job for crawlers.
Search engines and AI bots aren't reading your page the way a person does, they're parsing the document structure to understand what matters and how it's organised. One H1 per page that states what it's about. H2s for your main sections. H3s for anything nested under those. Skip levels or use headings for styling rather than structure, and you make it harder for a robot to make sense of the page, even if it looks fine to a person.
A document that has been properly structured with correct heading tags from H1>H6 helps search engines understand the context of a webpage.
Use Google Search Console's URL inspection tool to see how Google visually sees your page
Google Search Console lets you inspect any URL on your site and see how Google actually renders it — the page as Googlebot sees it, not as you see it in your browser. Bear in mind there are thousands of different devices, screen sizes, and resolutions out there, and the way your page looks on your laptop is not how everyone else sees it, and not how Google sees it either.
If the rendered version looks broken, missing content, or nothing like the real page, something is wrong. JavaScript not rendering, content blocked by robots.txt, or a crawl error — the inspection tool will surface it. If something looks visually off, treat it as a problem that needs investigating. It's one of the most useful diagnostics available and it's free.
While you may be able to view your page correctly, Google and other crawlers will use a different set of browser tools. By using the visual inspect tool, you can see exactly how Google sees your site.
Multilingual SEO
Tips for ranking for multiple languages.
Set up hreflang tags
If you run multiple versions of the same page in different languages, hreflang tells Google which version to show to which audience. Without it, Google might serve the wrong language to the wrong visitor, or worse, see your pages as duplicate content competing against each other.
Each version needs to point to itself and to every other language variant, so search engines understand the full set of alternatives. Get it wrong and you can end up with the wrong page ranking in the wrong country, or pages cannibalising each other instead of working together. Most CMS platforms and SEO plugins can handle this for you, it's just often skipped entirely.
Correctly implemented hreflang tags can direct search engines and AI crawlers to find a foreign language equilvent of the URL it has landed on. These must be setup correctly or you risk penalising your site.
Crawalability
Having efficient crawlability allows search engines and AI crawlers to better analyse your site and understand which sections are more important than others.
Create a sitemap and submit it to relevant tools
A sitemap is a file that lists every page on your website and tells search engines where to find them. Without one, Google is relying entirely on following links to discover your pages — which means anything that isn't well linked internally could get missed entirely.
Most website builders generate a sitemap automatically. If you're on WordPress, a plugin like Yoast handles it. You just have to active it.
If you've built something custom, there are free online tools that will crawl your site and generate one for you. Once you have it, submit it directly in Google Search Console under the Sitemaps section — this tells Google exactly where it is and prompts a crawl. Bing and other services will have its own sections.
If you are vibe-coding, you can ask your AI Agent to create a sitemap for you itself.
Check it periodically. If you add new pages and they're not appearing in the sitemap, Google may not know they exist and something is wrong with your sitemap.
Check your crawl logs
Search engine crawlers don't have unlimited time or resources to spend on your site. They operate within what's known as a crawl budget — a rough limit on how many pages they'll crawl in a given visit. If that budget is being spent on pages you don't care about, such as filtered URLs, duplicate pages, or old parameters, then pages that actually matter may not be getting crawled at all.
Crawl logs are the raw record of every request a crawler made to your server — which pages it visited, when, and how often. Analysing them tells you whether Google is spending its time in the right places. You might find it's hammering a faceted navigation page nobody needs indexed, while ignoring a key service page you actually want to rank.
If you work in soley in an SEO or marketing department you may have to request these from the team that manages the servers. There are also tools like Screaming Frog, JetOctopus, and Cloudflare's that can help you pull and analyse this data.
Once you can see the pattern, you can start fixing it — blocking low-value pages via robots.txt, improving internal linking to priority pages, or cleaning up URL structures that are generating unnecessary crawlable variants.
Matthew Boyle
Founder, BadgerAlerts · ~20 years in digital marketing
Matthew Boyle has over 20 years experience in digital marketing covering a wide range of topic areas. He is available for a one-off consultation or ongoing support.
View my LinkedIn profile