Meta tags for SEO - SEO Guide 2019
- Meta Title tags
- Meta Description tags
- Alt tag
- Canonical tag
- Social tags
- Responsive design meta tag
- Robots meta tag
- Heading tags
Title tag best practices
The first content visible to the user in a search engine results page is its ‘Title’ defined under the head of HTML.
Page title influence visitors more than any other factor in SERP. A compelling page title decides and helps the visitor to click.
A page title is the first impression of your digital asset; every title should be well written, concise and clear.
A well-written title includes
- User perspective, write to please user not search bot.
- Keyword, try to write a compelling title starting with your keyword.
- It needs to be short, unambiguous and concise.
- Adding modifiers improves click through, e.g. a guide, cost calculator, contact us, nine tips, Do’s & Don’ts etc.
- A well-written title describes page attributes in a single line.
- Mentioning your brand name in a title helps in exposure and build trust with search bot & users both.
Things to avoid
- Avoid keyword stuffing in the title tag.
- Strictly avoid misleading content. E.g. a freelancer mentioning words like agency or company in the title. A trading company represents manufacturers in their title.
- Grammar mistakes
- Writing duplicate titles
Title tag character limit
Keep it under 60 characters is the best bet, Google typically displays 50-60 characters
Google typically displays the first 50–60 characters of a title tag. If you keep your titles under 60 characters, our research suggests that you can expect about 90% of your titles to display properly. There's no exact character limit, because characters can vary in width and Google's display titles max out (currently) at 600 pixels
Meta Description tag – how to write and best practices
The description is visible for the user and search engines both. Writing compelling descriptions is a critical part of on-page SEO.
Description tag is considered as crucial for search engines as the label is used to the indexing of web pages. The name includes content for the user, which further increases its importance.
Users correlate descriptions in conjunction with the titles before deciding to click a link. Relevant descriptions from the user’s perspective are considered best.
A compelling description tag starts with
- A user perspective,
- Includes your keyword as well
- Attention-grabbing tag lines. E.g. A £10 deal, pay as you go, website Support starting £20.
- Include brand name
- A well-written description tag always supports and best describe the title and URL.
- The description tag should be within the standard length of 150-160 characters.
Length of the meta description tag
Starting with 50 and should not exceed 160 characters. Going beyond the prescribed limit will not help in rankings, the goal is to provide relevant information and drive clicks.
When not to write a Meta description
Consider ignoring to write a Meta description if you are targeting more than 2 or 3 highly searched terms, probably short tail keywords. The basic idea is to give a free hand to search engine, allowing them to crawl the way they want.
Explicit declaration of Meta description may hinder the search engines from taking natural relevance resulting in lower SERP.
Next time if you are bidding on highly competitive keywords, make sure to let the search engine decide and present the Meta to the user.
Meta Keyword tag SEO
What’s the news on the Meta keyword tag?
Google doesn’t use the Meta keyword tag in search rankings. People started misusing the keyword tag and abusing it by over stuffing to manipulate search rankings.
Google had to announce in Sep 2009, that they are officially disregarding the Meta keyword tag in search ranking algorithm.
URL optimisation - SEO Guide 2019
How to optimise URL’s for SEO
Uniform resource locator plays a vital role in SEO, making URL’s user and search engine friendly is necessary.
An optimised URL receives a higher click-through and a chance to gain a better position in SERP.
A well-defined URL is readable; the aim is to achieve and serve the intent of the user seeking for targeted content.
In a most basic sense, the purpose of any given URL is to serve the human audience. Hence the more readable is, the better.
Hopefully, the below illustration will help you understand the difference in readability.
- Try to achieve readability, keeping in mind user intent. Including dynamic content in URL’s creates a deceiving impression resulting in a lower click-through
- Keep them short and concise, using multiple folders for the sake of stuffing a keyword to manipulate the search engine is deceiving. E.g. Bitvero/seo/seo-onsite/seoURL/url-optimisation
- Consider keywords in URL; users feel confident when they observe the required keyword in the URL while hovering the link in the browser, which results in a higher click-through.
Secondly, keywords in URL influence user intent in search, except a higher click-through in URLs including the targeted keywords.
4. Try to match keywords appear in title with those in URL’s, to enhance the overall performance, see the above example.
5. Including secondary words are not necessary, e.g. in, but, and, or, because, for, etc. unless it makes sense either.
6. Avoid using a hashtag # unless necessary; a fragment identifier is used to redirect the user to the part of the page for a unique content experience, use it with caution.
7. Case sensitive URL’s, the best practice is using a lower case, Linux/Unix servers are case sensitive in interpreting URL’s, they will consider a URL co.uk/sEO unique as compared to Bitvero.co.uk/seo
8. Use of Hyphen (-) as a preferred separator; however, search engines are well versed with an underscore (_) as well. Hyphen gives a better user experience.
9. Avoid stuffing of keywords at all cost, which looks spammy.
10. Consider redirection to one hop, multiple redirects degrades the user experience and makes the page slower. They sometimes frustrate mobile users creating a wrong impression.
Header tags hierarchy
A full heading structure maintains a pyramid upside down.
Always re-check the webpage and blog post for a well-defined heading structure.
Never use headings for the sake of it, try to make the best utilisation of headings keeping in mind H1 to H3 are necessary from a user perspective if content demands.
Another compelling structure
Heading structure depends on the flow of content, even compelling content looks and read boring without the presence of meaningful headings and sub-headings.
Heading 1 – The Main Heading
Heading 1 is the main heading of the page, you can keep any number of main headings in the page, and however, the best practice is to restrict the main heading of the page with only one title unless necessary.
In a conversation, John from Google confirmed the fact that we can use as many H1 tags.
Header tags best practices in SEO.
- A short and concise H1 is best. However, H2 can be extended/short both.
- Using keywords in the heading is a good practice, avoid stuffing and unnecessary pushing them in the headings.
- Try to use exciting headings, e.g.
- Are you bidding on all the wrong keywords?
- Five rules of page optimisation.
- How to write a compelling page title?
- Provide a meaningful structure
- H1 is the main title just a title of the book
- H2 are sub-titles, like headings to define different chapters in a book.
- H3-H6 serves multiple subsequent headings, like explaining of secondary & tertiary titles in different sub-titles.
- Use headings to break up content and increase readability
Image optimisation for the web- SEO Guide 2019
Why is image optimisation important?
Images play a significant role in the development of our web pages. Google has recently started supporting Web-P image format that provides a superior compression without compromising on quality.
Google Web-P image format uses lossless, and lossy compression, Web-P images using lossless are 26% smaller as compared to PNG and images using lossy compression are up to 34% lower in size than JPEG.
See the difference below
Web-P – bulk image converter
Convert your images to Web-p format; the free tool converts bulk images to web-p format at once.
Use CDN to upload images
Upload your Web-p images on a content delivery network. Instead of using your web host.
A cloud CDN is used to accelerate the speed of your content.
Imagine serving your customers from multiple locations instead of using a single site.
Yes and that’s the difference. The image gives you a clear picture.
Why use CDN to upload images?
Using a CDN provides benefits like failover protection, speed acceleration, reliability. CDN surpasses the traditional hosting & maintenance of digital assets.
Free and paid both options are available. However, the primary difference comes with the fact of how the CDN is handling and delivering the digital asset. E.g. most of the paid ones with an option of a free basic plan provides intelligent transformation of images as per device requirements.
AWS by Amazon is free up to 50 GB data transfer. However, the transformation service is unavailable, and the basic plan of Upload Care starts free and includes the intelligent transformation of images. The basic idea of Cloud Flare is also open and doesn’t support image transformation. Image engine begins at $100 and includes image optimisation.
Image Transformation in CDN
Image optimisation in CDN refers to intelligent resizing or transformation of images as per device requirements.
A CDN is right. However, you may not utilise the full potential if you are missing automatic image optimisation.
Make sure to choose the CDN carefully to get the best of both worlds.
Significance of anchor text in a hyperlink
Anchor describes the target page not only for the users but also for the search engines. Users can navigate well if link anchors are placed sensibly throughout the website pages.
A clear anchor text with URL passes link juice to the target page, during navigation when visitors encounter anchor text, they either click immediately or later on depending upon the relevance.
Things to avoid while placing anchor text
Avoid placing an anchor text for the sake of passing link juice only, users typically understand the behaviour and lose trust by considering them as spammy, and they may likely avoid the anchor or in worst case close the browser.
Avoid placing too many anchors, which again describes the spammy criteria in a given page.
Avoid deceiving users by placing a wrong target link in an anchor; the practice is unethical.
Anchor text best practices
- A well placed anchor text in line with the given content
- A highly relevant target page
- Includes keywords logically
- In a given page consider placing not more than one anchor for the same target
Hyperlinks are of 3 types
- Inbound links, coming to your site
- Outbound links, going out from your site
- Internal links, connecting within your site
Earning an inbound link is challenging and essential for search engine optimisation process. A high-level reputation is necessary to win a quality link, make efforts to make the fame.
Then give a link to someone you trust, the relationship is given and takes, who will link you back if you do not reciprocate. Although it must be done carefully, keeping in mind the relevance and trust of the reference you are using.
Make your webpages accessible with active internal linking site-wide. You can increase user retention rate by doing so. Furthermore, search bot loves crawling and index your webpage better.
More than 80% of the website serving the World Wide Web are two or three tiers. Means the link structure is just two or three levels deep.
A high rise building accommodates more and costs higher, likewise more deep the internal link structure the better chances of indexing in search engines, alongside more traffic converts the higher value.
As per Neil Patel
Internal linking is one of the SEO’s most valuable weapons.
Why? Because it works
There are no definitive criteria for link structure in the website. Few, e.g. are
3 tier link structure
2 tier link structure
Google bots always look for fresh and updated content, you can increase your chances of indexing and crawl by publishing content regularly, and that also increases the site structure and depth, which helps you multiply traffic and acquire more customers.
The image shows the crawling criteria
Typically the home page is the most linked page as compared to subsequent lower level pages.
The upside down pyramid shows the home page keeps the highest link earning potential. And that’s only possible due to the presence of all the lower pages. Which finally makes the value and the sheer presence of it.
All about sitemaps- SEO Guide 2019
A sitemap is a file that consists of all the URLs of a given website. With the help of a sitemap, search engines can find and index all of your website pages quickly.
Search engine bots crawl the link structure to index website pages. They crawl all the connected links to reach the last one.
A sitemap gives search engines access to crawl the full site structure of any depth using just one file.
Search engine bot crawls which one of them faster?
In the image below, the map on the left is the website navigation, in which search bot will crawl four links to reach the European page,
Whereas search bots can index all the webpages by visiting the XML map on the right.
The difference is enormous, and when it comes to crawling billions of webpage regularly to maintain the fresh index, crawling time becomes crucial.
Importance of a sitemap reflects with websites containing thousands of page when sitemaps assist search bots in accessing the link directory in single file.
A sitemap is especially helpful
- For websites containing a large number of pages and a deep link structure
- Sites doing frequent updates by adding fresh content
- Sites with a weak internal linking structure
- Sites lack external links
Including a page in a sitemap cannot guarantee successful indexing. Web pages are crawled and indexed even if they do not exist in a sitemap.
XML sitemap tags,
Location & Last modified, both the tags work brilliantly. However, people often ignore them.
Location tag is compulsory and keeps the canonical version of URL’s, with or without ‘www’, the label should also include the exact HTTP or https version.
Last mod tag
Last mod tag is optional although highly recommended as it tells the search bots when you have last updated your website pages.
John Muller acknowledges in a post that search bots refer to last modification date.
However, manipulating the time without updating content is unethical for which Google may penalise your website.
Google recommends using a maximum of 50 MB file size and up to 50,000 URLs in a single sitemap.
However, if the link structure is beyond 50k, it is recommended to divide and create multiple sitemaps, e.g. for a large e-commerce retailer, categories like Apparel, Home & Garden, Kids, etc.
You can optionally create a sitemap index file (a file pointing to a list of sitemaps) and submit to Google.
Google accepts the sitemap syntax of video, images, news.
Video sitemap is an excellent way to invite Google to find and crawl new & existing videos.
Basic guidelines for a video sitemap
- You can create and publish a separate sitemap for videos or may also include your video sitemap in your existing one.
- Avoid publishing videos not related to the main page
- Make optimum use of all the available Metadata for videos; important ones are, URL, video location, title, and description, content, duration, rating, publish date.
- Google will ignore the sitemap entry If it fails to discover video content
- Make sure to avoid blocking the source file or the video player from accessing Google bot.
- Recheck the robots file and optimise it to allow Google to crawl the page, video and thumbnail URL’s
- Google verifies the video content by matching it with the given Metadata and the page content; Google will not index if it doesn’t match.
Example video sitemap with tags
Example showing a webpage hosting a video with all the available cards Google uses.
How Google crawls a video?
Google returns a highly relevant video search when it understands the video/audio content, along with the Metadata and page text.
Google crawls the video if in a supported video encoding format and extracts the audio and video content in a limited capacity.
Google also uses the structured data, VideoObject tag
Best practices to publish videos
You can make your videos eligible for search results only if you follow the best practices. They are
- Google must be able to find your video, embed your videos in HTML using tags video , embed or object
- Provide a high-quality thumbnail for the video
- Include a video sitemap for an explicit declaration of a video, and try to make the best use of all given tags.
- Ensure the relevance of your content with the video you are publishing. E.g. a video showing the recipe of a fried chicken and the page describes the rice chicken.
- Make use of structured data VideoObject.
XML image sitemap
Images are essential for webpages; a dedicated XML map helps in faster indexation.
However, modern-day SEO approach suggests using structured data image object JSON-LD schema.org/ImageObject mark up, as it well defines image properties requires during search engine callouts.
The exception to the rule applies when the site is selling images like a stock photo website, or an e-commerce portal which includes thousands of copies.
Save your crawling budget as far as you can.
Google News Sitemap
You need to register your site with Google news publisher centre before using this sitemap, publish your most recent articles up to two days old. However, they remain in the Google index for up to 30 days.
Use the Google search console to submit.
Tags using this sitemap are, publication, date, title
The purpose of an HTML sitemap is to give access to all your site pages to your visitors, and that’s the only relevance to it.
They generally appear in website footers as a tertiary part. However, users rarely click and go through your HTML sitemap if your site internal linking and navigation are acceptable.
Search engine bots don’t bother even if it’s present or not. So if users are not clicking and you still keeping for the sake of providing it on the website, only opens the door of keeping stale information.
A better option is to avoid publishing.
Dynamic XML sitemap
Generating a sitemap as soon as you publish fresh content is cumbersome unless you love doing so.
Dynamic XML sitemaps are automatically updated when changes occur. You can use a Yoast SEO plugin to dynamically render an XML sitemap if you are using a WordPress CMS.
XML sitemap optimisation
Make sure to include only those pages in XML sitemap which you want to be index with Google.Your website may include a thousand pages; not all of them require indexing. E.g. canonical, duplicate, site search pages, archived, comments reply etc.
This way, you can save your crawl budget as well.
From SEO perspective Include only relevant pages in the XML sitemap, although other pages allow crawl and not blocked either.
Sitemap verification & reporting.
Excluded and valid both URL’s are available in three different categories Use the Google search console to manually submit the sitemap, its verification and a detailed reporting using coverage in GSC.
- Indexed but not submitted in sitemap
- Indexed and submitted successfully
- Submitted URL is marked no-index
An optimising sitemap helps understand the more important pages in your website, using no-index and no-inclusion in sitemap enables you to avoid appearing your website on irrelevant searches and clicks.
In the above screen, by including the less critical 841 pages, we are only telling Google to give precedence to the valid ones.
Otherwise, if the site could have included all the 1000 pages in sitemap, the relevancy of the valid pages serving only 20% appears to be much lower as compared to the relevancy of the excluded pages serving almost 80% of the capacity.
Google Analytics- SEO Guide 2019
Google Analytics for SEO
A free and brilliant software by Google, analytics gives you an in-depth report of your audience, their behaviour, conversions & acquisition.
Meaningful data backed with excellent representation, Google knew the fact from the user perspective and prepared the exhaustive metric tool.
We are covering the overview of different sections of Google analytics and their significance. Moz has brilliantly presented a beginners guide to Google analytics.
Google Analytics journey is pretty straight forward
The dashboard will show you the report which you may not have encountered before, precise detail related to your website, which is not just limited to users, revenue, conversions etc. but traffic sources, referrals, channels, location, time, devices, top pages, and much more.
With Google Analytics, you can answer questions related to your website.
- Numbers of visitors visiting my website daily/weekly/monthly/yearly?
- Users are visiting from which location? And how many numbers of areas?
- How much time users are staying on my website? And what is the monthly average?
- Who is bouncing from my website means users who didn’t like
- And exactly which page they didn’t like?
- Am I getting old or new visitors and their count?
- What time of day do I receive most hits in a week?
- Who else is talking about my website on the internet?
- How many referrals are they sending to my website?
- Am I popular in my business?
- Is worth to spend time with social channels?
- What is the age & gender of my visitor?
- Do I appear in Google and from what keywords?
- How my website is performing in search engines?
- Can I track the activity of my visitor?
- How can I track conversion?
Much more questions Google Analytics can answer for you Analytics maintains the historical data of your digital asset, which means you can easily compare data within specified periods. Google Analytics dashboard showing three different types of metrics.
Analytics is into five major sections and their respective subsections.
Audience overview in Google Analytics will open the door to the insightful view showing users, new users, sessions, page views, Avg. Session duration, bounce rate etc.
You can share, save, and export the reports, understanding the metrics will give you the real meaning behind the figures
Users are the visitors coming to your website; they include new and returning both. However new users are only those coming to your site for the first time,
Sessions refer to the activity of the user from the start page till the exit page, whatever user does in between counts as a session, inactivity of 30 minutes ends the active session.
A single user can view as many numbers of pages; page views refer to the total number of pages by all the users.
The time spent by users on your website and the average of it are Avg. Session duration.
Bounce rate is when a user jumps off or exit from your website either by closing the browser or to a different website. Analytics will take you to step further and will show you more relevant sections. They are
As the headings suggest, these sections are straight forward and easy to understand.
Acquisitions in Google Analytics
The sections elaborate from where you are acquiring, mainly channels,
Visitors are coming to your website by directly typing your website URL as a result of your other marketing sources. Direct traffic is higher in big brands due to their popularity. We call it direct traffic as the source is indirect and not from any given digital source
What is organic traffic in Google Analytics?
The traffic you are acquiring as a result of a natural appearance in Google is organic.
Organic Rankings Google
You can only win natural appearance with SEO. And that’s the only channel. Which is considered best in terms of ROI; competition is fierce for organic rankings on each keyword, and you need to do precisely all the SEO detailing to achieve the results.
Apart from doing SEO, There is no other technique to earn organic traffic for your website.
Referral traffic in SEO
If someone refers you to visit is called a referral, earning a reference is challenging, and Google gives weight to links acquired from a reference.
E.g. site A is referring site D for excellent service or a product. Google understands a paid or earned referral. Do not underestimate Google in that sense. Hi-quality referral links are essential for a digital asset to rank higher in SERP (search engine results page).
Promoting your website socially and the traffic acquired is the accumulation from the common source. Few accessible social sources are Facebook, Twitter, LinkedIn, Instagram, and Pinterest etc.
Social promotion is quite challenging, as it involves the science of understanding human behaviour.
You can see direct and organic are two significant sources of traffic when you first hit and start building your audience through digital marketing.
Referral and social follow the duo as they significantly depend on your popularity
The section includes numerous other tabs related to the four primary channels, e.g. landing pages & their impressions, user flow etc.
Behaviour section – Google Analytics
Behaviour is all about the response of your digital asset from a performance perspective. How your website is responding is essential, and the metric includes
Exhaustive details about your site content, like a most served landing page and the avg. Time on page, pages which are giving the most bounce rate, the slowest pages in your website. These are essential during website site audits and performance report.
Load time effects performance, site speed section gives you insightful data using which can help you to understand the blockages.
Conversions – Google Analytics
As the name suggests, the tab deals with goals completions, goal value, and goal conversion rate. You need to create the goals first before you see conversions.
- Fill a form can be the goal,
- Make a booking or purchase is a goal
- Download an e-book is a goal
- Staying on the website is a goal that increases Time on site.
You can set an individual goal to each booking stage, e.g. a hotel converts booking in two or three steps, Room selection > billing details & payment
The system also provides a full section for E-commerce to check transactions, revenue, order value etc. interestingly it also recognises coupon codes you offer and provide details attached to it during conversion.
You can see individual sources in goal flow
Robots.txt setup & optimisation- SEO Guide 2019
As the name suggests, robots.txt file controls how search engine bots crawl your website, placing a robots.txt file allow, disallow & partially allow search bots from crawling.
And by optimising the robots.txt file, we explicitly declare which files and directories to crawl & index by search-bots.
Robots.txt file is a part of robots exclusion protocol REP, web standards to regulate how search engines crawl & index the content on the web and present the information to the users.
REP also includes the directives for meta-robots, directories, sub-directories and site-wide instructions on how search engines should index links, either follow or no-follow.
Robots.txt configuration SEO
Practically, robots.txt file handles the software program (user-agent) on how to crawl and index the website or parts of the website.
A necessary robot.txt file looks like
User-agent: [user-agent name]
Disallow: [Declare URL’s to exclude from crawl]
Example 1 is telling all user-agents not to crawl any directory, sub-directory or web page.
Example 2 is reverse and telling all the user-agents to crawl all directories, sub-directory and web pages
You can see the massive difference in both the above examples by just placing a forward slash the entire website is blocked for user-agents.
It is imperative for webmasters to handle the robots.txt carefully
To further customise,
You can see we can declare explicitly what to crawl and what not on our website.
Googlebot is one of the most popular one, User-agent: Googlebot Other user-agents of Google
|Google Mobile (featured phone)||SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/22.214.171.124.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)|
|Google Smartphone||Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)|
|Google Mobile Adsense||(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)|
|Google AdsBot (PPC landing page quality)||AdsBot-Google (+http://www.google.com/adsbot.html)|
|Google app crawler (fetch resources for mobile)||AdsBot-Google-Mobile-Apps|
Data source: KeyCDN
Google fetch tool – Search console
Use fetch tool in Google search console and test how Google bot crawls and renders a URL.
Google will accurately tell you how your web page is visible to search spiders, what is visible to them and what not. That includes website images, text, scripts.
The coverage report in GSC will also show you the last crawl, agent and indexing
Bing bot, the user agent of bing search
Yahoo search agent
Slurp is the user agent of yahoo, user-agent: slurp
And likewise, DuckDuckGo, Baidu, Yandex, they all keep user-agents to crawl the web,
Baidu works in China, and also have additional web crawlers for images, videos, news, search, wish list.
Alexa from Amazon
Amazon uses Alexa web crawl for internet rankings.
You can also specifically allow / disallow different user agents
In the above example, you have entirely blocked slurp, a yahoo search agent to crawl & index
Likewise, below example instructs the bingbot, the user agent of Bing, to wait for five milliseconds before crawling each page.
You can note that crawl-delay won’t work with Google-bot and this is by default. The most popular bot is maintaining its monopolies.
Robots.txt file working
Search engine crawlers execute two critical functions
- To crawl the webpages by following links, hence we also call them web spiders
- Index each link they encounter in their database and feed the search engine for those looking for information via search.
Before performing a crawl, by default search engine crawlers looks for a robots.txt file to check for permissions & directives.
And if they didn’t find a robots.txt file, they will crawl the whole website including directories, sub-directories and individual webpages.
It is imperative the robots.txt file should be present in a root directory, and with name ‘robots.txt’, all small case, no sentence or upper case is allowed.
See the image below
Facts to know about the robots.txt
- The file is publicly accessible, make sure to avoid putting sensitive information, yourdomain.com/robots.txt
- txt is different for each sub-domain and root domain. Hence blow.example.com will keep its robots.txt and will not share the robots.txt you have mentioned for the root domain. Example.com
- Robots file is case sensitive as discussed above
- Malicious crawlers bypass robots file by ignoring them completely, e.g. malware robots
The syntax used in robots.txt
User-agent: generally a crawl bot, using the asterisk (*) means all user agents
Disallow: restrict user-agents to crawl, using forward slash (/) means the entire site is disallowed from the crawling
Allow: the command is only applicable to Googlebot, which means Googlebot can access the file, directory or a sub-directory, even though the parent directory is disallowed.
Crawl-delay: the command does not apply to Googlebot. The command prompts the user-agent to crawl after a specific time delay, e.g. four milliseconds
Sitemap: the command is used to declare the location of a sitemap.
To maintain the privacy of website owners, sometimes they don’t want file or directories to be indexed and made public.
Maintain your privacy using robots.txt
Say directory containing your pictures in your blog, sensitive documents in your website you don’t want them to appear in search engine results, CMS directories and files keeps no relevance for search engines, e.g. wp-admin
Webmasters now control which part of the website they allow crawling and in which manner. They can specifically block a particular from entirely form crawling.
The method gives incredible power to webmasters to control the display of information and maintain their privacy.
SEO best practices for robots.txt
- Ensure that you aren’t mistakenly blocked user-agents of Google, Yahoo, Bing. They are considered primary in search.
- Always optimise robots.txt file by explicitly declaring what exactly you want search engines to crawl and index
- Use a dynamic robots.txt, to avoid frequently updating the data with fresh content. If you are using WordPress CMS, All in one SEO or Yoast plugin offers the functionality.
- Use Google search console to check for a successful submission; coverage report will accurately show you the links crawled and indexed.
Webmasters Setup (Google Search Console)
SEO Guide 2019
Google search console empowers webmasters to check and fix the errors & shortcomings their digital assets are facing through a comprehensive set of powerful tools.
Formally webmasters tools, GSC is famous for its accuracy, the tool is quite popular and one of the favourites among webmasters community.
Reports are insightful with in-depth analysis criteria and easy to understand.
Google has designed it, keeping in mind the users’ perspective.
Sections are divided carefully to cover the website from all the necessary angles required for the user
Coming on board is simple; a sign up is all you need along with the verification of ownership.
You can add multiple properties in a single GSC account and must claim the ownership of each one with a verification process.
The performance report helps analyse the traffic, search queries, the most and least searched pages and from which location, on which devices.
URL inspection tool
You can now inspect individual pages of your website by inspecting URL, the analysis includes crawl errors, testing live URL, request indexing,
Find crawl errors, test live URL’s, and check how Google bot sees your web page
Then coverage report will show you statistics related to indexing, crawling and discovery,
Furthermore, enhancements for mobile, search box and logo. You will find if the page is mobile friendly and even view the crawled page along with the exact date of crawl.
CoverageThe report will show you the detailed analysis of pages on your website, which includes
- Pages you have excluded from crawling
- Total number of valid pages successfully crawled
- Number of pages indexed with issues
- Pages not indexed for some reason
Check indexing & crawling issues
The report is showing valuable data which webmasters can use as key factors to analyse their digital assets. E.g. the above screen shows the website includes more than 1100 URL’s and out of which 841 excludes due to their NON-relevancy with the content of the website, these are PDF’s, admin pages, supporting images, etc.
You can see non-relevant pages are way too high then the relevant ones, imagine the trust of your website with search engines if they could have crawled all of them.
Perhaps it may likely the website may start appearing with all the wrong keywords not related to your business due to the high-ratio of all the non-relevant pages.
The value of even the relevant content can diminish in the scenario. And the bounces due to non-relevancy tells search engines the asset is non-performing on critical phrases and drop rankings.
You can add sitemaps to your website in this section. Submitting a sitemap is an excellent way to declare the priority and importance of website pages for search engines.
Although using dynamic sitemap is recommended if content publishing is frequent.
The system will show the last submitted sitemap along with its status and discovered URL’s
Sitemaps for search engines are always submitted in XML format, which you can view any time. E.g. example.com/sitemap.xml
Search engines still crawl the website even though the sitemap is missing; however, best practice is to submit the sitemap for better crawling and indexing of your website pages.
Mobile First Indexing- SEO Guide 2019
Google first announced in past July 2018 that page speed is one of the ranking signals for mobile searches,
Google has already published on 28th May 2019, that starting 1st July 2019, Google search is moving to mobile-first-indexing.
Now the mobile version of the website will play a significant role in determining the rankings.
Google was testing mobile first indexing when they first announced it on 4 Nov 2016
Mobile first indexing means Google will crawl the web using a ‘Smartphone Google Bot’ by default.
You can see Google Smartphone bot working in action using the URL inspection tools in search console.
If the website pages are mobile friendly, it remains unaffected.
By default, the Smartphone bot is active for all the new and previously unknown website.
Google says a single website for desktop & mobile
As per John Muller, Google recommends keeping a single version of your website for both desktop and mobile, basically for three strong reasons.
- Information becomes stale if previous versions are not updating.
- It creates confusion among users to use which version as sometimes both the versions are different by interface and functionality.
- Confusing to search engines as well during indexing.
Now with Mobile first indexing, the confusion of keeping a different sub-domain for a mobile website is quite clear.
Webmaster and SEO community faced controversy in past as the scenario was not precise.
Google will keep sending a notification to that website still not mobile friendly; you will find them in GSC.
How Google determines mobile first indexing
According to John Muller, Google determines the mobile first indexing based on content parity, which includes images, text, videos, links, also how the site is using structured data & Metadata as well.
You should double check all the factors if you are about to launch or planning a website redesign.
The web has evolved from being desktop-centric to mobile indexing. Google will keep monitoring the performance of its new user-agents.
Can you learn more about mobile indexing and speed score in our recently published article Website redesign? 95% of websites suffering speed score
Structured data for SEO- SEO Guide 2019
The code to achieve the criteria of a meaningful web forms the basis of structured data and Symantec web provides the foundation of it.
Logic is simple that ‘Machines works with strings, not things.’
Symantec web is the cumulative effort of Google, Yahoo, Bing, and Yandex
And machines don’t know which apple we are talking? And in what context and relation?Symantec deals with the study of the relationship of words, symbols, phrases. E.g. we all know that an apple is a fruit. However these days you can refer to apple as machines too.
Likewise, Mount Everest is the highest peak known to man; the two words bear’s special significance in the dictionary and keeps a relationship of one of nature’s symbol.
We train machines to understand the relationships between words, symbols and phrases.
Symantec web is the combined effort of Google, Bing, Yahoo, and Yandex.
Furthermore, structured data is the markup for machines to understand the human language, which includes its vocabulary and grammar.
E.g. The sentence
Eiffel tower is in Paris
We very well know one of the seven wonders and the structure is in a place name Paris. On the contrary.
However, with structured data, a machine only considers 24 characters in memory, including spaces and returns the result only when the exact phrase appears.
Search bot without structured data vocabulary?
We cannot even imagine how the search will look like if considered in that sense.
Search companies realised the fact in early days and knew since starting they require a language to train machines.
Thanks to structured data which helps the machines to understand the human world.
After applying structured data, the same machine is capable of understanding that:
Eiffel tower is a structure, made of iron and located in a place called Paris, which is a part of Europe. And one of the Seven Wonders of the World.
You can now see the difference and the power of structured data,
What exactly is structured data?
Structured data is the mark-up to help machines understands the meaning of a given language; it’s divided into two parts.
And typically we call structured data as the vocabulary with the specific and established rule set.
Google defines structured data
Google Search works hard to understand the content of a page. You can help us by providing explicit clues about the meaning of a page to Google by including structured data on the page.
Structured data is a standardized format for providing information about a page and classifying the page content; for example, on a recipe page, what are the ingredients, the cooking time and temperature, the calories, and so on
What is a schema, Microdata, Microformats, RDFa?
Our digital gurus are not unanimous, which results in emanating of different vocabularies & grammar, and thus different formats are born.
In a most basic sense, information is conveyed through a language, whether machine or human, in both the languages two things are primary
- Vocabulary (set of words)
- Grammar (set of rules to use the vocabulary)
Example of vocabulary for structured data markup for SEO.
Keeping in mind the above, we can classify digital jargons as follows.
Microformats is the only one to provide both vocabulary & grammar.
To understand further you can go through our detailed article on structured data for SEO & winning Rich Snippets
Google is on its way of evolving machines, building them to understand the human needs & behaviour, computers in the form of voice search are booking tables for dining, ordering food for refrigerators. Playing the favourite music, driving cars, and much more.
Search is still the most loved tool and widely known from the long list of products & services Google provides.
And the direct impact of these update influences searches optimisation and its core elements.
Above are all the core elements of SEO, and they are critical for any subsequent digital marketing campaign.
Accuracy of SEO tools is a question as they fail to determine important aspects of SEO, like content quality, the relevancy of Meta titles, link anchor etc.
However, Ahrefs audit tool provides a comprehensive report, the latest and my favourite as well.
The best practice is to use the SEO tool along with opting a manual SEO audit process to ensure the health of your digital assets.
Google search console is indispensable in finding errors in your website, which no other tool can. Consider installing that by default.
Any effort in search engine marketing will prove to be worthless if website pages are missing the critical elements of the search engine optimisation process.
So you need to be extremely careful in executing SEO & SMM with your digital assets.
Ahrefs SEO audit tool
Sales process optimisation
The optimisation process is critical for your digital assets to perform. However, other sales optimisation metrics are also available.
Google says, chose SEO carefully.
Deciding to hire an SEO is a big decision that can potentially improve your website ranking or may damage its reputation as well.
Online presence analysis
A manual audit report is beneficial for professional SEO analysis.
Companies proactively maintain their brand, apt for online reputation management analysis for brand monitoring and health. The service is specially designed for those companies aggressively making online sales.
The process is beneficial in finding the fault of an affected brand reputation.
A detailed report showing metrics of non-performing assets & user behaviour with reasoning with a possible solution.
The conversion rate optimisation is again a brilliant sales metric to see if your digital assets are not working.
Non-performing assets can be a significant hurdle for online sales; NPA’s can seriously damage your website reputation by increasing the bounce rate.
The process signifies quantitative and qualitative both the methods and is reasonably simple
I want to reiterate, consider the web-pages as your digital assets, use your intellect to find out why they are non-performing.
Publishing webpage for the sake of it is entirely different than considering them as your digital asset. And if you want your digital asset to perform entirely depends, upon your marketing approach.
A wrong approach will end up making your investment fragile, whereas you are ready to reap fruits with the correct one. The choice is yours. So be careful with digital marketing.
And that’s what Google says "Chose SEO carefully, says Google"
“Deciding to hire an SEO is a big decision that can potentially improve your site and save time, but you can also risk damage to your website and reputation.
Make sure to research the potential advantages as well as the damage that an irresponsible SEO can do to your site.”
More stuff for you,
Over to you
If you have done the reading, check your website pages, fix the errors and you will see them gradually improving in searches over time.
Don’t forget to chime if you may find the content fascinating. I would love to hear your comments & feed.