Research On The Web – Finding The Right Sources And The Right Tools

There are two basic things that I guess any researcher will always need to get in order before s/he can be effective – the right *sources*, and the right *tools*.

Finding the right sources – reliable and trustworthy ones – is not always straightforward on the web. The usual indicators of reliability that we find in print and other media are not always there, or when they are there, they can be misleading. As for finding the right tools, the “traditional” search engines that many web-researchers have come to rely upon are becoming less and less effective (according to a recent estimate, they now cover only around 30% of web-based content).

Here are a few basic tips that you may find useful in guiding your quest for the right tools and the right sources on the web (assuming, of course, that you have not already sorted this particular challenge out for yourself).

Finding The Right Sources

It is easy to confuse form and content anywhere, but even more so on the web. With WYSIWYG html editors freely available now, anyone can easily make their content look highly professional. It’s cheap and easy to publish and the entry barriers are very low. This means that it can be very easy to be deceived by a professional-looking site into believing that the content and information is just as professionally produced.

So, here are some questions you may well ask yourself when visiting a website for research purposes:

  • Is the author an authority on the topic?
  • Does he/she have the experience or educational background needed
  • Is the author involved in research?  If so, which institutions is he/she associated with?  Are they reputable ones?

And here are some suggestions on how you could go about answering some of the above questions:

  • Check for an email address on the site, and email the author to find out more.
  • Check for hyperlinks on the site which lead to more information about the author.
  • Check the site itself, which can present some cues as to quality. For instance, is the site  hosted on a ‘free server’ or does it have its own domain name? If the site is clearly hosted at no cost, then the maxim “You get what you pay for” MAY apply. Check for the ‘last updated on’ date on the site. From this (if it is present, of course) you can get an indication of how well-maintained the site is.
  • Note the author’s email address, which can tell you a lot.  For instance, the author of an article or posting on the topic of the latest version of Java is probably more likely to have authority if his email address is john@java.com than if it is john@xtra.co.nz.
  • Visit the website that goes with the author’s email address. (For those who are not very au fait with the way the web works: the usual procedure is to add a ‘www’ prefix to the top-level domain of the email address – eg for john@java.com, the web address would be www.java.com.)
  • Search the internet under the author’s name to see where else s/he has been posting. You can search the web, the newsgroups, and discussion forums. Newsgroups and discussion forums particularly can yield a lot of useful information. For instance, does the author post on the topic mainly to hobby groups or to professional groups and discussion forums?  What is the general ‘tenor’ of the groups to which this author regularly posts? For instance, you find an article on the web titled: “Scientific research says astrology is true”, written by John Smith. If you searched the newsgroups under John Smith’s name, and found postings to newsgroups such as alt.crystals, alt.ufo, alt.tarot, alt.astrology, alt.palmistry, you would probably draw different conclusions about this author than you would if your search told you he regularly posted to newsgroups frequented by scientists and academics.

Finding The Right Tools

A constant problem on the internet is information overload. Another problem for researchers is the ‘invisible web’ – the vast amount of data which does not appear on the traditional search engines. On one recent estimate, only around 30% of websites appear on the search engines; and, in addition to this, the vast amount of information stored in databases on the web is not ‘crawled’ by most search engines.

If the traditional search engines are not serving you well, why not try some of the newer ones? One that has received good reviews is AllTheWeb.

“Bots” are also becoming increasingly popular. These are automated search agents that can do some of the necessary sifting and filtering work for you.

An excellent place to find information about bots on the web is BotSpot. It provides links to around 20 different categories of bots – including “news-bots” (which search news sites), government bots (which search government sources); “update bots” (which track selected websites for updates).

A couple of the more well-known and worthwhile commercial search bots are “Bullseye” and “Copernic’.

Bullseye works with many search engines and databases, and (based on your query) can create customised results which draw on data from around 300 search engines and 600 databases on both the “visible” and “invisible” web. It can also refine the search process – for instance, by removing duplicates and unrelated material.

Copernic is a “meta” search engine, which searches other major search engines, and allows focused searching according to subjects which the searcher defines.

Happy researching!

By Nicole Bishop