posted by randfish

let's try a little excercise...

common features of spam domains include:

• long domain names
• .info, .cc, .us and other cheap, easy to grab tlds
• short registration period (1 year, maybe 2)
• high ratio of ad blocks to content
• javascript redirects from initial landing pages
• use of common, high-commercial value spam keywords like "mortgage," "poker," "texas hold 'em," "porn," "student credit cards," and related terms
• many links to other low quality, spam sites
• few links to high quality, trusted sites
• high keyword frequencies and keyword densities
• small amounts of unique content
• very few direct visits
• very few links sent out in (non-spam) email to the site
• registered to people/entities not associated with trusted sites
• not frequently registered with services like yahoo! site explorer, google webmaster central or live webmaster tools
• rarely have short, high value domain names
• often contain many keyword-stuffed subdomains
• more likely to have longer domain names
• more likely to contain multiple hyphens in the domain name
• less likely to have links from trusted sources
• less likely to have ssl security certificates
• less likely to be in directories like dmoz, yahoo!, librarian's internet index, etc.
• unlikely to have any significant quantity of branded searches
• unlikely to be bookmarked in services like my yahoo!, del.icio.us, faves.com, etc.
• unilkely to get featured in social voting sites like digg, reddit, yahoo! buzz, stumbleupon,  etc.
• unlikely to have channels on youtube, communities on facebook or links from wikipedia
• unlikely to be mentioned on major news sites (either with or without link attribution)
• unlikely to register with google/yahoo!/msn local services
• unlikely to have a legitimate physical address/phone number on the website 
• likely to have the domain associated with emails on blacklists
• often contain a large number of snippets of "duplicate" content found elsewhere on the web
• unlikely to contain unique content in the form of pdfs, ppts, xlss, docs, etc.
• frequently feature commercially focused content
• many levels of links away from highly trusted websites
• rarely contain privacy policy and copyright notice pages
• rarely listed in better business bureau's online directory
• rarely contains high grade level text content (as measured by metrics like fleisch-kincaid reading level)
• rarely have small snippets of text quoted on other websites and pages
• cloaking based on user-agent or ip address is common
• rarely contain paid analytics tracking software
• rarely have online or offline marketing campaigns
• rarely have affilliate link programs pointing to them
• less likely to have .com or .org extensions
• almost never have .mil, .edu or .gov extensions
• rarely have links from domains with .edu or .gov extensions
• almost never have links from domains with .mil extensions
• rarely receive high quantities of monthly visits
• rarely have visits lasting longer than 30 seconds
• rarely have visitors bookmarking their domains in the browser
• unlikely to buy significant quantities of ppc ad traffic
• rarely have banner ad media buys
• likely to have links to a significant portion of the sites and pages that link to them
• extremely unlikely to be mentioned or linked-to in scientific research papers
• unlikely to use expensive web technologies (microsoft server & coding products that require a licensing fee)
• likely to be registered by parties who own a very large number of domains
• unlikely to attract significant return traffic
• more likely to contain malware, viruses or spyware (or any automated downloads)

for high quality content domains, the opposite is true (at least, for a good percentage of these). now think about the sites you're building - which features apply to them? what could you do differently to be more like the "high quality" category and less like the "spam"?

btw - love to hear your take on features you think are common to spam, or to high quality sites.

do you like this post? yes no

posted by randfish

i've talked in the past about the various possibilities of the future of search engines and the biggest threats to google's dominance, and today i've been able to follow up on some of that prodding into that "vertical search fracturing" possibility.

imedix is one of the more exciting vertical search engines on the market. not only are they a recipient of the best new startup award from the crunchies, they're also very highly regarded in the marketplace by a lot of smart players. not everyone's a fan, but they're in an area of high opportunity and have executed remarkably well from both a development and marketing perspective.
_

_

the biggest "difference" that imedix offers is the inclusion of a social network that integrates with the search results. above, you can see i searched for allergies, and along the sidebar are people who share that interest and are interested and willing to talk about it. this takes advantage of some big psychological strengths in the health field - namely the need to "not be alone" when dealing with health issues. i'd imagine that imedix's community is held together by stronger bonds than most, and that, potentially, gives them a powerful edge.

last week, i got the opportunity to interview iri arimav and amir leitersdorf frrom imedix (cmo & ceo, respectively) about they're progress to date and some of their search technology issues:

imedix started because of a personal health need for both of us. since we both love the web, we saw it as an untapped resource for people looking to make better health related decisions. we felt that searching alone using the popular horizontal search engines was definitely not the right way to find and share health information. not only were the results irrelevant in most cases, they also left us unaided and with a strong sense of anxiety. we knew that there was excellent information online, but it was organized poorly. at the same time, it was difficult for us to communicate easily and quickly with other people that had valuable personal health experiences and knowledge. we decided to build a health search engine that will be powered by the patients for the benefit of patients. we built a prototype that improved into an alpha version that upgraded to what you see today on imedix.

from a vertical search perspective, do you see an ability to pull search traffic mindshare away from google/yahoo!/live? do you think people will ever come on the web and think - "i need to search for health/medical issues, so i'll use imedix, not google."? or is your strategy more like webmd - to build a portal and attract search traffic from the major engines?

we believe that the experience of finding and sharing health information on the web is about to change dramatically. an exceptionally different and valuable experience can pull traffic mindshare away from traditional web properties. the experience imedix offers to consumers is very different than google/yahoo or live. people nowadays are attracted to communicating with each other and engaging in various ways. people want more than search and browse. the growth of social networks that took substantial traffic from traditional portals in the last 3 years is a good example. we are experimenting with an innovative marketing approach now and it looks very promising. we are continuously introducing new features that allow for the creation of valuable user generated health content and most importantly listen to the needs of our fast-growing community. we believe that if you create value and a worthy experience, the rest will follow.

it appears that imedix crawls and indexes websites, just like the major search engines, but rather than indexing the entire web, you've limited your search to only domains pertaining to health. a few questions on that - do you hand select the domains to be included? if so, do you worry that the information is more narrow, or do you think searchers will feel confident knowing the results include more authoritative sources and fewer blogs, scrapers, etc.?

imedix crawls a subset of the web to index the most informative and relevant health articles available. using our patented technology we crawl and index numerous web directories and medical databases resulting in almost any known medical site in the english language. this way, we are able to cover all of the symptoms, conditions and treatments known to medicine.

in terms of building a search engine crawler, did you custom develop something yourself or use a technology like nutch? does imedix maintain all those inverted-keyword databases or do you use third party technology? and, how many pages (approx.) do you have in your index?

our crawler is a multi threaded, distributed computing technology that was developed in-house and is capable of crawling hundreds of web sites in parallel. our crawler relies on a high bandwidth network and while using a single database maintains an average rate of crawling half a million pages per hour without burdening on the crawled site's performance.

the crawler detects similarity between pages thus avoiding "over crawling" and is also capable of detecting frequency of change in a site's content so to optimize the crawl scheduling and gain maximum data update for each crawl session.

the indexer was developed in-house, while relying on open source projects for representing documents in a space vector model, we developed a grid computed application based on divide-and-conquer algorithms that provide the ability to index hundreds of documents per second into binary files that can be instantly searched while the indexing is continued in the background.

the number of documents in our index varies a lot since we are constantly increasing the number of sites crawled, while deleting pages that were flagged as irrelevant both automatically (by heuristics of the crawler or indexer) and also by manual processing that is done by our staff. in our recent versions we have used indexes of 10-20 million health pages depending on the factors stated above.

as a follow-up with regards to the ranking algorithm - is it something you've done in-house? did you end up using a modified version of something like pagerank? trustrank?

our ip resides with our ranking algorithms that analyze the feedback received from our users in order to recognize patterns of useful pages. the ranking formula is constantly and automatically updated according to the users' feedback. the learning machine itself is built upon an ensemble of modern algorithms in the machine-learning field. the classifying algorithms are focused at bringing very high precision in predicting the probability of a page being a good match given a certain query. our proprietary technology is also developed with the assistance of our chief scientist, prof. yuval shahar who is the head of the medical informatics research center in ben gurion university and has more than 15 years of experience in the most advanced health information retrieval and artificial intelligence technologies. prof. shahar holds a ph.d. from stanford university in medical information sciences and is a certified medical doctor.

imedix, in addition to being a search engine, leverages users to help build a community - what made you choose that path and how have your users been responding?

we decided to choose this path because we felt that people want to be empowered and can contribute so much from their experience and knowledge. if we build the right tools we can organize and leverage this collaborative effort. although we regularly read forrester, jupiter research, emarketer and all the other major research companies we believe that listening to your users is the most important thing regarding building successful products. our path is really a reflection of our conversations with our users. we are fortunate that our users like to use imedix and that the internet community decided that we are worthy of winning the best new startup of 2007 at the crunchies worldwide competition.

imedix has obviously been a hit in its first year, winning the crunchies award for best new startup - with regard to publicizing the product, what have been your strategies to date and where have you seen the most success?

thank you very much for your kind words. we feel very fortunate to be in the right place at the right time. one of the strategies that worked well for us was to truly spend time learning and understanding the products that existed out there and get to know our audience. not just read reports, but engage in an honest and an open discussion with bloggers, opinion leaders and patients. we were in contact with hundreds of these people and we learned together with them and with the help of our users what we needed to develop. i believe these people enjoyed the process and decided to share it with their friends and readers. this had a tremendous effect on our visibility and traffic.

finally, with high expectations and a startup environment, i imagine things can get fairly overwhelming - how have you and the rest of the imedix team done with work/life balance? any recommendations you have for a startup - things you'd change or do differently the next time around?

the startup environment is indeed very demanding. the past year has been pretty tough for all of us. we’ve worked around the clock and had a hard time balancing our personal lives. eventually, we are all happy and satisfied but several lessons were learned along the way.

we would recommend the following to every start-up ceo/founders:

• have 1 day every week in which you do not check your emails.
• understand that people are the most important resource in a start-up company. make sure that they spend time with their families and encourage them to take vacations.
• keep your employees informed in the company’s progress and activities. if you expect everyone to share the effort then you definitely need to share the fruits.
• and finally.... celebrate! even if you don’t have the time. don’t let those wonderful moments pass you by.

thanks, iri & amir - i'm not entirely sure i can live up to all of those suggestions, but will certainly try. much appreciated!

btw - some interesting thoughts to consider for search marketers:

• how do you target content towards vertical search sites that could potentially be whitelisting domains?
• are you currently seeing any significant search traffic from vertical engines outside the major properties?
• is traffic from a site like imedix potentially more or less valuable/focused to site owners?

my biggest concern is that marketers who don't pay attention to potential future landscapes may miss out on big opportunities to leapfrog the competition when those changes do occur.

do you like this post? yes no

posted by sarah bird, esquire

do you like this post? yes no

posted by randfish

before i was inundated with the responsibilities of running a company and managing a few hundred emails a day, i used to spend a lot of time testing theories about how the search engines handled certain elements on a site or page. i'd test the engines to find answers to questions like:

• does a keyword perform better or worse if it's higher up in the code of a page? (better)
• what's better, bold or strong tags? (used to be strong, now they appear equal)
• does a link with exact anchor text for a query perform better than one that has other words in the anchor text? (exact appears to be better)

note: my tests on these are more than a year old, so things may have changed.

obviously, to test the answers to questions like these, you need a very tightly controlled environment, and even then, your tests might reveal answers, but not the relative levels of impact. sure, having a keyword on a page in strong tags is better than not, but by how much? if one link from the crappiest pr1 page gives more of a boost, is it really worthwhile?

i've talked about this testing phenomenon in the past in a sphinn thread, about whether nofollow sculpting has any impact (i've copied the relevant bit below):

step 1: register a new domain (preferably one with a domain name that has no results in google - like yorkfabuzapeloh.com or such)

step 2: link to that domain's homepage from some social media profiles or pages you control (but make sure they're very obscure and hard to find so no one else discovers and links to it - this is pretty easy to do)

step 3: create 6 pages on the site, the homepage (a) with two links to pages (b) and (c), pages (d) and (e) - both linked to by page (b) - and page (f) linked to from page (c). it's important to make sure that (b) is the first link on the homepage (a) and (c) is the second link.

step 4: target a nonsense keyword on pages (d) and (f), which are linked to by pages (b) and (c) respectively.

step 5: wait until google has indexed all the pages (usually only a couple days if you link to them from a few sources), then run a search for the nonsense keyword you targeted on (d) and (f). page (f) will rank first, because there's more link juice pointing to it than to (d), as (d) is only getting half the link weight provided by page (b) while (d) is getting all of (c)'s link weight.

step 6: add a nofollow to the link from page (b) to page (e), which we haven't done anything with until now. wait until google respiders, then check the results again. (d) should now be ranking in front of (f), because it's receiving the same link weight as (f) but the original link from the homepage (a) to (b) is higher up on the page, which gives it a tiny bit more weight.

we've replicated this experiment as have several others, and certainly any global link weighting system similar to the original pagerank formula would lead you to this conclusion as well.

and i used another test we've performed internally at last week's sempdx conference, which created a bit of confusion, and is, ultimately, the reason for this blog post.

directly following my keynote, a question was asked in which link placement on a page became relevant. i commented that it was important to note that only the first anchor text to a given target page would be counted by google (we haven't yet tested yahoo!/msn), but there were a great number of audience members who came up to me during the day asking for clarification -- even rebecca! and thus, even though we usually keep this kind of information internal (jane's planning to release a pro guide with lots of these tests later this year), i figured the beans had already been spilled, so it's my responsibility to clean up the mess.

here's what i mean -- let's say that on your website's homepage, you have two links to your blog. the first link is in the top level navigation, and the anchor text is "blog." the second link is in the body of the homepage and reads "celebrity news blog." that second link's anchor text is not going to help the blog page rank for "celebrity news" because google doesn't appear to count the anchor text from multiple links to a target from a single url. here's a visual example:

hopefully this clears up the misconception i created at the conference and helps get everyone thinking about the value of testing. it can be complex and time consuming (we run our tests on three nonsense domains, verifying that we get the same results every time), but rewarding. obviously, even armed with just the knowledge from the test described above, there's a lot of extra thought to put into how your website's internal link structure should function (and yes, you can use nofollow on the first link if you want the second link's anchor text to count - let's test this - orgzhetwarhyu... tyynhaurslfhgn).

p.s. two good questions were asked in the comments that deserve addressing in the post. first, this would appear to apply to the position in the code, not on the actual visual representation of the page, as google isn't currently running 30+ billion documents through visual page analysis. second, as far as pr "leaks" go, ideally you'd only want one link from any page to any other, but the original pr formula appears to do this for you, as they don't consider multiple votes for a url by a single page to provide benefit (each page can only vote for another once).

p.p.s. on stuff like this, it's never a good idea to just take my word for it (or anyone else's) - run the tests yourself and see the results you get. since the engines are evolving all the time, the results might be different in six months or six days.

do you like this post? yes no

posted by randfish

on thursday morning (yesterday), seomoz reached 2000 pro members and we had a mini celebration at the office to commemorate.

to honor the occasion, mystery guest stayed up until midnight the night before, baking us fantastic, shortbread, chocolate-dipped cookies in the shape of 2s and 0s.

left to right: danny, rand, sarah, jane, gillian, susan, nick, scott & christine

susan moskwa had stopped by to do a whiteboard friday (which should be posted any minute now) and joined us in the celebration.

obviously, i'm ecstatic. it may be an arbitrary number, but it's been symbolic to me and many of the other mozzers for months now as a potential tipping point for the service. i'm really proud of what the team at seomoz has achieved - buiding great tools and a terrific website, interacting and inspiring people throughout the seo community and working together to build an atmosphere that makes me excited to go to work every morning.

we're not done - not by any means. in fact, to be perfectly honest, the pro toolset and value proposition today is only the tip of the iceberg when it comes to what we'd like to offer in the future. several of us got together for a nearly two-hour meeting (we got a bit sidetracked by some engineering challenges) today on features and functionality for a completely new tool that we think will be a remarkable boost for competitive analysis and brand tracking.

in addition to congratulating my team and sharing a few photos, i'd also like to use this post to officially announce our intention to launch v4 of seomoz (the 4th iteration of the site since its birth in 2004). that launch is expected to arrive right around halloween of this year (october 31st), and will feature a ton of new material, new functionality, a new look and feel and a considerably faster load time (as we're also planning to add load balancing and more robust, scalable code).

however, we can't possibly predict every feature our members (that's you!) might enjoy, so please feel free to leave comments with your suggestions for the pro section, the free stuff, the blog, youmoz, the marketplace, events or any other part of the site. in the past, we've gotten many of our best suggestions from our membership, and are hopeful that this time is no different. obviously, i can't promise that we'll include every feature, but with a longer development cycle, we'll be able to accomplish quite a bit. no suggestion is too big or too small, so please do contribute.

finally, a huge thanks to all our members - both pro and non. early next week, we're on schedule to reach 70,000 total seomoz members, another milestone, and looking over the blog, the marketplace, q+a - the areas where we get to interact as a community, i'm amazed at the quality of people, content and discussion. if we could send you all delicious shortbread cookies and champagne, we totally would :)

p.s. have you noticed how good the youmoz content has been lately? seriously, there are some very good posts going up there of late - bravo, youmozzers!

do you like this post? yes no

yahoo snags joanne bradford, microsoft's ex-sales chief, who clearly was not happy running a startup. joanne is a pro, this could do wonders for yahoo's display sales.