Chromatik was an elaborate demo, the result of a long effort on both the back-end and the front-end. It indexes one million images. For all of them, a color signature was built and indexed. Our current intuitive user interface, exploits this index to help you filter and select images by choosing a combination of colors, luminosity or text.

A large number of people tried and liked the Chromatik demo so much that we received several requests to integrate it into the official Exalead search site. And because the demo ran relatively bug free and smoothly, our friends thought it was a piece of cake. Of course, it was a bit more work than perceived. So where are the challenges?

1) The front-end side

A lot of questions are to be answered:

• How will I adapt the GUI of my application to integrate the new features ?
• Are all these new features necessary ?
• What is the feedback we’ve received on the different features ?
• What is the added value of these features ?

The answers to these questions will impact the total amount of space on the GUI we will take for surfacing them.

2) The back-end side

Let’s begin with a little theory:

Theorem of the factor 10 effect:
No matter how good a developer you are, if non-trivial code has been designed and tested with only N elements, it won’t work without modifications when applied to 10 * N elements.

Demonstration: Rather simple: if you don’t believe it, try it yourself…

In this case we wanted a factor 1000, so we knew it would need some adjustments but when you know this theorem, the advantage is that you can anticipate and the experience we have gathered with these situations at Exalead enables us to know most of the bottlenecks.

Example 1: Chromatik needed 300MB RAM which is quite good for 1M images. But now, if you multiply this number by 2000, it gives you 600GB RAM which is quite large even if the final index is distributed over multiple machines.
We therefore decided to reduce the richness of the colors, while maintaining usability, migrate from version 4.6 to version 5.0 of Exalead CloudView and use a more compressed encoding. In the end, it now only costs 9GB.

Example 2: When you want to analyze two billion images, you need to have a robust code, which means that’s able to handle all sort of images even those which do not have a valid RFC. It’s not that easy, when even the most used library in the world for basic image manipulation can crash on some images as we reported.
The result was that this run spotted some bugs in our code that we hadn’t seen before and therefore had to fix.

Example 3: The demo was initially a single machine application. We needed to use the distributed system framework included in the CloudView technology to be able to run the whole process of extracting, crawling, indexing in only a few weeks. This framework really helped us transform the single machine demo to a fully load-balanced and monitored application. This use case is a little different than our standard www.exalead.com chain, so we discovered and tweaked a few points in the code that were cumbersome.

The purpose of this integration was to offer a new service to the users of the exalead.com search engine and improve the robustness of the Chromatik technology. We now better understand the impact of different tweaks on color indexing.

Transforming a demo into a real product is not as easy as it seems. I hope this post helps you understand why a lot of companies only show you demos but never real live applications.

At Exalead, we don’t sell demos to our customers; we sell tested and robust solutions. We make sure we work hard to test and uncover all the issues so our customers’ implementations go smoothly.

What is this graphic about?
It represents a semantic analysis of the relationship between terms used on the Web to speak about Intellectual Property Rights in the French language.  Each node symbolizes a word or a group of words and each edge connects two expressions when these are co-cited in more than 120 000 web pages.  Each color refers to a “semantic cluster”, which is a bunch of words than concern the same topic.

How can I get this type of graphic?
After an extraction of related terms found on Exalead databases and a manual filtering phase, the project team receives a GDF file with ordered data.  Then, the exploitation of this file by Gephi combined with a specific algorithm leads to the data “spatialization”. Then color filters highlight different semantic clusters.

Here is one of the first demonstrations of Gephi with real-time spatialization of several keyword clusters. In this video, the blue color refers to a “genetics” cluster, orange nodes relate to terms about biology and laboratories, green ones concern words speaking about controversy in the domain of GMOs and purple nodes relate to innovation and research development in biotechnology.

First up: answers to the two most frequently posed webmaster questions:

1) Why doesn’t my site appear (or why does it only partially appear) when I do a site search (i.e., typing “site: mysitename.com” in the search box)?

All or part of your site may be inaccessible to our robots. Try the following to improve your performance:

• Make sure that all pages are accessible by at least one static link.
• Place links to your most important content on every page of your site.
• Keeping in mind that certain dynamic pages can’t be accessed by our robots, move content as needed to static (or simply more accessible) pages (see “The Road to Better Site Indexing – Introduction and Episode 1”)
• Be sure the robots.txt file in your root directory is not blocking access to our crawler (use our robot checker form to test accessibility).
• Create a site map (see “The Road to Better Site Indexing: Episode 3, Sitemaps”) and submit it on our site.

2) Why doesn’t my site appear for a given keyword?

• First, check to see that the keyword is in our index for your site. Enter the keyword in the search field, along with “site:mysitename.com” to limit the search for that keyword to just your site (replacing “mysitename.com” with your domain name, of course). If it is not indexed, follow the steps for question 1 above.
• Refine the keywords in your site so they are as specific as possible. It could be the keyword you are checking is too general, and sites that larger, more relevant and/or more popular are ranking ahead of your site for that keyword.
• Verify that the content of your site corresponds well to the keyword. It’s not enough for a keyword to simply appear, it must be integrally related to the rest of the site content.

You’ll find further info on keyword relevancy in Search Engine Optimization (SEO): More Old-School Than You Think.”

And be careful out there! Stick to keeping your content fresh and relevant for your target audience. Reverting to tricks like hidden text, duplicate content, spam link exchanges or other such tactics to improve your ranking could get you banned from our index (for more info, see “The Road to Better Site Indexing – Episode 2”).

You’ll also find general webmaster tips in our site’s help pages.

zdnet.fr	thatvideosite.com
comedycentral.com	videonetart.com
AskANinja.com	wat.tv
wideo.fr	blip.tv
veoh.com	onowa.com
nasa.gov	video.on.nytimes.com
latelelibre.fr	sports.espn.go.com
feeds.reuters.com	stage6.divx.com
stupidvideos.com	livevideo.com
video.lequipe.fr	archive.org
channels.ourmedia.org	revver.com

Now that the RSS mode has been activated, all that’s needed to add a new source is to locate its corresponding RSS feed. So if you find a good video source, send us the feed URL.

We currently support the Media RSS format, adopted by the majority of video content distributors: http://en.wikipedia.org/wiki/Media_RSS

All you need to do is send us your feed URL so our crawler can fetch it.

Once your feed is submitted, our crawler will check back regularly to verify that the video is still available.

An example of a video feed:

<?xml version=”1.0″ encoding=”utf-8″?>
<rss version=”2.0″ xmlns:media=”http://search.yahoo.com/mrss/”>
<channel>
<title>My site</title>
<link>http://www.mysite.com/rss/mrss.xml</link>
<description>Videos published on my site</description>
<item>
<author>jane56</author>
<title>Interview with Tom</title>
<link>http://www.mysite.com/video/1</link>
<description>Tom responds to my questions about the new product.</description>
<guid isPermaLink=”true”>http://www.mysite.com/video/1</guid>
<pubDate>Mon, 25 Nov 2007 08:42:00 +0000</pubDate>
<media:content url=”http://www.mysite.com/player/1/interview_de_tom.swf”
type=”application/x-shockwave-flash”/>
<media:content duration=”325″ >
<media:thumbnail url=”http://www.mysite.com/vimages/1.jpg” width=”340″
height=”250″ />
<media:keywords>Tom, interview, new</media:keywords>
<media:rating scheme=”urn:simple”>nonadult</media:rating>
<media:category>Entertainment</media:category>
</item>
</channel>
</rss>

<guid>:
-The guid tag contains the URL of a page where the video can be found. When the user runs a search and clicks on a result, he/she will be directed to this URL.

<thumbnail>:
- The thumbnail tag contains a link to a descriptive image for the video.

<pubDate>:
- This tag is for the publication date of the video.

<media:content>:
-This tag contains a direct link to the video. ‘Type’ is a standard video MIME type.

<media:keywords>:
- A list of keywords associated with the video.

<media:category>:
- One or more categories associated with the video.

<media:rating scheme=”urn:simple”>:
- Indicates if the content is ‘adult’ or ‘nonadult’ (suitable for minors) in nature.

This list is not exhaustive. See http://search.yahoo.com/mrss for further specification details.

Contact us if you have any technical questions!

But being a perfectionist, Bot wasn’t entirely satisfied with his own method. Was he overlooking a significant thread? Leaving an important page unturned? He had a hunch he could do better.

Leaving important content in the dustbin of unindexed pages was just the sort of slip-up that really peeved Bot’s equally perfectionist client Betty, a.k.a. “The Webmaster.” Betty had specifically called on Bot to crawl her entire site, and Bot had missed several pages.

To get their relationship back on the right track, Bot had an idea: he would ask Betty to tell him flat out everything she wanted him to know about her site. And being a guy always in the know, Bot knew just what tool Betty could use to set the record straight: a sitemap.
He proposed; she accepted.

Now Betty can rest easy knowing all the content she wants to share with the world will be indexed. And just what is this handy tool known as a sitemap?
It’s actually not much more than a laundry list of links. Constructing one is a snap. You simply create a text file listing the URLs you want indexed, along with any key facts you want Bot to know (like how often a file is updated), and place it anywhere you’d like, giving Bot the location in your robots.txt file, for example at the root of your web site: http://www.example.com/sitemap.xml.

Sitemaps can be written in XML (the preferred method), or communicated via syndication feeds or simple text files. A sitemap in XML looks something like this:

<urlset xmlns=”http://www.sitemaps.org/schemas/sitemap/0.9″>
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>http://www.example.com/catalog?item=12&desc=vacation_hawaii</loc>
<changefreq>weekly</changefreq>
</url>
<url>
<loc>http://www.example.com/catalog?item=83&desc=vacation_usa</loc>
</url>
</urlset>

You can visit http://www.sitemaps.org/ for all the details. It’s the official site of the Sitemaps protocol, which was first proposed by Google, then fleshed out through discussions with MSN, Yahoo and Ask. It’s now the standard adopted by Google, Yahoo, Ask, and, as of July 2007, Exalead.
But bad guys consider yourselves forewarned: Bot knows not every webmaster is not as straight up as Betty. He stays a step ahead of all nefarious sitemap tricks, checking out every list of links spun his way and skipping right over bum lists.

Sébastien

What’s a link
farm? Let’s look at the randomly chosen site http://www.rc-car.ravemart.com. At
first glance, this remote control car ecommerce site seems to be a typical
small biz site plying its wares in the typical way.

Now take a closer
look by clicking on some of the text links at the bottom of the page. You’ll
see this site has decided to aid the search engines by providing links to
thousands of its friends’ sites. Click on the generically titled “Link” and
you’ll find links to the company’s “Partners” under categories such as Debt
Consolidation, Vitamins, Legal Services and Sweepstakes. Or click on
“Sponsored Links” for fast access to Cheap Viagra, Casino And Poker
News, Psychic Readings, or Online Dating Services.

Similarly, visit www.all-carpets.com where under “Resources” you’ll not only find links
for Home Flooring and Kitchen Design (not too far off base), but also Cash Back
Credit Cards, South African Zulu Culture, and Offshore Banking (uh-oh, we’re in
left field now…).

Even larger
companies and organizations may participate in these types of link programs,
where all members link to all other members, often regardless of the relevance
of the content, in the somewhat desperate hope of augmenting their popularity
and hence appearing higher in search results.

If you’re a web
searcher trying to find quality results, don’t worry, we remain vigilant .
If you’re a site owner, avoid participating in link farms. You may find the
strategy backfires as your site is demoted or even dropped from some search
engine databases. Instead seek out quality reciprocal links with sites with
whom you share a genuine relationship, and you’ll build the kind of true
popularity both visitors and search engines appreciate.

Next episode: The
Integration of RSS Feeds.

Sebastien, Web
Team Head Chef

Firest, you should know that pages
accessible uniquely through JavaScript or through form submissions are not
reachable by search engines and therefore they cannot be indexed. And there is
no means for a search engine to know whether it’s missing some pages in a site,
whether the missing page count is 10 or 10,000 (outside of site maps, which I
will discuss in a future post).

Next, let’s refresh ourselves on the
fundamental methods search engines use to find the pages they index: 1) They
follow a submission made by a human being (0,0001% of cases), or 2) They follow
a link from another page. Therefore, if there is a link to a given page, the
probability that it will be indexed is high. Alternately a personal site with
no external links to it has little chance of being indexed by a search engine.
So more links are always better, right?

Not necessarily. It should be understood that from the point of view of
a search engine, a risk arises not from a dearth of links, but from too many.
Why? Because search engines seek to provide the most relevant results for
visitors, returning pages with the content most likely to match visitors’ needs
and expectations. A site that arrived at the top of the results solely because
there were tens of thousands of links to it would not pass this test. In fact,
an overabundance of external links may indicate a “spamming” campaign aimed at
search engines and be an indicator of poor site quality.

Here are two cases of what we’ll call legitimate ‘overabundance,’ an overabundance of links due to valid, non-spamming factors that can be properly managed by search engines.

Case 1: User Sessions

When you visit
an e-commerce site, unique “session” information will often be assigned to your
computer. This information uniquely identifies your particular connection and
visit. It may include, for example, a unique ID for your computer and a code
for your browser version or geographic location.

This session information tracks your movements,
preferences and selections as you navigate a site. This is not for nefarious
ends, but is rather used to perform practical tasks like maintaining items in
your shopping cart, showing prices in your local currency or displaying a list
of products you’ve viewed. This session information is most often added to the end
of the URL (web address) for every page you visit.

For instance, say you are visiting Amazon.com and you
navigate to a Stanley wrench set. The URL displayed in your browser is

http://www.amazon.com/Stanley-92-716-Combination-Wrench-22-Piece/dp/B000JPUCT0/
ref=sr_1_7/002-6118145-0432018?ie=UTF8&s=hi&qid=1181650669&sr=1-7.

Only the first part of the URL,

http://www.amazon.com/Stanley-92-716-Combination-Wrench-22-Piece/dp/B000JPUCT0/,
is needed to locate the product information for this wrench set. The rest,
“ref=sr_1_7/002-6118145-0432018?ie=UTF8&s=hi&qid=1181650669&sr=1-7″

is session information for your
particular visit.

A search engine may come across thousands of links
like the longer address, each of which may appear different because unique session
information is appended, and because each may show different user-dependent
content such as navigation history, promotions, or recommended products. But
any search engine worth its salt can discern the repetitive addresses from the
essential URL, and will know this is not a case of spamming.

Case 2: Calendar Menus

Some sites let you navigate through their
content by clicking on a calendar. For example, you may be able to peruse news
articles or events on a site by choosing a date or date range.

Such menus generate links like:
http://www.ecvd.eu/index.php?option=com_events&task=view_month&Itemid=32&year=2011&month=09&day=12

A competent search engine will know which
of these types of links returns valid content and which does not, and what
baseline URL should be included in a search index. In other words, having a
zillion external links for events on dates from 1950 to 2060 for a site with ten
events will definitely not boost that site’s ranking .

Now you may say these two cases look like
easy ones for a search engine to manage, and you’d be right. The real
difficulties arise from the following three cases, because (scoop!) there are
unscrupulous people out there ready to do anything to improve their search
engine ranking. You’ve most likely encountered their handiwork when using a
search engine other than Exalead.

You run your search and click on a page you
think is relevant, only to encounter an endless list of meaningless links or
keywords, a pastiche of content “borrowed” from other more relevant sites, or
an endless loop of promising links that ultimately go nowhere.

These types of pages are generated by the
folks at the top of our list of ballot-box stuffers, those trying to improve
their search engine rank through:

* Link farms and keyword stuffing,

* Content scraping, including the abuse of
RSS Feeds, and

* Creating content labyrinths.

We’ll be covering these tactics in upcoming
episodes. In the meantime, you can see why search engines may need to limit the
number of pages they index for a site. This ‘quota’ is determined based on the
site’s reputation, the duplication of its content, and a thousand other
parameters, all factored in an attempt to keep the game honest so web searchers
get the most relevant search results possible.

Sebastien, Head Chef, Web Team

This post is dedicated to these developers.

Javascript is a rich language, it is object oriented and easy to learn.

The basics
The Mozilla Developer Connection is the best documentation source for Javascript. To learn the syntax and the basic data types, read A re-introduction to Javascript.

Objects
In Javascript, an object is basically a hash. The simplest way to create an object is:

var o = {};
o.name = "hello";
o.setName = function(name) {
  this.name = name;
}

In this example, o is an object, its name property is set to hello, and its setName property is a function.

Classes and inheritance.
Javascript handles the notion of class in an unusual way:

• a Class is an object of type Function
• this object has a magic member called prototype
• when invoked with the new operator, a new empty object is created and the prototype is copied inside.

Example:

var MyClass = function() {
  //contructor
  this.name = "default name";
};
MyClass.prototype = {
  //prototype
  setName: function(name) {
    this.name = name;
  }
};

var o = new MyClass();
alert(o.name); // "default name"
o.setName("new name");
alert(o.name); // "new name"

There are multiple ways to write classes. For a full overview, read the excellent Classical inheritance in Javascript.

Access control
In Javascript, everything is public. The best way to preserve the notion of privacy is through convention. For instance, at Exalead we prefix private members with an underscore.

Everything Dynamic
In Javascript, everything can be changed at all times. Even methods. This is particularly useful when coding event based User Interfaces:

o.onReceiveSomeEvent = function() {
// do something
};

It is even possible to enrich basic data types with your own methods by adding methods to their prototype.

Example:

String.prototype.blank = function() {
  return /^\s*$/.test(this);
}
"hello".blank(); // false
"     ".blank(); // true

The DOM – Document Object Model

Javascript runs in the browser in an HTML page. A representation of that page – the DOM – is available to Javascript through the global window object. Of course, it would be too easy if the DOM were identical between browsers. How then, do people write cross browser code? There are 2 approaches at least:

Approach #1:

if (navigator.appName == "Microsoft Internet Explorer"
    && navigator.appVersion >= "4.0") {
  element.attachEvent("onclick", function() {alert("click")});
}

if (navigator.appName != "Microsoft Internet Explorer"
    && navigator.appName != "Netscape") {
  element.addEventListener("click", function() {alert("click")});
}

This is the most trivial approach and the least elegant and scalable. It makes sharing code a nightmare and will inevitably make you hate Javascript. Don’t use it, ever.

Approach #2:

if (element.attachEvent) {
  element.attachEvent("onclick", function() {alert("click")});
}
if (element.addEventListener) {
  element.addEventListener("click", function() {alert("click")});
}

This is a lot better. This approach uses one of javascript’s strengths: testing the existence of a function. It provides maximal compatibility with minimum browser knowledge.

That’s all for today folks! In my next post, we’ll dissect the Prototype Javascript Framework.

- Damucho, for the WebDev team