Chromatik was an elaborate demo, the result of a long effort on both the back-end and the front-end. It indexes one million images. For each image, a unique color signature was built and indexed. Our current intuitive user interface, exploits this index to help you filter and select images by choosing a combination of colors, luminosity or text.

A large number of people tried and liked the Chromatik demo so much that we received several requests to integrate it into the official Exalead search site. And because the demo ran relatively bug free and smoothly, our friends thought it was a piece of cake. Of course, it was a bit more work than we initially expected. So where are the challenges?

1) The front-end side

A lot of questions needed to be answered:

• How will I adapt the GUI of my application to integrate the new features?
• Are all these new features necessary?
• What is the feedback we’ve received on the different features?
• What is the added value of these features?

The answers to these questions will impact the total amount of space on the GUI we will take for surfacing them.

2) The back-end side

Let’s begin with a little theory:

Theorem of the factor 10 effect:
No matter how good a developer you are, if non-trivial code has been designed and tested with only N elements, it won’t work without modifications when applied to 10 * N elements.

Demonstration: Rather simple: if you don’t believe it, try it yourself…

In this case we wanted a factor 1000, so we knew it would need some adjustments. When you know this theorem, the advantage is that you can anticipate potential problems. And the experience we have accumulated from similar situations at Exalead help us predict most of the bottlenecks.

Example 1: Chromatik needed 300MB RAM, which is quite good for 1M images. But, if you multiply this number by 2000, you have 600GB RAM, which is quite large, even if the final index is distributed over multiple machines.
We therefore decided to reduce the richness of the colors, while maintaining usability, migrate from version 4.6 to version 5.0 of Exalead CloudView, and use a more compressed encoding. In the end, it now only costs 9GB.

Example 2: When you want to analyze two billion images, you need to have a robust code, which means that’s able to handle all sort of images even those which do not have a valid RFC. It’s not that easy, when even the most used library in the world for basic image manipulation can crash on some images as we reported.
The result was that this run spotted some bugs in our code we hadn’t seen before and therefore had to fix.

Example 3: The demo was initially a single machine application. We needed to use the distributed system framework included in the CloudView technology to be able to run the whole process of extracting, crawling, and indexing in only a few weeks. This framework really helped us transform the single machine demo to a fully load-balanced and monitored application. This use case is a little different than our standard www.exalead.com chain, so we discovered and tweaked a few cumbersome points in the code.

The purpose of this integration was to offer a new service to the users of the exalead.com search engine and improve the robustness of the Chromatik technology. We now better understand the impact of different tweaks on color indexing.

Transforming a demo into a real product is not as easy as it seems. I hope this post helps you understand why a lot of companies only show you demos but never real live applications.

At Exalead, we don’t sell demos to our customers; we sell tested and robust solutions. We make sure we work hard to test and uncover all the issues so our customers’ implementations go smoothly.

Tuesday 23 at the Enterprise Search Summit West, Exalead announced the CloudView family of products. Available in Q4 of 2008, this family includes Cloudview OEM, Cloudview Search and Cloudview desktop.

Drowning in data…

Data exists locally on a variety of devices, behind the firewall with databases and legacy systems and outside the firewall with SaaS applications and on the Web. That’s a whole lot of data from a whole lot of different sources. Peering into this enterprise content cloud is hard work for companies that need to be able to bring together a veritable mountain of unstructured and structured data and make it actionable for faster business decisions.

Bottom line: this content cloud has created the demand across all industries for information access platforms with better connectivity, better interoperability and better scalability. Being able to expand their view into various data sources (and into structured and unstructured data) in a way that brings meaningful context and understanding (i.e. not data in a vacuum) equates to an enormous competitive advantages for any enterprise.

Of course, this requires a platform that can do several things: handle a huge amount of data, scale as the company and data pool grows and changes, and allow flexibility in customizing the solution for specific industries. The old guard search tools simply aren’t cutting it anymore in this evolving space.

170 customers worldwide and 100 million unique viewers a month means that we here at Exalead have a clear view of the emerging issues that face enterprise customers. Or perhaps put differently, we have a Cloud-y View of what companies need to stay afloat through the next wave of information access challenges.

Designed for those who need unified and simplified access to all data saved on their PCs -documents, messages, attached files, contacts, appointments, images, audio and video files…- exalead one:desktop is now available in its new version 4.6

exalead one:desktop belongs to the exalead one:search range of products. The professional edition of exalead one:desktop enables the ability to search simultaneously on your PC, your enterprise servers and the Internet.

In addition, an enterprise search solution should accommodate an organization’s tolerance for latency in permissions changes. During the process of generating results, a secure search engine needs to match the authenticated user’s access rights with the retrieved documents’ access rights (defined in Access Control Lists, or ACLs).

To determine a user’s read rights for a particular resource when executing a search query, there are two basic options, each with a different impact on latency:

• Real-Time ACL Checking (sometimes referred to as “unmapped security” or “post-query filtering”): the search platform checks the ACL at query time for each individual search result returned by a query.

• ACL Indexing (also known as “mapped security” or “ACL mapping”): at indexing time, information regarding the users and their rights is appended as metadata to the index entry for a document. Consequently, at query time, the search platform simply checks the metadata attached to each index entry rather than checking with the source application’s ACL.

Because Real-Time ACL Checking is hard to manage and puts a heavy load on the server and incumbent systems, Exalead recommends ACL Indexing as a standard approach. It is the fastest, most scalable option. It also offers a very high level of granularity.

However, there is latency in permissions changes with ACL Indexing, which may be unacceptable in certain contexts. Therefore, exalead one:enterprise can optionally combine the two ACL checking methods to optimize performance while ensuring a responsive approach to changes in security access rights.

To learn more about how security works in the context of enterprise search, and about Exalead’s approach to security, you can request a copy of our white paper “Secure Search Solution” by emailing us at contact@exalead.com.

Also included in exalead one:enterprise 4.6 is a unique HTML extraction toolkit, which is designed to take advantage of the structural indicators that are already built-in to HTML pages. Exalead has added a variety of new features that will appeal to IT Managers and Original Equipment Manufacturer (OEM) partners, including several new application programming interfaces (APIs). As a result, exalead one:enterprise can serve as a flexible platform for white box solutions.

Included is a new enterprise connector for Microsoft Office SharePoint Server 2007, allowing Microsoft users to enjoy extremely high scalability – several billions of documents – the ability to index multimedia files, enriched query processing and Exalead’s Search by Serendipity experience.

Read the Press Release
More details