Exalead Blog - The blog of Exalead

Jean Marc brought you a very delightful post about Chromatik last week with a lot of beautiful images. I will now describe in more detail how it was built. As with the DVD you perhaps watched last night, I am afraid there will be fewer big special effects in this blog than in Jean Marc’s post, but I hope to give you an insightful view of what happened behind the curtain.

Chromatik was an elaborate demo, the result of a long effort on both the back-end and the front-end. It indexes one million images. For all of them, a color signature was built and indexed. Our current intuitive user interface, exploits this index to help you filter and select images by choosing a combination of colors, luminosity or text.

A large number of people tried and liked the Chromatik demo so much that we received several requests to integrate it into the official Exalead search site. And because the demo ran relatively bug free and smoothly, our friends thought it was a piece of cake. Of course, it was a bit more work than perceived. So where are the challenges?

1) The front-end side

A lot of questions are to be answered:

• How will I adapt the GUI of my application to integrate the new features ?
• Are all these new features necessary ?
• What is the feedback we’ve received on the different features ?
• What is the added value of these features ?

The answers to these questions will impact the total amount of space on the GUI we will take for surfacing them.

2) The back-end side

Let’s begin with a little theory:

Theorem of the factor 10 effect:
No matter how good a developer you are, if non-trivial code has been designed and tested with only N elements, it won’t work without modifications when applied to 10 * N elements.

Demonstration: Rather simple: if you don’t believe it, try it yourself…

In this case we wanted a factor 1000, so we knew it would need some adjustments but when you know this theorem, the advantage is that you can anticipate and the experience we have gathered with these situations at Exalead enables us to know most of the bottlenecks.

Example 1: Chromatik needed 300MB RAM which is quite good for 1M images. But now, if you multiply this number by 2000, it gives you 600GB RAM which is quite large even if the final index is distributed over multiple machines.
We therefore decided to reduce the richness of the colors, while maintaining usability, migrate from version 4.6 to version 5.0 of Exalead CloudView and use a more compressed encoding. In the end, it now only costs 9GB.

Example 2: When you want to analyze two billion images, you need to have a robust code, which means that’s able to handle all sort of images even those which do not have a valid RFC. It’s not that easy, when even the most used library in the world for basic image manipulation can crash on some images as we reported.
The result was that this run spotted some bugs in our code that we hadn’t seen before and therefore had to fix.

Example 3: The demo was initially a single machine application. We needed to use the distributed system framework included in the CloudView technology to be able to run the whole process of extracting, crawling, indexing in only a few weeks. This framework really helped us transform the single machine demo to a fully load-balanced and monitored application. This use case is a little different than our standard www.exalead.com chain, so we discovered and tweaked a few points in the code that were cumbersome.

The purpose of this integration was to offer a new service to the users of the exalead.com search engine and improve the robustness of the Chromatik technology. We now better understand the impact of different tweaks on color indexing.

Transforming a demo into a real product is not as easy as it seems. I hope this post helps you understand why a lot of companies only show you demos but never real live applications.

At Exalead, we don’t sell demos to our customers; we sell tested and robust solutions. We make sure we work hard to test and uncover all the issues so our customers’ implementations go smoothly.

Windows 7, the newest version of Microsoft Windows, is scheduled to hit the shelves October 22. Worried about how your favorite desktop search application will function if you upgrade? No need to panic! Exalead Desktop is already functional with Windows 7!

And by cat, I mean our North American expansion and our new product line CloudView .
In June, we appointed Paul Doscher (former CEO of open source BI leader Jaspersoft) our North American CEO.

Tuesday 23 at the Enterprise Search Summit West, Exalead announced the CloudView family of products. Available in Q4 of 2008, this family includes Cloudview OEM, Cloudview Search and Cloudview desktop.

Drowning in data…

Data exists locally on a variety of devices, behind the firewall with databases and legacy systems and outside the firewall with SaaS applications and on the Web. That’s a whole lot of data from a whole lot of different sources. Peering into this enterprise content cloud is hard work for companies that need to be able to bring together a veritable mountain of unstructured and structured data and make it actionable for faster business decisions.

Bottom line: this content cloud has created the demand across all industries for information access platforms with better connectivity, better interoperability and better scalability. Being able to expand their view into various data sources (and into structured and unstructured data) in a way that brings meaningful context and understanding (i.e. not data in a vacuum) equates to an enormous competitive advantages for any enterprise.

Of course, this requires a platform that can do several things: handle a huge amount of data, scale as the company and data pool grows and changes, and allow flexibility in customizing the solution for specific industries. The old guard search tools simply aren’t cutting it anymore in this evolving space.

170 customers worldwide and 100 million unique viewers a month means that we here at Exalead have a clear view of the emerging issues that face enterprise customers. Or perhaps put differently, we have a Cloud-y View of what companies need to stay afloat through the next wave of information access challenges.

“Where did I save this file?” If that question sounds familiar to you -or any less polite version of it- exalead one:desktop is the tool you need.

Designed for those who need unified and simplified access to all data saved on their PCs -documents, messages, attached files, contacts, appointments, images, audio and video files…- exalead one:desktop is now available in its new version 4.6

exalead one:desktop belongs to the exalead one:search range of products. The professional edition of exalead one:desktop enables the ability to search simultaneously on your PC, your enterprise servers and the Internet.

In addition to the existing functionalities of Exalead’s desktop product which include easy navigation, search in real-time, unlimited scalability, indexing an unlimited number of documents in 120 file formats (300 in the professional edition) and email sources (including Microsoft Outlook, Lotus Notes and Thunderbird, etc), the new tool offers the following features:
- One click tabbing between desktop, web, image, video and Wikipedia™ search – allows for easy and intuitive navigation around the browser-based interface
- Easy personalization of tools in the navigation pane and optional tag cloud display for related terms enabling the user to further refine their results according to their needsFor the professional edition:
- Complete integration with exalead one:enterprise – for a consistent search experience on the PC, web and corporate intranet
- User interface white label functionality – enables companies to completely brand the user interface and personalize results through XML API
exalead one:desktop is also available in a free downloadable version. To get it, click here.To read the Press Release, click here.To get more information on exalead one:desktop product, click here.

A good search platform provides easy, quick, universal access to business information while ensuring users can only see and access content for which they have the necessary read authorization. Ensuring confidentiality in the context of enterprise search means that the search solution doesn’t seek to replace or even alter the existing security policies.

In addition, an enterprise search solution should accommodate an organization’s tolerance for latency in permissions changes. During the process of generating results, a secure search engine needs to match the authenticated user’s access rights with the retrieved documents’ access rights (defined in Access Control Lists, or ACLs).

To determine a user’s read rights for a particular resource when executing a search query, there are two basic options, each with a different impact on latency:

• Real-Time ACL Checking (sometimes referred to as “unmapped security” or “post-query filtering”): the search platform checks the ACL at query time for each individual search result returned by a query.

• ACL Indexing (also known as “mapped security” or “ACL mapping”): at indexing time, information regarding the users and their rights is appended as metadata to the index entry for a document. Consequently, at query time, the search platform simply checks the metadata attached to each index entry rather than checking with the source application’s ACL.

Because Real-Time ACL Checking is hard to manage and puts a heavy load on the server and incumbent systems, Exalead recommends ACL Indexing as a standard approach. It is the fastest, most scalable option. It also offers a very high level of granularity.

However, there is latency in permissions changes with ACL Indexing, which may be unacceptable in certain contexts. Therefore, exalead one:enterprise can optionally combine the two ACL checking methods to optimize performance while ensuring a responsive approach to changes in security access rights.

To learn more about how security works in the context of enterprise search, and about Exalead’s approach to security, you can request a copy of our white paper “Secure Search Solution” by emailing us at contact@exalead.com.

The release of exalead one:enterprise 4.6 offers several new enhancements that are designed to help organizations easily configure and customize business applications, including hybrid vertical search applications. In addition, exalead one:enterprise 4.6 offers new connectors for enterprise databases and a unique developer toolkit designed to easily structure unstructured data.

Also included in exalead one:enterprise 4.6 is a unique HTML extraction toolkit, which is designed to take advantage of the structural indicators that are already built-in to HTML pages. Exalead has added a variety of new features that will appeal to IT Managers and Original Equipment Manufacturer (OEM) partners, including several new application programming interfaces (APIs). As a result, exalead one:enterprise can serve as a flexible platform for white box solutions.

Included is a new enterprise connector for Microsoft Office SharePoint Server 2007, allowing Microsoft users to enjoy extremely high scalability – several billions of documents – the ability to index multimedia files, enriched query processing and Exalead’s Search by Serendipity experience.

Read the Press Release
More details