Chromatik was an elaborate demo, the result of a long effort on both the back-end and the front-end. It indexes one million images. For each image, a unique color signature was built and indexed. Our current intuitive user interface, exploits this index to help you filter and select images by choosing a combination of colors, luminosity or text.

A large number of people tried and liked the Chromatik demo so much that we received several requests to integrate it into the official Exalead search site. And because the demo ran relatively bug free and smoothly, our friends thought it was a piece of cake. Of course, it was a bit more work than we initially expected. So where are the challenges?

1) The front-end side

A lot of questions needed to be answered:

• How will I adapt the GUI of my application to integrate the new features?
• Are all these new features necessary?
• What is the feedback we’ve received on the different features?
• What is the added value of these features?

The answers to these questions will impact the total amount of space on the GUI we will take for surfacing them.

2) The back-end side

Let’s begin with a little theory:

Theorem of the factor 10 effect:
No matter how good a developer you are, if non-trivial code has been designed and tested with only N elements, it won’t work without modifications when applied to 10 * N elements.

Demonstration: Rather simple: if you don’t believe it, try it yourself…

In this case we wanted a factor 1000, so we knew it would need some adjustments. When you know this theorem, the advantage is that you can anticipate potential problems. And the experience we have accumulated from similar situations at Exalead help us predict most of the bottlenecks.

Example 1: Chromatik needed 300MB RAM, which is quite good for 1M images. But, if you multiply this number by 2000, you have 600GB RAM, which is quite large, even if the final index is distributed over multiple machines.
We therefore decided to reduce the richness of the colors, while maintaining usability, migrate from version 4.6 to version 5.0 of Exalead CloudView, and use a more compressed encoding. In the end, it now only costs 9GB.

Example 2: When you want to analyze two billion images, you need to have a robust code, which means that’s able to handle all sort of images even those which do not have a valid RFC. It’s not that easy, when even the most used library in the world for basic image manipulation can crash on some images as we reported.
The result was that this run spotted some bugs in our code we hadn’t seen before and therefore had to fix.

Example 3: The demo was initially a single machine application. We needed to use the distributed system framework included in the CloudView technology to be able to run the whole process of extracting, crawling, and indexing in only a few weeks. This framework really helped us transform the single machine demo to a fully load-balanced and monitored application. This use case is a little different than our standard www.exalead.com chain, so we discovered and tweaked a few cumbersome points in the code.

The purpose of this integration was to offer a new service to the users of the exalead.com search engine and improve the robustness of the Chromatik technology. We now better understand the impact of different tweaks on color indexing.

Transforming a demo into a real product is not as easy as it seems. I hope this post helps you understand why a lot of companies only show you demos but never real live applications.

At Exalead, we don’t sell demos to our customers; we sell tested and robust solutions. We make sure we work hard to test and uncover all the issues so our customers’ implementations go smoothly.

Press photography is about to undergo fundamental changes in its business model. Up until now, it has been “buy photo content, store content, wait for client, sell photo”. They have seen the signs, or possibly heard Clay Shirky’s message about the death of old business models in the newspaper industry

…there is one possible answer to the question “If the old model is broken, what will work in its place?” The answer is: Nothing will work, but everything might. Now is the time for experiments, lots and lots of experiments…

Press photography is soon to experiment with a new business model: moving from content owner to content producer and distributor, moving from photos to videos.

They plan (I learned this last night from a friend in the photography world) to no longer wait for customers to come buy their photos, but rather to harvest the vast amount of locally produced, real time news images and to use their photojournalistic experience to filter and edit and package to produce their own content: story, video, editing. They plan to sell or give away a packaged widget to (online) news outlets. Activating the widget will send control back to the press photography site, which then streams its story and manages it own advertising revenue from the stream.

What does this shift portend for information retrieval?

Information retrieval began as text based storage, gradually incorporating other formats, including structured textual information from databases.

Then search engines began to index photos and videos. Google added its Image tab to its home page around October 9, 2001, and its Video tab around August 14, 2006. We added our Exalead Video search tab about a year later.

These first video search systems used the same methods as with image search, using the text found around the raw data sources, and the text in file names, and in links pointing to the multimedia file, to serve as index terms.

But as more and more information may only be presented in video form, it will be necessary to index the content of videos, too. We do this in our experimental video search system, VoxaleadNews.

But what I didn’t realize until last night is that systems like Voxalead might not just be some cute add-on to a search engine, but that this might actually be what future search engines are: that information might only come in video, and that things like text and even still images, will become some quaint, outdated media, like engravings are now.

There is certainly no shortage of debate on the definition, scope and relative viability of social CRM (SCRM). Check out Social CRM: Strategy, Technology or Passing Fad? by Bob Thompson, for a recent opinion and some cross references to others’ thoughts. It’s clear that interest and momentum is high but plenty of room remains for clarification. I, for one, would like to see more in the way of reference use cases across CRM and adjacent functions to better understand processes, data usage and interdependencies. I just can’t imagine adoption hitting stride until companies can see how functional pieces and related data need fit together. All you have to do is take a look at Gartner’s rather crowded Social Software hype cycle to get a sense of potential process and data overlaps and the accompanying urge to tap the breaks a bit. Paul Greenberg, a regular contributor to ZDNet, promises more discussion on related business models and strategies..

We see that companies are beginning to get a sense of the social conversations that involve their products and services. A subset of those companies is having direct conversations with customers through social media channels. For example, one of my good friends does PR for a video game company and is crushed by the need to address misinformation and negative sentiment and to bring some control to the conversation about his clients. We look forward to going beyond the hype – seeing more examples of how companies are really building the processes and technologies around social CRM.

At Exalead, we’re working on Social CRM related technologies. Constellations is an Exalead Labs experiment that looks at connections between people and between web sites. It presents a graph of related Web sites or people.  The experiment is built upon our Web index. The approach to relating people and things is flexible and can be based upon a configurable set of factors.

At Exalead, we believe that Search-Based Applications, or SBAs, are another (I could say more “sustainable”) solution to this problem. The key to efficiently handling large amounts of data is to make sure that data access has a strong “Spatial Locality“. Quoting Wikipedia, achieving spatial locality means that “if a particular memory location is referenced at a particular time, then it is likely that nearby memory locations will be referenced in the near future.” The main problem with relational databases is that they have very poor spatial locality, because the objects they store are spread across a large number of different tables. High-end CRM or ERP solutions typically store their data on as many as 65,000 different tables, each table being stored at a different disk location. Imagine how many different disk locations the system needs to touch just to display information about a customer or a product on a call center agent’s screen, or to produce a complex BI report. Poor spatial locality leads to huge requirements for disk access, which is the main performance bottleneck for databases today.

SBAs are built on a very different data model, centered around the notion of a “business item”. A “business item” is a self-contained object corresponding to a “real-life” entity that is manipulated by the application and understood by the end-users. For example, in a CRM application, business items would be the Contacts, Opportunities and Leads that are viewed by the business users. Unlike applications built using a relational data model, a business item-centric storage strategy allows for great data spatial locality, since the pieces of information required to answer complex, multi-criteria search queries are all part of a single business item type, and hence stored close to each other on a disk. The performance gap between this local approach and the spread-out relational data model grows exponentially wider as the amount of information applications need to store increases.

Several ‘Web celebrities’ spoke about Cloud Computing, a theme picked up by some lesser-known gurus as well.

That is the case of Jeremie, an R&D Engineer at Exalead, who was chosen as a speaker for an Ignite session.

What is an Ignite session? It’s a session in which each speaker has 5 minutes to talk about a technical Web-related issue, with 20 slides that rotate automatically every 15 seconds -  before an audience composed of fellow “geeks,” of course.

Jeremie thus spoke for 5 minutes on a trendy topic in data storage solutions: “key value stores”. This data storage system appears to be more efficient than traditional relational database management systems, but it is also more simplistic. According to Jeremie’s point of view, one should be vigilant regarding the immaturity of some of these solutions. Moreover, they make data cross-referencing more complicated, and they cannot be used for every type of application. These simple tools have been created for very specific needs (e.g., there are key value stores in Exalead’s low index layers).

Here is Jeremie’s session at Ignite Velocity 09:

An interesting point is that several of our key competitors in the space took a few steps backwards between now and 2007. Endeca, Google and Autonomy all took hits on the “ability to execute” scale, Microsoft FAST lost some ground on vision. We don’t want to speculate about why exactly this is, but it does seem to indicate that the demands of the space is shifting as it grows (Gartner estimated that globally the total 2007 software revenue in the enterprise search market neared $860.3 million and is expected to grow to $1.5 billion by 2012), and that perhaps it is time for a new breed of search vendor to take the lead.

According to the report, “Visionaries demonstrate imaginative and insightful approaches to the information access technology market… and possess architectural flexibility and creative means of establishing relevancy.” Of course, we don’t wear blinders ; we know Gartner thinks that we need to establish more traction and financial resources to make the jump to the Leaders quadrant, but with 170 customers worldwide and 100 million unique viewers a month we feel we are in a position to do this.

Additionally, we just found out we received a high ranking on this year’s Software 500 list (with a 310.8% growth rate); in fact, from 2007, our rank jumped by more than 70 points. We were also named one of the Top 10 companies in the $10-30 Million revenue range as well as one of the “Top 10 Leaders in Software Revenue Growth Overall.

With these numbers and our recent expansion into the U.S., the Leaders quadrant could be just around the corner….

It is true that “cloud computing” is nebulous in nature.

We at Exalead have our own view of the cloud, perhaps better termed the “enterprise content cloud.” It is a cold, hard truth that enterprises are generating more and more information and that this information is becoming disassociated from the businesses users who need to access it. Data exists…well…everywhere– on a slew of devices, behind the firewall and outside of the firewall– and it has become a challenge to not only access this data but to even find it to make effective business decisions. Given all of these different silos of information, it is no wonder that there is a rapidly expanding chasm between the business decision makers and the information they need. In fact, only 10% of executives report that information to make a decision is usually there as needed. The data is as far away and, sadly, as inaccessible as the clouds in the sky.

Taking a step back, this fracture between information and business decision makers within the enterprise can be extended to mirror the growing disassociation between the enterprise and meeting its computing needs. Stallman’s arguments against cloud computing are rooted in his open source zealotry, so there isn’t a whole lot to say to that in response. Ellison may feel that cloud computing is antithetical to what he has accomplished at Oracle and maybe feels as if he missed out on an opportunity to capitalize on a growing movement. Regardless, both pundits should pay attention to what Galen Gruman pointed out several months ago– sure, cloud computing is nebulous, but it is most clearly realized as a way to meet business needs without a lot of investment in new infrastructure, new employees, or new software.

On-demand computing represents a multi-billion dollar market that touches a variety of verticals and has an all-star line-up of players. If you look at the software companies that have had the most IPO success over the past few years, it has been on-demand companies– SalesForce.com, Taleo, RightNow, Omniture, and HireRight, to name a few. When you are faced with a juggernaut, you have to adapt to its presence, which often requires tagging it as a way of identifying what you are dealing with– and so, cloud computing was born.

Bad rap or not – cloud computing is very much a movement that is forcing new opportunities and challenges onto enterprise users.

1) 82% of the market is frustrated with their existing search solution.

2) Catchphrases abound in this space. Attivio’s got “active intelligence.” Autonomy’s got “meaning based computing.” Every search vendor is clamoring for a slice of the enterprise search pie and working to differentiate themselves with words.

But put these two facts together, and it creates an interesting question: is there enough support behind the catchphrases to satisfy customers?

We learn what is most important in terms of product features and functionality from our customers:

- the need to easily navigate around their internal and external data for business decisions
- the need to scale to accommodate large volumes of unstructured and structured data
- the need to customize the way data is archived and tapped into to allow for maximized search efficiency

Especially as the cloud becomes more and more integral to data search, to stay relevant and up-to-speed, search vendors will need to develop search platforms that can scale for these new paradigms of information access. Kenneth Oestriech put it best <http://cloudcomputing.sys-con.com/node/652486> when he posted on his blog about external cloud-like architectures making their way into the enterprise and revolutionizing the way that the internal IT infrastructure is handled.

Given the level of dissatisfaction that seems to plague the market, and given that leading organizations want search tools that offer greater scalability and performance to reflect both the changes in how data is stored and the sheer amount of data that exists within most enterprises, it seems that all enterprise search vendors should think carefully about the legs on which their catchphrases stand. Does the product that holds up the image reflect changes in the market and the needs of customers?

Just a little food for thought…

“For the third consecutive year, Exalead’s solution has impressed us with its rapid and efficient search and by its precise refinement of results. Their innovative search approach is a real added value for their clients and makes our experts take notice, watching them year after year,” comments Hugh McKellar, editor-in-chief of KMWorld.

Our motivation is at a maximum during the fall, which is full of new product announcements and events.

To see the complete list of “KMWorld’s Trend-Setting Products of the year,” Click here.

Beyond all the hype and the predictions, it is interesting to take a step back and ask exactly what semantic search is and what it means for the enterprise. Are we already using semantic technologies today? What does it change for me as a search engine user?

This post introduces a series of articles on semantic search and its applications in Exalead’s technology. Before we venture further, let’s begin with a very basic explanation what semantic search is.

“Semantic search” is information search in which a search engine better understands (or at least behaves as though it better understands) what you are really looking for when you type a search query.

It is made possible by “natural language processing” technologies, which seek to teach computers to understand and use language the way we human beings do, that is to say to be able to read, write, analyze or speak human languages like English or Spanish.

Semantic search, therefore, goes beyond mere keywords to leverage context and collective knowledge for a clearer understanding of your request (just as we do when communicating with each other). This is true whether the semantic search engine seeks to understand natural language sentences like “I’m looking for a good book on gardening,” or simply to better understand the range of possibilities and connections embedded in a keyword request like “sales presentation.”

Semantic search on the Web, when presented in a format that’s neither overwhelming nor overly restrictive, certainly provides a more enjoyable, more fruitful search process for the general public.

For businesses, semantic search offers a significant ROI by enabling faster, more accurate information retrieval, and by fostering creative discovery and collaboration (with connections between people and data being an integral part of semantics-based navigational systems for business).

All applications critically relying on that accuracy, like compliance and e-discovery or Business Intelligence and Information Lifecycle Management, are leveraged by the ability of the search engine to deliver the right information, even when the user has difficulty formulating what he or she is looking for.

In short, semantic search is a way to add value to your data and capitalize on your company’s knowledge and information.

The next post will extend these thoughts and present the additional features and functionalities brought by semantic technologies in a search engine, how it enriches your experience as a user and eventually benefits the company as a whole, enhancing opportunities and competitive advantages.

Reference: Exalead one:search Semantics White Paper