Thursday, December 22, 2011

Machines are becoming intelligent, I am fooled by information overload, Hello Search engine designers please help me!


Today, five billion people have cell phones. Two billion people using the Internet. That is, too many devices, too many people produce/consume information. The ever increasing amount of information available to us is mind-blowing. According to EMC, there are presently 800,000 petabytes (one million GB) in data storage. And according to the University of California, American homes consume nearly 3.6 zettabytes (one million petabytes = one zettabyte) of information per day. It’s expected to grow, EMC expects a 44-fold increase in data storage by 2020.

If I turn around, Data/Information is all around me. I am stuck on the information overload. My email inbox, text messages, Social network feeds, RSS feeds, updates, ebooks, notes, todo's etc My brain throwing me in a infinite for loop where I am not able to focus on a given task at hand . The more the information its hard for brain to manage it but its good for machines to learn better if it gets more information. I am becoming an information-consumption machine.

Our brain has lot of limitation, one of the important one is that its not scalable. Due to physical limitation of the brain - the storage, processing(Understanding, interpreting etc), Limited input/output etc. But this problem is solved in computing. But what computing needs to solve is an ever increasing information overload. What I need is a full-fledged search engine.

- to search/recommend my private information (such as my desktop, email, sms, my comments, tweets etc)
- to search/recommend my protected information (All my friends/relatives/work information)
- to search/recommend publicly available huge amount of information

Current search engines allows to search mainly the last point, if I have luck, I may get right information on the first SERP.

I think there are two ways we can get high quality, personalized information depends on the task at hand. One is “filtering”, current search engines allowing me to filter the information by words, web sites, stop words, etc. next generation search allows to filter by concepts (an infinite number of database fields). The second one is joining, currently no search allows to join information silos.

I believe that our brain has ability to store/process knowledge (not raw data nor information) this is the main difference between us and other animals. One or more (may be all are one way or other joined) ontologies are developed when we learn from childhood. Everyone has their own ontology, There is no global common ontology.

In order to design better search engines, one has to understand human brain and how it retrieve information for the given query. Secondly people are coming to search engine not to simply search, they have a task at hand, they want to complete it. In other words

Task at hand = plan -> search -> Analise -> Make Decisions

A full-fledged search engine is a hybrid engine that I envision should allows me to plan, this is an important activity, to match my ontology and machine ontology, in other words this will help to set up the context of search. Secondly to search by concept filters and joins datasets on the fly. Thirdly visually analyze the result to quickly get the picture. Finally corresponding statistical information must be included with result to make best decision. All these engines – strategy engine(plan), search engine(search), visual engine (analyze), Prediction engine (decision making) should be integrated on social platform.

Since we are in hurry, Current search engine allows to type few words and returning results and we make decision based on the algorithm.
Who will invest time on planning, analyzing and decision making? Since current search engines already taught us how to search by hurry. If all the above tasks are automated, users will understand and can spend few seconds on planning, analyzing and decision making process to get quality results, hence better decisions and make their life better and healthy.

1 comment:

Senth.M said...

I agree with your point, information overload always a hectic to get into a final desired result.
Am looking forward to a best idea-focused search engine, not only for search results....