I want a search engine for explore

A surprisingly large amount of my effort is dedicated to something that looks like search. Sometimes this leans more towards an active process where I’m looking for something specific - as in dating or job hunting, other times it’s more trial and error where all I know is what ‘feels right’ - as in making friends at school, or searching for meaning.

I always wish I can be better at various aspects of these processes: when looking for something specific, I want to find the path that would take me there the soonest; however, if I don’t know what I’m looking for, or only have a partial clue, I want to find out what the remaining clues are, so to speak. Life does often reveal these clues naturally over time, serendipitously, and as the Chinese like to say, many things are 可遇不可求 (you may chance upon, but you can’t seek it out). But I get impatient and often want to fast track the discovery, and the beautiful thing about search is that it lets you seek out things you could otherwise only chance upon.

There’s a lot of cross flow between the two sides, such as racing towards a goal and realizing it doesn’t ‘feel right’, or seeing the journey as the destination. Over time I’ve looked at both sides through a few different lenses: exploit vs explore, modeling the external world vs my internal world, deep learning vs RL. I realized there’s a lot more tools to support the ’exploit’ side (i.e. Google), and I want more tools to support the ’explore’ side.

Building a tool to optimize for exploration feels a bit antithetical and counterintuitive, but I think that’s because no good one exists. Recommendation engines come somewhat close: let’s say you just found a clothing brand you liked, its style is a bit more specific than something like ‘Y2K’, you have a hard time finding other brands in this style with Google, but simply liking a few Instagram posts from the brand and the ad algo will send you an infinite supply of alternatives. In this example, the quality ceiling of the rec system is quite low, it will never beat a well curated store for the given style, but the implicit signal from the social graph still works much better than search engines.

I believe this is why listicles still exist on Google, because a human-curated ‘10 best Sydney restaurants for a date night’ beats the top 10 non listicle Google results. However, if you’re looking for ‘best Sydney restaurants that follow Fergus Henderson’s nose to tail dining philosophy’ you’re pretty much out of luck because no sane person will write such a listicle (as much as I hope someone would).

I want my search engine to be a bit bolder, explore more, and be more opinionated. Googling ‘best thing for a young ambitious technically competent person to pursue’ should not return a medium article urging the reader to ’never stop learning’ and ’take the initiative’, instead it should return a diverse set of results, from open problems in ML and biology to how to become a successful high frequency trader to high impact public policy areas.

Perhaps one way to frame the gap is that current search engines feel like they’re serving the result that best satisfies the ‘average’ user, and this hypothetical human might not want to read about in-ovo sexing, but an ambitious technically competent young person sure might.

Another way to frame it is I want a search engine without mode collapse.

P.S. I really want LLMs with similar properties. Models have the additional advantage of being able to act in the world (such as asking me more clarifying questions). I wish they could truly hold multiple opinions as equally likely for a substantial amount of time.