'Intelligent' quick search?

[Vampyre]

Thanks for the good suggestions @Vampyre. Can I share the current code with you just to review as the search is quite complicated (in my eyes anyway!). Not expecting you to make the changes.

Of course, Peter. Email or message me and I'll take a look.

[AndyC]

How about a UNIONised query with exact matches coming first, followed by the fuzzy results, and giving them a "weight". In SQL Server this would be something like:

SELECT *, 1 AS weight FROM Title WHERE title_name = 'Jetpac'
UNION
SELECT *, 2 AS weight FROM Title WHERE title_name LIKE 'Jetpac%'
UNION
SELECT *, 3 AS weight FROM Title WHERE title_name LIKE '%Jetpac%'
ORDER BY weight, title_name

Very simplified I know - there's Publisher names got to be added etc, but it would return results with an exact match first, then anything that begins 'Jetpac' then anything with 'Jetpac' contained in it. In SQL Server UNION removes any duplicates (UNION ALL would keep the dupes) and I'm presuming MySQL has similar.

A search for 'Elite', that someone mentioned, would bring that game back first, then (probably) anything by Elite Systems then mop up anything that's got Elite somewhere in it.

You can do that better in SQL Server with a CASE in the order by clause rather than a union, i.e.

Order By
Case When Title = @search Then 1 Else 0 Desc,
Case When Title Like '%' +@search+ '%' Then 1 Else 0 Desc

...etc ...

Other SQL variants usually support the same or similar syntax for handling better sorting.

[4thRock]

We can use the sorting power of Google's algorithm and limit the results to Spectrum Computing.
Just google "renegade spectrumcomputing.co.uk" or "elite spectrumcomputing.co.uk"
You should get the original releases as the first results.

[PeterJ]

Excellent suggestion [mention]4thRock[/mention]!

[clebin]

The way to do it properly, longer-term, would be to install an open-source search engine like Elasticsearch and index everything into it. You can then do a full document search and apply weighting to individual fields like title.

You could implement the entire search through this, but I find writing Query DSL can get a bit convoluted once you introduce filters and whatnot. Instead you could just use it for the 'simple search' and fall back to a direct database search for everything else.

[Einar Saukas]

The way to do it properly, longer-term, would be to install an open-source search engine like Elasticsearch and index everything into it. You can then do a full document search and apply weighting to individual fields like title.

A ZXDB-based site using Elasticsearch already exists, it's called ZXInfo.

[PeterJ]

Thank you [mention]clebin[/mention] and [mention]Einar Saukas[/mention].

Yes, the Quick Search implementation over at zxinfo is excellent

Unfortunately I have no experience in Elasticseach. Anyone interested in contributing their time?

[PeterJ]

Following millions of votes (well, actually just 10...) in the poll about search order 9/10 people expressed a wish to go back to just ordering by title. Sorry [mention]Morkin[/mention], and its back how you like it [mention]8BitAG[/mention].

Just to put things into perspective, searching for Elite in the quick search brings the original game in at result number five, and searching for JetPac brings the original game as result number six. Both results are on the first page.

[8BitAG]

Please don't continue to tag me in this sort of way. I've already said I wasn't going to continue to participate in this discussion. I'm glad you've had some really good suggestions of how to achieve an exact match search as the first result, which is the sort of result I'd expect from a quick search box.

[PeterJ]

Apologies (again!), just thought you would want to know the search results ordering had gone back to how you wanted.

Please just remember that quick search is supposed to be the broadest possible search, as the box text says (Search by title, author publisher). So the idea of the exact match doesn't fit the purpose of that box.

I think the idea from [mention]Vampyre[/mention] is excellent for implementing in the Title search box and we can look into that. That would give you exact match first.

For most searches the advanced search is much better (Although because of the alpha search your results again may not appear as the first result).

I think ElasticSearch is probably a step to far, although I appreciate the suggestion.

[clebin]

Thank you @clebin and @Einar Saukas.

Yes, the Quick Search implementation over at zxinfo is excellent

Unfortunately I have no experience in Elasticseach. Anyone interested in contributing their time?

I'm busy with a project in my spare time at the moment but if it's still useful in a year or so, I'll be happy to help. I'm a PHP developer and I've implemented Elasticsearch in my work. Sorry I can't be of more help right now.

[Morkin]

Following millions of votes (well, actually just 10...) in the poll about search order 9/10 people expressed a wish to go back to just ordering by title. Sorry @Morkin, and its back how you like it @8BitAG.

Just to put things into perspective, searching for Elite in the quick search brings the original game in at result number five, and searching for JetPac brings the original game as result number six. Both results are on the first page.

No worries, appreciate looking into it... The advanced search is right on the front page, and since the option of searching for "all games" has appeared I've found it works fine.

One reason I was interested in what people were entering in the quick search box was to see for example if everyone was really only using it for games, or if people were looking for other stuff (like software houses, people etc.).

[PeterJ]

No problem.[mention]Morkin[/mention]. I will get the inputs to the quicksearch box directed to a text file and we can review them

[Morkin]

No problem.@Morkin. I will get the inputs to the quicksearch box directed to a text file and we can review them

"Search results for quick search"

Code: Select all

zythum by mirrorsoft
zythum
sam fox strip poker
zythum
stonk
skyrim
pac man

[kolbeck]

The way to do it properly, longer-term, would be to install an open-source search engine like Elasticsearch and index everything into it. You can then do a full document search and apply weighting to individual fields like title.

A ZXDB-based site using Elasticsearch already exists, it's called ZXInfo.

But using Elasticsearch does not solve the initial problem. For example, on ZXInfo searching for Jetpac - gives by coincidence the "real" Jetpac as first result - https://zxinfo.dk/search/jetpac

Searching for Manic Miner, the "real" Manic Miner ends up in 18th place. - https://zxinfo.dk/search/manic%20miner -

there's no way of knowing what's the real deal, unless someone manually put it into the engine. And last, sorting just makes this worse

(FYI - ZXInfo sorts search results, based in scoring by Elasticsearch - which more or less is an indication of how good the fit was)

/T

[PeterJ]

Thank you [mention]kolbeck[/mention]. You explain the complexities of search very well. That's why people are paid huge amounts of money to work on it.

[4thRock]

Please just remember that quick search is supposed to be the broadest possible search, as the box text says (Search by title, author publisher). So the idea of the exact match doesn't fit the purpose of that box.

Yes, that was the original idea but perhaps it too generic now. We have so much more information than when we started, so search results are more complex and unpredictable.

[4thRock]

... to see for example if everyone was really only using it for games, or if people were looking for other stuff (like software houses, people etc.).

If 90% of the searches are for titles (games) it kind of solves the problem I mentioned above.
If we limit results to titles, it would be easier to sort the results.
But we need more data to know that. I'd say 2 or 3 months worth of searches.

[kolbeck]

Thank you @kolbeck. You explain the complexities of search very well. That's why people are paid huge amounts of money to work on it.

It’s a well known real-world problem
When people at work asks me what it takes to make a search like google- they never really want to hear the answer

/T

[Morkin]

It’s a well known real-world problem
When people at work asks me what it takes to make a search like google- they never really want to hear the answer

/T

...Basically move up from the search box and type in your browser bar instead...

[clebin]

But using Elasticsearch does not solve the initial problem. For example, on ZXInfo searching for Jetpac - gives by coincidence the "real" Jetpac as first result - https://zxinfo.dk/search/jetpac

Searching for Manic Miner, the "real" Manic Miner ends up in 18th place. - https://zxinfo.dk/search/manic%20miner -

there's no way of knowing what's the real deal, unless someone manually put it into the engine. And last, sorting just makes this worse

(FYI - ZXInfo sorts search results, based in scoring by Elasticsearch - which more or less is an indication of how good the fit was)

/T

You can tailor your queries to deal with such situations. In most cases, adding a little weight to the title will result in decent search results. That falls apart with "Manic Miner" but you could fix that with an additional boost for an exact match in the title field. That would ensure the original appears above "Manic Miner Blah Blah Edition".

Another option would be to apply a small negative score to that phrase (or similar) appearing in the 'Mod' field, so that all mods & clones are pushed down the search results. That would help in situations like, for example, the user searches for "Jet Set Willy" and gets Jet Set Willy first as expected, but Jet Set Willy II appears beneath a load of clones & mods because the 'match_phrase' boost hasn't kicked in. Or because someone wrote "Jetset Willy".

Elasticsearch isn't a black box, it just requires a little massaging and knowledge of the data to return the results you want.

[PeterJ]

Great that we come above that other Spectrum site! My monthly payment to Sundar Pichai helps.

[clebin]

Great that we come above that other Spectrum site!

Involuntarily, my brain inserted an extra 'h' into that sentence!

[Morkin]

...Don't get too excited...

[PeterJ]

Thanks for raining on my parade [mention]Morkin[/mention]

If you put ZX Spectrum in the search like you did with Jetpac we will probably be above them.

If doing serious checks on search positions I believe it is best to sign out of Google (if using Chrome), and clear your cache and search history