Predictive Search

Broken link? Feature request? Anything related to the Spectrum Computing website here.
User avatar
ZXDunny
Manic Miner
Posts: 237
Joined: Tue Nov 14, 2017 3:45 pm

Re: Predictive Search

Post by ZXDunny » Wed Jul 17, 2019 11:25 am

RMartins wrote:
Wed Jul 17, 2019 10:42 am
ZXDunny wrote:
Wed Jul 17, 2019 10:22 am
PeterJ wrote:
Wed Jul 17, 2019 9:39 am


I had to CTRL + F5 then reboot on W10.
No effect here, I've cleared cookies, refreshed hard with CTRL+F5 and rebooted Win10.

Still no predictive search.
Did you notice that you need to use a different URL ?
https://spectrumcomputing.co.uk/index_new.php
Aha! That was the problem - I was just clicking on the banner at the top of the forum :)

Feel a bit of a pratt now!
0 x

User avatar
PeterJ
Site Admin
Posts: 1306
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: Predictive Search

Post by PeterJ » Wed Jul 17, 2019 12:39 pm

No worries @ZXDunny

Just glad you are now able to test.
0 x

User avatar
ZXDunny
Manic Miner
Posts: 237
Joined: Tue Nov 14, 2017 3:45 pm

Re: Predictive Search

Post by ZXDunny » Wed Jul 17, 2019 1:30 pm

Yeah, it works. It's a bit weird typing in "Cyb" and getting results that have "cyb" somewhere inside them, rather than starting with it. We may need to add in some modifiers for the user to specify where the results need to match.
0 x

User avatar
RMartins
Manic Miner
Posts: 391
Joined: Thu Nov 16, 2017 3:26 pm

Re: Predictive Search

Post by RMartins » Wed Jul 17, 2019 1:41 pm

ZXDunny wrote:
Wed Jul 17, 2019 1:30 pm
Yeah, it works. It's a bit weird typing in "Cyb" and getting results that have "cyb" somewhere inside them, rather than starting with it. We may need to add in some modifiers for the user to specify where the results need to match.
That's the current behaviour, of the search_by tables.

"Search" is a big subject. I know because I was responsible (tech lead) for web platform of "Yellow Pages" in my country, for about 3 years.
Doing search properly, requires a very different approach than using SQL and is something that takes constant effort to maintain.

Here the approach being followed, is to have the best we can just using SQL, so some limitations apply, but we will try to improve whenever we can.

Some things still need to be debated, taking into account, what features we would like to have and what can be done with an SQL only solution.
1 x

User avatar
RMartins
Manic Miner
Posts: 391
Joined: Thu Nov 16, 2017 3:26 pm

Re: Predictive Search

Post by RMartins » Wed Jul 17, 2019 9:42 pm

To the ones, that keep an eye on the Tab Icons, there is a proposal for a new one on this new page.

It was based on the site Logo, but its design was adapted to fit the size restrictions of a favicon.
Previous favicon was just a rescale of the site logo, and becomes unreadable and blurry at smaller versions.

NOTE: It might not show up on every browser, since there are a lot of settings for the favicon, and several formats, so some might still be missing.
0 x

User avatar
Metalbrain
Dizzy
Posts: 80
Joined: Thu Feb 15, 2018 2:14 pm

Re: Predictive Search

Post by Metalbrain » Thu Jul 18, 2019 8:07 am

Tested in several browsers, it's working on all of them.

Regarding the Sir Fred problem, it can be found if we skip the space (like looking for sirfr). Looking just for "Sir ", we only get these results:
Sir Clive
Sir Clive and Mr. ZX
Sir Computers

and looking for "sir " in lowercase, we get nothing
0 x

User avatar
RMartins
Manic Miner
Posts: 391
Joined: Thu Nov 16, 2017 3:26 pm

Re: Predictive Search

Post by RMartins » Thu Jul 18, 2019 2:10 pm

"Sir Fred", or any other Title that is searched using spaces, will not easily find a match.

And the reason for this, is that search tables, effectively remove the spaces in the titles.
... replace( lower(title),' ','' ),

I'm not sure what was the reason/motivation behind removing the spaces, but I'm sure there must have been one.

The problem with removing spaces is that then it becomes possible to find text formed by concatenating 2 previous words.
For example, we can find "Green Beret" by searching for "nbe", which is not what users expect.

Another example, is that searching for "green beret" doesn't return any result.
But searching for "greenberet" will get you a result for "Green Beret".


From a usability stand point, humans typically search for words or parts of words, they never expect to find substrings of concatenations of words.


Another implicit usability rule, is that if users are very specific about something, is because they are passing an intent.
Examples are Upper case letters or special chars like "ñ".


So, if user uses Upper case in the search string, then it means they are looking for that specific case, for example to distinguish the start of a Word or a Name. But if the user uses all lower case, then it means that any match is acceptable including in the middle of a word or name.

If user uses special chars, it means they explicitly want to find that specific accentuated char. Searching for "aña" means that the user wants matches like "Cañas", but not "Anagrams" or "Zemana", but we currently get those.
However, if the user searches for "ana", We expect all of these to be valid hits: "Cañas", "Anagrams" or "Zemana".

The good news, is that all of the above, can be done using just SQL, a pré-processed search table and collation rules.

These are all things, that need some debate, to find the best compromise for an SQL only solution.
Last edited by RMartins on Thu Jul 18, 2019 3:15 pm, edited 1 time in total.
0 x

User avatar
RMartins
Manic Miner
Posts: 391
Joined: Thu Nov 16, 2017 3:26 pm

Re: Predictive Search

Post by RMartins » Thu Jul 18, 2019 2:18 pm

On another note, take into account, that the autocomplete feature, can not show a huge amount of results, so the result is being limited to the first 10 entries, listed in alphabetical ascending order.

If the user does not find the result he expects, he needs to drill down into a more specific content, by continuing to write a larger search string.

Also remember, that quick search, includes: Titles, Aliases, Authors and Publishers.

So, 10 of each are gathered and sorted individually, and then sorted as a whole (this guarantees the results are deterministic for the same query) and only the first 10 are shown (of potentially 40 results).

If you are unlucky, it can be 10 of the same type: 10 titles or 10 Authors, etc... so you might need to drill down until you get to the sweet spot, where what you are searching for, actually shows up in that 10 slot window.
0 x

User avatar
Einar Saukas
Manic Miner
Posts: 937
Joined: Wed Nov 15, 2017 2:48 pm

Re: Predictive Search

Post by Einar Saukas » Thu Jul 18, 2019 2:37 pm

RMartins wrote:
Thu Jul 18, 2019 2:10 pm
I'm not sure what was the reason/motivation behind removing the spaces, but I'm sure there must have been one.
The motivation is that users searching for either "Knight Lore" or "Knightlore" will work in both cases. Searching for either "Chop Lifter" or "Choplifter" will also work in both cases. Incidentally the exact spellings are "Knight Lore" and "Choplifter", but nobody needs to know that.

Ignoring special characters has the same motivation. Otherwise nobody would ever find "Cadàveriön". And it has to work both ways, otherwise searching for proper spellings like "iniciación" would not find anything, because most foreign titles imported from Martijn's WoS were missing accentuation (we are gradually fixing it but it's going to take some time).

Thus basically "search_by" tables were created to help searches work as good as possible. They were never intended to help autocomplete, and I doubt there's a single solution that would work for both. You may need to create your own auxiliary tables for that.
0 x

User avatar
RMartins
Manic Miner
Posts: 391
Joined: Thu Nov 16, 2017 3:26 pm

Re: Predictive Search

Post by RMartins » Thu Jul 18, 2019 3:05 pm

Einar Saukas wrote:
Thu Jul 18, 2019 2:37 pm
RMartins wrote:
Thu Jul 18, 2019 2:10 pm
I'm not sure what was the reason/motivation behind removing the spaces, but I'm sure there must have been one.
The motivation is that users searching for either "Knight Lore" or "Knightlore" will work in both cases. Searching for either "Chop Lifter" or "Choplifter" will also work in both cases. Incidentally the exact spellings are "Knight Lore" and "Choplifter", but nobody needs to know that.
And I have nothing against that.
But that is why search is a hard topic.

For these kinds of wanted matches, a specific extra entry should be created for it, instead of just clipping the space on the existing one.
What I mean is the search table should have both spellings (or how many that title is usually searched for).

This is why I said, that a well fined tune search engine, takes time and effort to maintain.
Einar Saukas wrote:
Thu Jul 18, 2019 2:37 pm
Ignoring special characters has the same motivation. Otherwise nobody would ever find "Cadàveriön". And it has to work both ways, otherwise searching for proper spellings like "iniciación" would not find anything, because most foreign titles imported from Martijn's WoS were missing accentuation (we are gradually fixing it but it's going to take some time).
I also do not have anything against that.
However, it should not be "the only way" to search (case insensitive and accent insensitive).

For example, maybe add some entries in the "aliases" tables could be kept as search related aliases for titles.

NOTE: we can use a different COLLATION, on each query condition for example.

We can determine the collation to use, depending on the input search string for example (if user used upper case, or used special chars).
Look at my explanation on user intent. I can make an example later, if needed.
Einar Saukas wrote:
Thu Jul 18, 2019 2:37 pm
Thus basically "search_by" tables were created to help searches work as good as possible. They were never intended to help autocomplete, and I doubt there's a single solution that would work for both. You may need to create your own auxiliary tables for that.
What I'm proposing, is that we can in fact use the same system, and we probably should.
We just need to refine it a bit, to fit the intended user searches.

Any user would expect that if he can find something using quick search, to be able to find it on advanced search using the same text on the proper field.

But anyway, these are intended as ideas for debate.
Just trying to help out improve the search.
0 x

Post Reply