New WoS and ZXDB

[akeley]

What strikes me is how completely unnecessary all this actually is. All they had to do is say something to the tune of "ah, let the bygones be bygones, ZXDB is an awesome piece of work and we will use it from now on". Problem solved.

[Einar Saukas]

What strikes me is how completely unnecessary all this actually is. All they had to do is say something to the tune of "ah, let the bygones be bygones, ZXDB is an awesome piece of work and we will use it from now on". Problem solved.

Exactly. Or just claim "I really had a much better database (that nobody ever saw) but the dog ate it so let's use yours".

[Einar Saukas]

BTW about my last test, it was based on a list downloaded 2 days ago (June 16, 2020) from new WoS.

If we run the same test using a list downloaded yesterday or today, there will be 1 difference, because this comment was edited yesterday:

https://worldofspectrum.org/archive/sof ... rs-persson

This page says "Updated On: June 17, 2020". It seems they have now started to update new WoS manually.

This is even more bizarre. Between the stupidity of redoing years of work already done, or the immorality of using ZXDB in disguise while criticizing it... they have chosen to do BOTH?

[PeterJ]

Having read through all that Einar has posted, and links to various comments, this from the 6th June is the one that makes me smile the most...

[Pegaz]

Having read through all that Einar has posted, and links to various comments, this from the 6th June is the one that makes me smile the most...

"Always the two Sith Lords there are, a master and aprentice."

Only in this case, the aprentice is the older one...

[Einar Saukas]

Having read through all that Einar has posted, and links to various comments, this from the 6th June is the one that makes me smile the most...

It's interesting to read how much hard work they have done to fix thousands and thousands of errors. In particular, they mentioned fixing the order of certain releases (some re-releases were shown as released first, and vice-versa). For instance, it was recently mentioned here and here.

It turns out that's easy to check too!

Let's take a look at the same list of titles downloaded from new WoS. Columns "publishers" and "all_publishers" are just garbage (they are either empty or just contain the word "Array"). But it's possible to obtain original publisher, together with original title, from column "title_publisher".

This column is inconsistent for programs with unknown publishers. In Earthbound, this column contains "Earthbound ()", but in Spectrofon this column is empty. No problem, that's easy to handle. The tricky part is supporting games originally co-published by different publishers:

Code: Select all

select * from (
select e.id,e.title,group_concat(b.name order by p.publisher_seq separator ',') as pub 
from entries e left join publishers p on e.id = p.entry_id and p.release_seq = 0 
left join labels b on b.id = p.label_id group by e.id) as z 
inner join x_newwos x on z.id = x.old_id
where x.title_publisher <> concat(z.title,' (',coalesce(z.pub,''),')')
and not (title_publisher = '' and pub is null);

We will get 218 differences. That's a lot...

But wait! Do you remember I mentioned that the list of publishers in new WoS came mostly from old WoS, rest of game content came from ZXDB? It turns out a few publishers had slightly different names in old WoS. For instance "U.S. Gold Ltd" in ZXDB was "US Gold Ltd" in old WoS, "Unique Computer Services [1]" in ZXDB was "Unique [1]" in old WoS, etc. Let's try this query again, this time applying these conversions:

Code: Select all

select * from (
select e.id,e.title,group_concat(b.name order by p.publisher_seq separator ',') as pub 
from entries e left join publishers p on e.id = p.entry_id and p.release_seq = 0 
left join labels b on b.id = p.label_id group by e.id) as z 
inner join x_newwos x on z.id = x.old_id
where x.title_publisher <> replace(replace(replace(replace(replace(replace(replace(
concat(z.title,' (',coalesce(z.pub,''),')'),
'U.S. Gold Ltd','US Gold Ltd'),
'Unique Computer Services [1]','Unique [1]'),
'PowerSoftware','Powersoftware'),
'CheetahSoft Ltd','Cheetahsoft Ltd'),
'MmcM','MMCM'),
'(Raf,','(RAF,'),
'(Raf)','(RAF)')
and not (title_publisher = '' and pub is null);

Now we will get only 14 differences in original publishers. That's better...

But wait! What if they messed up in the order of certain co-publishers when extracting this information from ZXDB? Even co-publishers of the same release were stored in a specific order in old WoS, but perhaps they didn't know it. Let's try to keep exactly the same original co-publishers, and just switch the order they are listed:

Code: Select all

select * from (
select e.id,e.title,group_concat(b.name order by p.publisher_seq separator ',') as pub 
from entries e left join publishers p on e.id = p.entry_id and p.release_seq = 0 
left join labels b on b.id = p.label_id group by e.id) as z 
inner join x_newwos x on z.id = x.old_id
where x.title_publisher <> replace(replace(replace(replace(replace(replace(replace(
replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(replace(
replace(replace(replace(concat(z.title,' (',coalesce(z.pub,''),')'),
'U.S. Gold Ltd','US Gold Ltd'),
'Unique Computer Services [1]','Unique [1]'),
'PowerSoftware','Powersoftware'),
'CheetahSoft Ltd','Cheetahsoft Ltd'),
'MmcM','MMCM'),
'(Raf,','(RAF,'),
'(Raf)','(RAF)'),
'CityAceE,AlexUzer,Grand','AlexUzer,CityAceE,Grand'),
'Digital Prawn,Einar Saukas,Dr BEEP','Einar Saukas,Digital Prawn,Dr BEEP'),
'Einar Saukas,Dr BEEP,Digital Prawn','Einar Saukas,Digital Prawn,Dr BEEP'),
'Infotek Team,Alliance Team,Optimal Group','Alliance Team,Infotek Team,Optimal Group'),
'J. Korytkowski,A.J. Lipski,A. Ferenc','A. Ferenc,A.J. Lipski,J. Korytkowski'),
'Kiwi,RetroCoder,Sadako,Lee_Dc','Kiwi,Lee_Dc,RetroCoder,Sadako'),
'Marek Radkowski,Janusz Zietal,Jarek Druzbinski','Janusz Zietal,Jarek Druzbinski,Marek Radkowski'),
'Mark Woodmass,Paul Harrison,Dean Hickingbottom','Dean Hickingbottom,Mark Woodmass,Paul Harrison'),
'O. Hohlov,K. Afendikov,V. Rubtsov','K. Afendikov,O. Hohlov,V. Rubtsov'),
'Richard Swann,Miles Kinloch,ewgf','Miles Kinloch,Richard Swann,ewgf'),
'Simbols,Milytia,Triebkraft','Milytia,Simbols,Triebkraft'),
'Tankard,Hammer,Kamikaze','Hammer,Kamikaze,Tankard'),
'Triebkraft,4th Dimension [2],bfox','4th Dimension [2],Triebkraft,bfox'),
'Tygrys,Cat-Man,Voyager','Cat-Man,Tygrys,Voyager')
and not (title_publisher = '' and pub is null);

Bingo! Absolutely all 24368 titles have exactly the same title and original publishers in both new WoS and ZXDB 2018. Yet another fantastic coincidence.

[Mike Davies]

It turns out new WoS is using ZXDB! More precisely ZXDB version 1.08 released on September 2, 2018.

This is a remarkable piece of investigative work. Exemplary.

I guess WoS now needs to acknowledge that it's powered by ZXDB, like every site using data from it. If that can be done in good faith - to acknowledge that the many datafixes and improvements in data quality -- that's going to be beneficial to the community as a whole. ZXDB is now the primary source of archival data on ZX Spectrum games.

Congratulations, Einar, you have succeeded in quite the impossible task, with your infinite patience and humility.

[namco]

Essentially this boils down to:

Once a c*nt, always a c*nt

I noticed the change over earlier in the week and wondered how long before someone would mention it here.

I've been having a look at it on and off through the week and was met with slowdown to load the site, 500 server errors, lack of new games (already mentioned here and it's unsurprising) and ajax-type loading on the db inputs being occasionally slow.

WoS as a website has lost its charm.

As I mentioned on Twitter:
End of an era.

Additionally, my Spectrum YT videos used to reference WoS via link (apart from the newer games) as I didn't have an issue with the site itself - nostalgia. They are from now, and in the future, referencing this site.

[edited last line for clarification purposes]

[ketmar]

offtopic ;-)

Spoiler

i really want to say "thank you alot!" to Lee and others in WoS website team. because i found this site (and ZXDB) only after they rolled up "new WoS", which can't even work with javascript turned off. really great work there. i stoped caring about WoS forums after they switched the engine (their new forum engine is awful), and only used WoS as a games database. (tbh, i was never an active forum participant at the first place.) and when they made the site unusable for me (i'm browsing with JS off), i finally decided to search for a new home. and found ZXDB and Spectrum Computing. with alot of names i remember and respect here. yay. it somewhat renewed my interest in Speccy (you can see that i made quite alot of bugfix commits to Zymosis after 4+ year hiatus, for example).

so, thanks to WoS team for breaking WoS so bad that i had to look for a new home! and thank you all for ZXDB, Spectrum Computing, and all other Speccy-related work!

[RMartins]

I'm lost for words, to even comment this.

I just wish I could have access to my account to fix my threads in WOS to have the image links corrected.
But my account is still locked.

Apparently, they don't even value the content they have in there.

[djnzx48]

BTW about my last test, it was based on a list downloaded 2 days ago (June 16, 2020) from new WoS.

If we run the same test using a list downloaded yesterday or today, there will be 1 difference, because this comment was edited yesterday:

https://worldofspectrum.org/archive/sof ... rs-persson

This page says "Updated On: June 17, 2020". It seems they have now started to update new WoS manually.

This is even more bizarre. Between the stupidity of redoing years of work already done, or the immorality of using ZXDB in disguise while criticizing it... they have chosen to do BOTH?

I see they have a 'Change Request' form on each page for people to suggest corrections. Perhaps that was just a one-off addition from someone visiting the site?

[moroz1999]

I see they have a 'Change Request' form on each page for people to suggest corrections. Perhaps that was just a one-off addition from someone visiting the site?

Yep, this actually works. I've used it yesterday to suggest a country change for myself, and admin have changed it.

[cha05e90]

Yep, this actually works. I've used it yesterday to suggest a country change for myself, and admin have changed it.

Ah, ok. That makes sense. So they want us to bring WoS DB to ZXDB 2020 state. May last some months/years, but it's not impossible. Especially when you copy and paste stuff from Spectrum Computing to WoS' change requests...

[Vampyre]

Yep, this actually works. I've used it yesterday to suggest a country change for myself, and admin have changed it.

Ah, ok. That makes sense. So they want us to bring WoS DB to ZXDB 2020 state. May last some months/years, but it's not impossible. Especially when you copy and paste stuff from Spectrum Computing to WoS' change requests...

Yup, like you say, certainly not impossible. The only thing that makes it unlikely for me is Einar mentioning the new WOS IDs he added that suddenly appeared in new WOS (and I take it have now disappeared?). The likelihood of adding these, in the exact same order Einar has to generate the same ID, is highly unlikely without an import. Plus - given the comments made by WOS admins in regards to ZXDB over the years - why would they give a damn what ID's ZXDB is using? So I think this data is an import - which once you know the structures of both databases shouldn't be that hard to do, given time.

[ketmar]

i wonder if WoS will start adding new software in random order now. and will they blame ZXDB for being "desynced" then?

their stupid ego games only makes things worse for everyone. i still hope that they will come to their senses, and will simply import the whole ZXDB (with proper credits). a snowball's chance in hell, i know. but i still hope.

[PeterJ]

i still hope that they will come to their senses, and will simply import the whole ZXDB (with proper credits).

100% agree with that.

[Pegaz]

i wonder if WoS will start adding new software in random order now. and will they blame ZXDB for being "desynced" then?

their stupid ego games only makes things worse for everyone. i still hope that they will come to their senses, and will simply import the whole ZXDB (with proper credits). a snowball's chance in hell, i know. but i still hope.

We can hope, but not too much.
It’s been a whole day since Einar posted his findings on the wos forum and there’s absolutely no response, just complete ignoring.
Obviously Rorthron was absolutely right, when he predicted something like this.
This is extremely shameful and dishonorable behavior, but I can't say I'm surprised.
After all, this silence is also the answer, loud and clear...

[Ralf]

i still hope that they will come to their senses, and will simply import the whole ZXDB (with proper credits).

Yes, I suppose now that their plan was to add content of ZXDB partially and in random order with some slight changes to pretend they are doing it all themselves from the scratch.

I wouldn't care about it at all but they also made this "Change Request" form so people could submit their corrections. So I guess some well intentioned people not realising what's going on may start to send them some submission and corrections.

And this way we are going to have two ZX Spectrum databases comming from the common source being developed independently. This is going to be a nightmare

[hitm4n]

If this were me, i would write off WOS as dead in the water (its what i did when SpectrumComputing started). What can it give us thats not provided by SpectrumComputing? (I would like to know, seriously.)

I would re-write the "acceptible use policies" for the database usage and specifically deny all use by Lee Fogarty and the WOS domain. You lot are all being very diplomatic, but this sh*t with Lee has gone on for years and this is what it adds up to? This has to stop, he's taking the absolute P?s?...

I'm VERY interested in what he says about this.

[Ast A. Moore]

What can it give us thats not provided by SpectrumComputing? (I would like to know, seriously.)

Tons of historical and useful technical information on its forums. It’s a goldmine.

[hitm4n]

Genuine question. That is fine, a good point. How does that require the ZXDB data ?

[ketmar]

I would re-write the "acceptible use policies" for the database usage and specifically deny all use by Lee Fogarty and the WOS domain.

that won't stop 'em, tho. there are rules for using ZXDB already, and those rules weren't respected. you can add more rules, but it will be easy to ignore them regardless of their number. ;-)

besides, this will turn ZXDB into proprietary database. you can have either "free for anyone" (respecting the license, of course), or "proprietary", there is no middle ground there.

if anything, such clause will only make things worse. i don't believe that anybody here wants a war with WoS. quite the contrary, we want this nonsense to stop (sorry for talking for other people, but i think i'm right here ;-). Einar did everything he could to make this happen. the ball is in WoS team ballpark now.

[djnzx48]

I would re-write the "acceptible use policies" for the database usage and specifically deny all use by Lee Fogarty and the WOS domain.

That doesn't sound like a good idea at all, and it would go against the entire 'open' nature of the database.

This situation reminds me of what happened with Wikia years ago. Despite a whole lot of wikis migrating from there after it became really obnoxious, it's always the ad-ridden Wikia articles that appear at the top of search results. Since their policy didn't allow the migrated-from wikis to be deleted, they stick around and continue to get updated by people who don't know any better.

[PeterJ]

Genuine question. That is fine, a good point. How does that require the ZXDB data ?

[mention]Ast A. Moore[/mention] is right about the WoS forum with posts way back into the past which are useful and of historical interest. The forum as you say [mention]hitm4n[/mention] does not have anything to do with ZXDB. We can split them into two separate entities.

[PeterJ]

I don't believe that anybody here wants a war with WoS. quite the contrary, we want this nonsense to stop (sorry for talking for other people, but i think i'm right here . Einar did everything he could to make this happen. the ball is in WoS team ballpark now.

You are completely correct there.