New WoS and ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Pobulous
Dizzy
Posts: 58
Joined: Wed Nov 15, 2017 12:51 pm

Re: New WoS and ZXDB

Post by Pobulous »

No-one has posted this, yet, from the WOS facebook group: This was buried in a thread asking why WOS error pages were leaking browser and location data from other users.
Lee Fogarty
I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM.
If it's simple to resolve via a PM, why has Lee not resolved it via a PM?
1 x

User avatar
ketmar
Microbot
Posts: 167
Joined: Tue Jun 16, 2020 5:25 pm

Re: New WoS and ZXDB

Post by ketmar »

and not only with the same data, but with the same comments that wasn't present in the original db, and even with the same infoseek ids for things that weren't in the original db! this is a pure coincidence, of course.
0 x

User avatar
Mike Davies
Microbot
Posts: 137
Joined: Mon Nov 13, 2017 10:11 am

Re: New WoS and ZXDB

Post by Mike Davies »

Lee Fogarty
I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM.
And thus begins the attempt to gaslight people. Notice he doesn't cover any of the substance in Einar's statements. Just brush it off. The data examples Einar has shown did not exist in the WoS datafiles ZXDB was founded on, they were added into ZXDB -- mainly from Einar himself! -- and those changes aren't those where one person suggests a correction into two sources - it's clear that the source of the current WoS data is from ZXDB.

This is not good enough, Fogarty. It doesn't explain how text that is bespoke to ZXDB appears in WoS. It doesn't explain how image references that are specific to ZXDB are appearing in WoS. Not by a country mile is this explanation is an explanation. It's just another attempt to gaslight the community. If he's serious about wanting to resolve this, he needs to stop with the gaslighting, and enable an ability to make direct comments to direct statements that Einar has published.

Again, this is how this goes:

* Fogarty and Chandler issue a public apology to Einar that meets with Einar's satisfaction
* WoS states clearly that its updated data is from ZXDB, and it will abide by the licensing conditions of it.

Notice Fogarty is admitting to deleting data when it's been reported -- why only one issue at a time? This is called list-washing, the intention is to minimise the visibly breach of licensing, while still violating the licenseing terms. Fogarty should delete ALL DATA that came from ZXDB, regardless if it's reported or not, if his intention is for WoS to not use ZXDB.

It's a simple case of rolling back all the data to the last old-WoS update, and applying only the changes that didn't come from ZXDB.
Fogarty: This is all something that could have been sorted with a PM.
And in the time between 18 June 2020 through to 2 July, why hasn't Fogarty responded publicly or sorted with a PM Einar's concerns. That's two weeks, most of it silent where he could have talked this out with Einar. Why has he failed to explain why ZXDB-specific data appears in WoS pages? And how does that happen without touching ZXDB data?
2 x

User avatar
kolbeck
Microbot
Posts: 140
Joined: Mon Nov 13, 2017 9:04 pm

Re: New WoS and ZXDB

Post by kolbeck »

Mike Davies wrote:
Sat Jul 04, 2020 1:42 pm
Why has he failed to explain why ZXDB-specific data appears in WoS pages? And how does that happen without touching ZXDB data?
If you ask kids the same type of question, the answer is “by magic” :lol:
3 x
https://zxinfo.dk - another ZXDB frontend
https://api.zxinfo.dk/ - ZXDB API for developers

User avatar
PeterJ
Site Admin
Posts: 2466
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New WoS and ZXDB

Post by PeterJ »

Mike Davies wrote:
Sat Jul 04, 2020 1:42 pm
It doesn't explain how text that is bespoke to ZXDB appears in WoS. It doesn't explain how image references that are specific to ZXDB are appearing in WoS.
Very good points @Mike Davies.
0 x

User avatar
moroz1999
Manic Miner
Posts: 237
Joined: Fri Mar 30, 2018 9:22 pm

Re: New WoS and ZXDB

Post by moroz1999 »

My thoughts:
There wont be any peaceful resolution.
Lee won't admit that he has used a lot of ZXDB information.
No apologies would be made.
Current WOS team is lost to common sense or logical actions.

We need to accept this and move on, that's the new reality we are living in:
1. There won't be one single ZX information archive anymore. There was never, and this illusion will be lost completely.
2. Prepare for some actions from WOS. I cant tell what to wait for (DDOS, legal queries, gaslighting in social media - whatever), but don't let them take you by surprise. Search for the weak points and start thinking about defense and possible strategies in different situations.
3. The WOS won't be the only problem. As the community would grow, you would have to take measures you wont like now. It's better to think at about it now and formulate different situations for yourselves now already.
6 x

User avatar
RWAC
Microbot
Posts: 155
Joined: Sun Aug 18, 2019 9:59 pm

Re: New WoS and ZXDB

Post by RWAC »

moroz1999 wrote:
Sat Jul 04, 2020 7:32 pm
Lee won't admit that he has used a lot of ZXDB information.
Despite evidence to the contrary. Everyone knows the truth now so this stance just makes him look stupid. It's like the kid with a mouthful of sweets swearing blind he hasn't taken any.

Nobody really cares anyway. Just admit what we all know anyway and move on.
1 x

User avatar
Einar Saukas
Dynamite Dan
Posts: 1303
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

Multiple sources have confirmed the same words from Lee Fogarty posted at his own private Facebook group, that I cannot access. I wasn't planning to quote him, but since this is now in the open, there's no reason for privacy concerns anymore. He apparently forgot to post his claims in his own forum, so let me help him:
Lee Fogarty: "I think it needs clearing up exactly what "created by" means. And another reason the data has to go is some of the changes are dubious or un-needed. This is why WoS always wanted to credit people submitting changes, and list them on the whats new page.
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.
Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.. both parties will likely create the same table with the same data. There is a WoS admins group on FB that Einar was in, and posts still there where I am sending structure/data to him.
That seems to be changing now to we are using "their" data. Again - untrue. There are some left over bits from a very old import test that are being removed.
WoS currently has over 300,000 indexed pages. Not just software. The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.
This is all something that could have been sorted with a PM."
So that's the main point. Is new WoS simply using the same data from old WoS that was imported into ZXDB? Or is it using such an early version of ZXDB (from July/August 2016) that only contained old WoS data, so there's no need to credit ZXDB (despite literally about 50,000 fixes I did when importing this data)?

Unfortunately the answer is no. To understand the difference, let's take a look at ZXDB chronology. The summary below has plenty of links to prove everything, although I suggest ignoring the links for now and just reading from start to finish:
So that's the point. It comes back to something I wrote in my original post:
Einar Saukas;57777 wrote:I imported old WoS content with their help from July 2016 to August 2016. If they had used one of the early versions of ZXDB that they participated, without crediting ZXDB, I would leave it alone. However they chose to use a version of ZXDB from September 2018, in order to take advantage of over 2 years of other people's work, without crediting anybody. That's a ZXDB version released 2 years after WoS stopped supporting ZXDB and started attacking my work. About 1 year after ZXDB and SpectrumComputing were censored at WoS thus forcing ZXDB to move to another forum. Months after I was personally censored at WoS without ever receiving any explanation.
Everybody is probably asking now, how do we know that new WoS is using ZXDB 1.0.8? Is it really much different from the original data from old WoS?

I'm glad you asked :)

I will have to get technical now, but I will explain it so everyone can understand. We will compare new WoS content against old WoS and a few ZXDB versions. Let's see what happens!

To reproduce this experiment at home, you need MySQL (or even better MariaDB) and any SQL client (HeidiSQL, MySQL Workbench, DBeaver, etc). They are open source and free. Also download a few versions of ZXDB from Github (click on "commits" to find and download older versions) and load one of them into your database.

I already mentioned you could visit new WoS software page and click on "EXPORT CSV (ALL)", to download some of the data from all titles stored in new WoS. I know lots of people did it (you may still have an old copy of this file yourself, perhaps in your Recycle Bin?). Let's start with "software-20200616.csv" from a day after new WoS was launched (we will talk about files from different days later).

Here's a database script to import this file into a database. Even if you don't know SQL, you should be able to see it's quite straightforward:

Code: Select all

create table x_newwos (
  rows0 varchar(100),
  id int(11) not null primary key,
  title varchar(500),
  slug varchar(500),
  no_players varchar(100),
  turn_type varchar(100),
  entry_type varchar(100),
  availability varchar(100),
  comments varchar(5000),
  is_x_rated varchar(100),
  is_crap varchar(100),
  clone_of varchar(100),
  old_id int(11),
  title_publisher varchar(500),
  publishers varchar(500),
  all_publishers varchar(500),
  entry_groups varchar(500),
  distribution_status_type varchar(500),
  display_image varchar(500)
);

load data local infile 'software-20200616.csv'
  into table x_newwos character set utf8
  fields terminated by ',' optionally enclosed by '"'
  lines terminated by '\n' ignore 1 lines;
Now download the original Martijn's WoS internal file "maindb.dat". Hopefully Lee Fogarty declared it "open source" so I don't need to worry anymore about sharing it. If you don't believe this file is authentic, choose any game at random and compare the corresponding line in this file against the old WoS pages. Let us know if you spot any difference!

Here's a simple database script to import this file:

Code: Select all

create table x_entries (
  titlekey varchar(500),
  pubkey varchar(500),
  title varchar(500),
  release_year varchar(10),
  orig_publisher varchar(500),
  re_publishers varchar(500),
  memory varchar(500),
  players varchar(500),
  joysticks varchar(500),
  genre varchar(500),
  category varchar(500),
  language varchar(500),
  distrib_status varchar(500),
  schemetype varchar(500),
  downloads varchar(500),
  flags varchar(500),
  authors varchar(500),
  aliases varchar(500),
  id int(11) primary key not null,
  spot_num varchar(500),
  spot_genre varchar(500),
  spot_full_price varchar(500),
  spot_budget_price varchar(500),
  spot_disk_price varchar(500),
  spot_comments varchar(500),
  spot_publisher varchar(500),
  license varchar(500),
  groupname varchar(500),
  comments varchar(5000),
  series varchar(500),
  orig_price varchar(500),
  c64_ref varchar(500),
  spanish_price varchar(500),
  wikipedia varchar(500),
  typein_ref varchar(500),
  authoring varchar(500)
);

load data local infile 'maindb.dat'
  into table x_entries character set utf8
  fields terminated by '\t'
  lines terminated by '\n';
The first CSV file from new WoS didn't have much useful content besides title, original publisher, and comments. Comparing title and original publisher from old titles won't help, since old WoS rarely got this information wrong so it almost never changed. However comparing comments is very useful, since they are continuously improved in ZXDB with fixes, further details, etc.

Here's a simple SQL to compare comments (except backslashes) between 2 tables. Notice it only compares titles that existed in old WoS (i.e 24369 titles with ID below 28187) to give new WoS a better chance:

Code: Select all

select e.id,e.comments,x.comments from entries e
inner join x_newwos x on e.id = x.old_id
where replace(coalesce(e.comments,''),'\\ ',' ') <> replace(coalesce(x.comments,''),'\\ ',' ')
and e.id <= 28187;
From this comparison, you will get the following results:

Code: Select all

new WoS (software-20200616.csv) vs. old WoS (maindb.dat)        - 2583 differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.0 (April 2018)     - 5 differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.8 (September 2018) - 0 (zero) differences
new WoS (software-20200616.csv) vs. ZXDB 1.0.9 (October 2018)   - 1 difference
new WoS (software-20200616.csv) vs. ZXDB 1.0.69 (latest)        - 766 differences
As you can see, there's a lot more similarity between new WoS and current ZXDB, than between new WoS and old WoS.

What if you want to repeat this test yourself to believe it, but you only have a newer CSV file from a different day? No problem. Although the CSV format at new WoS has changed over time, any CSV file downloaded before 2 days ago (when all comments changed into a bloody mess) will do. You just need to add or remove a couple columns from the import script, based on the column names you can see at the top of your CSV file. For instance, here's the same script adapted according to the CSV columns from 2 days ago:

Code: Select all

create table x_newwos (
  rows0 varchar(100),
  id int(11) not null primary key,
  title varchar(500),
  slug varchar(500),
  no_players varchar(100),
  turn_type varchar(100),
  entry_type varchar(100),
  availability varchar(100),
  comments varchar(5000),
  is_x_rated varchar(100),
  is_crap varchar(100),
  clone_of varchar(100),
  old_id int(11),
  title_publisher varchar(500),
  release_year varchar(10),
  search_title varchar(500),
  known_errors text(30000),
  has_inlay varchar(4),
  has_loading_screen varchar(4),
  machine_type varchar(500),
  publishers varchar(500),
  control_types varchar(500),
  theme varchar(500),
  all_publishers varchar(500),
  machine_types varchar(500),
  entry_groups varchar(500),
  az varchar(500),
  distribution_status_type varchar(500),
  display_image varchar(500),
  index x_id(old_id)
);

load data local infile 'software-20200702.csv'
  into table x_newwos character set utf8 
  fields terminated by ',' optionally enclosed by '"'
  lines terminated by '\n' ignore 1 lines;
The same comparison using a more recent CSV file will show nearly identical results, except for 2 titles: Reckless Rufus (new comments added on June 26th) and Werner's Quest (new comments added on June 17th but later lost).

It's absolutely clear that new WoS is really using content taken from ZXDB, not from old WoS. Instead of just pointing a few examples, we have now executed a comparison involving all titles. Even better, I provided instructions so anyone can replicate this experiment at home to see by themselves. And this comparison demonstrated that new WoS content is very much different from old WoS, not so much different from current ZXDB, and absolutely identical to ZXDB from September 2018.

As promised, this is my final post providing evidences. There's no need to prove anything else.

So what now? Well, THAT QUESTION will require one more post. But it's late, so let's talk about that tomorrow.


NOTE: On June 18th, I did a similar test at SpectrumComputing and found 2572 differences (instead of 2583) between new WoS and old WoS. It's because I compared new WoS against old Wos data that was already converted to ZXDB in 2016. This new comparison now, directly between new WoS and old WoS, is even more accurate and indicates even more differences.


NOTE: Reproduced from my post at the WoS forum
9 x

User avatar
Einar Saukas
Dynamite Dan
Posts: 1303
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

UPDATE: Please download "maindb.dat" from here.
2 x

User avatar
Einar Saukas
Dynamite Dan
Posts: 1303
Joined: Wed Nov 15, 2017 2:48 pm

Re: New WoS and ZXDB

Post by Einar Saukas »

SO WHAT NOW?

I have a few requests to the WoS team. Feel free to point out if I'm asking anything unreasonable at all:

  • An apology for slandering me would be the honorable thing to do now. Frankly I'm not really expecting it to happen anymore, but I'm an optimistic person!
  • Please stop trying to disguise using ZXDB every time you are caught. It's not convincing anyone, it's just ruining everything. The line "Your honour, I'm refunding to banks the part of the money that the police has found, can I go now?" doesn't quite work. Now that it's proved new WoS is using ZXDB, anything else you change will be just "new WoS is using ZXDB but crippled". You won't get off the hook for using ZXDB dishonestly, and you will be providing a bad service for your own users. It's the worst of both worlds. A lot of users still visit WoS nowadays, either because they don't know any better, or because they just don't care. Everyone else that cares about your attitudes have already left, so you are just wasting your time trying to prove yourself to those left who obviously don't care. So please stop the whac-a-mole and focus your efforts to provide a decent service to your remaining users instead. Martijn spent decades building WoS credibility, his legacy deserves better.
  • Please stop trying to take credit for other people's work. Nobody is questioning who built the new WoS site and has been maintaining the WoS forum. Is it not enough to receive well deserved credit for those? However the database is not yours. The old WoS files were built by Martijn, maintained by him for decades with help from many contributors, then converted by myself into a proper database (with my questions answered mostly by Gerard and Martijn), then maintained by the ZXDB team for the last 4 years with help from many contributors. As I mentioned already, new WoS only uses the list of publishers you converted in 2017 (everything else is taken from ZXDB) and it still contains the same errors that I pointed out in 2016 and you claimed to have fixed in 2017. Removing even more improvements from ZXDB 1.0.8 won't solve anything and won't benefit anybody. Please do the right thing, credit ZXDB properly at new WoS, as a sign of respect for all the people that have been working hard to keep Martijn's initiative alive. Notice there's absolutely no need to credit me personally for anything, I never asked for it, just credit ZXDB to properly recognize everyone's effort, and let's get over it.
  • Please update new WoS to use latest version of ZXDB instead. Since you are already using it anyway, there's no point in keeping it outdated. It's bad for your own users to see wrong data already fixed everywhere else, and it's more work for the ZXDB team every time someone points out conflicting information between ZXDB and WoS, and we have to convince them that WoS is wrong.
  • Please respect the ZXDB license. It doesn't exist to take "ownership" over old WoS data or anything. It simply asks to give proper credit to other people's work, and to retribute your usage of everyone's contribution by simply contributing back. It's not too much to ask, is it?
  • Please stop trying to divide this community. In particular, ZXDB is a community effort, it will always remain open and freely available for everyone to use it honestly, including new WoS. An attempt from new WoS to compete, by taking an older version of ZXDB and then trying to update it independently, will prove nothing and it's just plain stupid. Instead, it would benefit everyone to work together and join efforts to improve it further. For instance, years ago Gerard was organizing magazines for WoS, so if he has anything to contribute with ZXDB now, it will be more than welcome. Likewise, the What's New page at new WoS shows that new WoS online update tool is getting used almost exclusively to add RZX files one-by-one, but a simple upgrade to latest ZXDB would already add everything automatically at once. However, if you don't want to cooperate, fine. As Lee Fogarty stated several times, data from WoS is "open source and fully available", thus from now on, I won't even bother to check if someone sees an useful update at new WoS and decides to send it to ZXDB too. Anyway, my offer for cooperation still stands, you know how to contact me publicly or privately, I'm still available, willing to help and make things work out. Either cooperate or leave it alone, it's your call. Just please cut the crap.
Now I have no intention to post about this problem again. Please don't drag me back.


NOTE: Reproduced from my post at the WoS forum
Last edited by Einar Saukas on Sun Jul 05, 2020 10:07 pm, edited 1 time in total.
19 x

Post Reply