New WoS and ZXDB

[Ralf]

That's fair what you are asking for Einar.

I would have yet another request, maybe this time not for admins but to some regular WOS fellows.

If you really don't care about all this buzz with ZXDB, then just don't take part in discussion.

I can understand that you don't care about databases, data accuracy, new releases, people's work,
one guy being unfair to another guy and you just want to babble a bit about Jet Set Willy.

I can understand that you lived last few years in a ground hole with a family of badgers and have no
idea about recent events.

But I can't understand you make emotional posts that you don't care. You know that you're contradicting yourself?
Leave it to the people who care.

[XTM]

This post by Einar feels like a cliffhanger at the end of a season of a TV series (like the two-parters at the end of a Star Trek series)

The ZX Files, S7E12 (Part 1 of 2), final lines of dialogue:

"Okay, Einar has posted. What's going to happen now?"

"Now we wait ..." (dramatic music as the camera closes in on whoever said that, then screen fades to black)

[Mike Davies]

Lee Fogarty from WoS Facebook Group:
...
From what I can make out, the claim is that we are using the ZXDB database. Totally untrue - the WoS db was created a long time before - using the original data files.

The claim is:

• the current version of WoS is showing data that wasn't in Martijn's WoS data files, but came from the October 2018 version of ZXDB.

We know this because comparing Martijn's WoS files and ZXDB of Oct 2018 shows the same differences. These are changes that are unlikely to have been made independently. For example, inserting an image into an entry's comments with a filepath that includes *zxdb*. How would WoS independently arrive at having a zxdb directory for images if it wasn't using ZXDB or referencing it?

Why would WoS refer to ZXDB ids of software (e.g. ZX81 titles that haven't been in WoS, and are currently not displayable in WoS) within the Spectrum ports of the title? Why would the WoS comment text on a title show ZXDB's raw linking format, instead of Martijn's WoS data file linked-reference style? And link to software with titles that don't exist in WoS, but do exist in ZXDB -- *and* use the ZXDB ids to that software?

Any group of people creating a database from an existing dataset will invariably create similar tables and structure. Things such as the machine types used - create a list for machine types.

Except, invariably the two independent groups wouldn't make the same mistake. But WoS made the same mistake as ZXDB: we know that ZXDB switched to using the machine type label *"ZX-Spectrum" (it was a mistake, based on an assumption that makes sense in Brazil), but why did WoS "independently" change from machine type label "ZX Spectrum" to the label "ZX-Spectrum"?

There are some left over bits from a very old import test that are being removed.
...
The software is a very small part of the database, and with the bits we are removing, comes to a minuscule amount.

There you have it folks, confirmation:

* Fogarty imported ZXDB into WoS "very old import test".
* Fogarty confirms that ZXDB-acquired data is in the current WoS because he is removing it as it's reported, and it "comes to a minuscule amount".

" No sir, I didn't steal that guy's money, I only have a miniscule amount of it"

So an import script for ZXDB into WoS did exist at a point in time, otherwise, how could a "very old import test" be done? If that import test was done in 2016/2017, Fogarty would have needed to take his time machine, pop forward to a date after October 2018, grab a copy of ZXDB version for October 2018, go back in time to 2016, so he could run ZXDB against his import script. I feel that's rather unlikely -- there's probably more interesting things you'd do with a time machine.

What's more likely to have happened is that Fogarty couldn't get Martijn's data files into his own SQL database. Either he didn't have the technical ability to get the import working, or he lost part of the data, and didn't have a working backup. So he borrowed an October 2018 version of ZXDB, and hoped no-one would notice. Because, he can't admit he failed where Einar succeeded, and he talked up the importance of his import scripts. And it's much easier to import data in an SQL database when the data dump is already a series of SQL statements.


Unsurprisingly, with the barest of scrutiny, Fogarty's claims don't hold water. That's why he doesn't respond directly to Einar's points, and prefers to reply from a safe distance away from the actual claims. Funny he doesn't respond to a claim on his own forum, in a section dedicated to discussion WoS.

[Rorthron]

There's another reason he took the data from ZXDB. The WoS files were hopelessly out of date, and it's easier to fix that by ripping off other people's work than doing it yourself.

To be honest, though, I'm not sure Lee's comments are worth this level of analysis. He's shown himself to be a transparent liar time and time again. No-one with knowledge of the facts and an ounce of sense would ever believe a word he says.

[Pegaz]

All this is true, but at the end of the day he is the winner.
He took ZXDB, ignores any attempt to resolve this matter and I’m sure he laughs at all of us and feels untouchable.
He simply doesn't care about anything or anyone, except for himself and his selfish interests.
No surprises, predictable and expected epilogue.
I hope Martjin doesn’t know about all this and has found peace of mind.
Martjin whom I remember, would never have allowed such a degradation of his wos.

[ketmar]

All this is true, but at the end of the day he is the winner.

nope, this is only a short-term win. the reputational damage is now irreversible. and that damage is not something that immediately bites you, but it will bite you hard in the future. "modern WoS" already lost alot of invaluable enthusiasts (and many of them are keeping the scene alive for decades!). as the time passes, "modern WoS" will lose more and more people interested in Speccy, and will become a forum used primarily for small talk. and there are alot of such forums in internet.

so in the end of the day, Lee will simply close "WoS", as it will become "too costly to maintain, and users are not really interested in Spectrum anymore".

all we should do now is just wait. no need to hurry, the time is on our side.

p.s.: no, i don't really want WoS to die. but in its current state i'd better see it closed. and it will happen sooner or later.

[Mike Davies]

He took ZXDB, ignores any attempt to resolve this matter and I’m sure he laughs at all of us and feels untouchable.

He can laugh as much as he likes. The community around ZXDB gets better every day, with no backwards steps. ZXDB is up-to-date, freely downloadable, multiple sites based on it, each pushing the envelope in different directions. There's still data out there that can be brought into ZXDB, so there's still interesting things to do.

That's something WoS had while Martijn was active, and after that WoS has lost that through inept and antagonistic leadership.

WoS being a broken 2-year-out-of-date implementation of ZXDB -- and going backwards -- isn't making a case for being the go to Spectrum archive on the Web. Is that really what the WoS community gets after a 5 year wait??

Thing is, all the praise for WoS is toward what Martijn accomplished. Fogarty took a successful and respected website and drove it into the ground. I'm astonished when respected people are suggesting it should just be shut down, for the sake of the community.

[Vampyre]

Well, Einar gave quite detailed arguments for his claims.
We will see soon what the response will be and who was right...

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

[Guesser]

Wow, 850 words to not actually say anything.

[PeterJ]

Just to make a point that (this has been mentioned previously) Einar, Dave and I had our PM rights on WoS removed some years ago.

I did read through the post with hope, but it doesn't seem to answer any questions. It reads like something Dominic Cummings would have written for Boris. Maybe this could be clarified in the Rose Garden?

Even though ZXDB is Einars baby I did tell [mention]polomint[/mention] earlier in this thread that Lee or Richard were welcome to contact me. Lee is a member here and his PMs are not blocked. They have not made contact.

[Rorthron]

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

Even by Lee Fogarty's standards that's risible!

[Stefan]

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

OMG like watching an infant try to string some words together into an incomprehensible sentence, but then not one, but multiple paragraphs of gibberish. After reading it a few times I still don’t know what he is trying to say.

[RWAC]

"Ok, time to clear this up."

Then proceeds not to clear anything up!

[R-Tape]

I can't see it. Since yesterday, Malwarebytes has been giving a Trojan warning for main site, forum, anything. Just me?

[PeterJ]

I can't see it. Since yesterday, Malwarebytes has been giving a Trojan warning for main site, forum, anything. Just me?

Ok, time to clear this up. After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. Other have previously said they had the old database anyway. What they actually had was a very small selection exported to a CSV file from the main database. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

Having said that, like I said, there will never be an agreement. I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek. I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

So, moving forward....

Having spent so much time with the data, it was clear that there were issues with the database. Issues that were so numerous, the decision was made to rebuild it, using the initial titles as a starting point - people will know this from the old CSV download - there's a lot more to it than just that one file, but it was a starting point. We've posted numerous updates over the last 2yrs, detailing the work we are doing, and where we are at. To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

As we neared the end of this, I was also putting together the new screens. It was intended to put this live at the end of July. In preparation for new scans (around 300gb), I purchased a higher spec server. Don't forget - we are an archive. It's a lot different to just linking to archive.org and hoping it stays up. This means we currently have 630gb files on the server. Some - such as the new inlays - aren't available yet.

The server move proved to be as bad as the last time, with the old cgi files not playing well. These scripts are almost 25yrs old. The only ones remaining that hadn't been moved to the new website were the infoseek related ones. So, one evening I switched the new pages on. I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live. However, the way the WoS database is constructed means that we can take change requests - not all can be actioned until we have the new set of data, but a lot can be. WoS has 91 tables in the database.

For an example of how much has gone into this, the old database has 28,699 titles & versions listed. This currently stands at 33,014 - and we haven't completed compilations or mag tapes yet. This is still phase 1, as we also have thousands of new scans to go through - they will be processed after we have finished compilations. This won't mean much to the average user - who cares that there are 5 versions of Licence to Kill and not the 3 currently listed? However, for the likes of collectors, this is important information.

The Infoseek engine currently has 308,807 items indexed, plus over 300,000 indexed, downloadable files. This isn't restricted to software - we have indexed every magazine page separately - something that the old WoS db didn't do. Because of the indexing, it has improved the infoseek results.

As for the data - I appreciate people are waiting for the advanced search to come online, but as it stands at the moment, with pages and sections missing, we still have far more information available to download than ever before - and based on feedback, that can only increase. The API is comprehensive and provides a lot more than the older one, and is also still being built upon. We have built this data up with accuracy at the front at all times. I am sure there will be some mistakes in there - it's to be expected.

So, what now?

The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

The correct database will be loaded in the next few days.
Post edited by Lee Fogarty at 6:59PM

[Pegaz]

What are the odds that Lee will just ignore Einar's arguments and respond with some feeble and false whataboutery?

https://worldofspectrum.org/forums/disc ... -db#latest

You're right, all this was expected.
With this autistic approach, he is just deliberately provoking Einar, without direct answer.
What a 42 carat plonker he is...

btw, the "Thanked by" list also deserves attention:
Update 1:
Thanked by (2): polomint, mik3d3nch

[Vampyre]

Am I reading the following correctly?

"Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself."

Is he saying that Einar was only allowed this data because he was part of WoS Admin (either as an actual Admin or just a member of the FB group) and shouldn't have distributed it as ZXDB?

If so, I also don't understand this contradiction:

"What Einar was given..."
"This was never released by Martijn, nor myself"

Then why the hell would Martijn have HELPED Einar to import the data into a different database structure? To satisfy some morbid curiosity?

I'll leave it others better in the know to put me right.

[namco]

Am I reading the following correctly?

"Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself."

Is he saying that Einar was only allowed this data because he was part of WoS Admin (either as an actual Admin or just a member of the FB group) and shouldn't have distributed it as ZXDB?

If so, I also don't understand this contradiction:

"What Einar was given..."
"This was never released by Martijn, nor myself"

Then why the hell would Martijn have HELPED Einar to import the data into a different database structure? To satisfy some morbid curiosity?

I'll leave it others better in the know to put me right.

I had to read that part twice myself.

And I'm fairly sure that databases have access privileges, if you didn't want data being "stolen" why was Einar given full access? That's what it seems to imply - unless I'm missing something here?

[Vampyre]

Am I reading the following correctly?

"Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself."

Is he saying that Einar was only allowed this data because he was part of WoS Admin (either as an actual Admin or just a member of the FB group) and shouldn't have distributed it as ZXDB?

If so, I also don't understand this contradiction:

"What Einar was given..."
"This was never released by Martijn, nor myself"

Then why the hell would Martijn have HELPED Einar to import the data into a different database structure? To satisfy some morbid curiosity?

I'll leave it others better in the know to put me right.

I had to read that part twice myself.

And I'm fairly sure that databases have access privileges, if you didn't want data being "stolen" why was Einar given full access? That's what it seems to imply - unless I'm missing something here?

The "never released" "database" Einar was given/"stole" was less a database in what we'd traditionally think of as a database and more a bunch of .dat files that he had to have the original creator help him to import into a standard database engine as they were difficult to interpret. I think I have that right - others know better than I.

So the "stole" isn't some hack of the WOS server or any other devious method. He was physically given the files by WoS, presumably by some file transfer method decided between both parties, so this "stolen" nonsense is now apparently to do with that it wasn't supposed to be used to create a database. Even though the conversation initially began to create a database from existing data as no no one wanted a couple of decades of work to disappear when Lee said he was leaving.

[Mike Davies]

Fogarty:

Ok, time to clear this up. After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

Fogarty says: "Nobody from ZXDB has contacted me at all". But I see 22 posts from Einar in one thread "New WOS and ZXDB" in the Updates channel, which is the same channel Fogarty posted this. That's the channel people report issues and bugs with WoS.

Are we to believe that Fogarty didn't see that thread on HIS Forum, in a channel DEDICATED to WoS issues, nor did any of the other WoS admins, nor regular member, or they categorically failed to mention it to Fogarty that Einar's thread warrants some attention and a response?

Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise.

*Fogarty* has said *otherwise* on the WoS forum, that he provided Einar with the WoS data files in 2016. The "never be an agreement" is entirely in Fogarty's head. The public record shows there has been agreement between Fogarty and Einar on this matter.

What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

Fogarty gave Einar the files in late 2016. Fogarty invited Einar to join the WoS admin group a year later. So how could Einar be given the data "as part of the WoS admin group"? Chronologically, that makes no sense.

To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

Every single title and release has been *Manually checked*, so they did see the ZXDB specific information - the broken images in the entry comment (because they used ZXDB image paths), the ZXDB unprocessed related entry links, linking to titles by a ZXDB id minted independently of WoS.

Either that means they did a P?s?-poor job in manually testing, or they knew they had imported the title data from ZXDB.

I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live.

Is that true? That the current "new WoS" database isn't the database? If that is the case, why is Fogarty deleting the ZXDB-isms Einar is pointing to? WHy not "remind" Einar this isn't the real database.

None of this answers the question of why this database, this WoS database, contains ZXDB-specific data, without attribution?

As of today, the old database can be taken and used at will.

It's been as such since 2016, when Fogarty declared the data open source, and free for the community to use. This is nothing revelatory.

The correct database will be loaded in the next few days.

So new WoS has been running the *incorrect* database for 19 days. 19 days of silence.

What complete hogwash.

[bob_fossil]

Ok, time to clear this up. After god knows how many pages of discussions, nobody from ZXDB has contacted me at all.

Firstly, there will never be an agreement as to what happened with the old data. I say it was take in its entirety without permission, Einar says otherwise. Other have previously said they had the old database anyway. What they actually had was a very small selection exported to a CSV file from the main database. What Einar was given (in my opinion as part of the WoS admin group - the facebook group is also called WoS Admins), was the entire database. This was never released by Martijn, nor myself.

Having said that, like I said, there will never be an agreement. I will however say that the WoS admins are very good at telling me when I am wrong - and in this case all have agreed and stated over the last couple of years that my opinion here isn't wrong. Everyone in the admin group were working (as far as we were concerned) on Infoseek. I should also point out here that our "secret" facebook group is the official WoS fb and currently has over 5,000 members.

So, moving forward....

Having spent so much time with the data, it was clear that there were issues with the database. Issues that were so numerous, the decision was made to rebuild it, using the initial titles as a starting point - people will know this from the old CSV download - there's a lot more to it than just that one file, but it was a starting point. We've posted numerous updates over the last 2yrs, detailing the work we are doing, and where we are at. To give you some idea, every single title and release has been manually checked. This has led to tens of thousands of changes. Not just checking titles - this has involved opening and checking every inlay and scan on WoS, commenting on price differences, advertised versions, etc.. It's been a mammoth task.

As we neared the end of this, I was also putting together the new screens. It was intended to put this live at the end of July. In preparation for new scans (around 300gb), I purchased a higher spec server. Don't forget - we are an archive. It's a lot different to just linking to archive.org and hoping it stays up. This means we currently have 630gb files on the server. Some - such as the new inlays - aren't available yet.

The server move proved to be as bad as the last time, with the old cgi files not playing well. These scripts are almost 25yrs old. The only ones remaining that hadn't been moved to the new website were the infoseek related ones. So, one evening I switched the new pages on. I made no secret that it wasn't the database we will be using, as it is still being processed (fingers crossed this week). I have said many time here, and on the facebook group that this data isn't accurate and will not be used when our new database is ready to go live. However, the way the WoS database is constructed means that we can take change requests - not all can be actioned until we have the new set of data, but a lot can be. WoS has 91 tables in the database.

For an example of how much has gone into this, the old database has 28,699 titles & versions listed. This currently stands at 33,014 - and we haven't completed compilations or mag tapes yet. This is still phase 1, as we also have thousands of new scans to go through - they will be processed after we have finished compilations. This won't mean much to the average user - who cares that there are 5 versions of Licence to Kill and not the 3 currently listed? However, for the likes of collectors, this is important information.

The Infoseek engine currently has 308,807 items indexed, plus over 300,000 indexed, downloadable files. This isn't restricted to software - we have indexed every magazine page separately - something that the old WoS db didn't do. Because of the indexing, it has improved the infoseek results.

As for the data - I appreciate people are waiting for the advanced search to come online, but as it stands at the moment, with pages and sections missing, we still have far more information available to download than ever before - and based on feedback, that can only increase. The API is comprehensive and provides a lot more than the older one, and is also still being built upon. We have built this data up with accuracy at the front at all times. I am sure there will be some mistakes in there - it's to be expected.

So, what now?

The old database is considered legacy. As of today, the old database can be taken and used at will. However, I would like all mention of myself & WoS removing. It is provided "as is" without any guarantee of accuracy or quality.

The correct database will be loaded in the next few days.
Post edited by Lee Fogarty at 6:59PM

N Statement lost, 0:858

[Vampyre]

Also bear in mind Mike, as Peter said earlier, that he and Einar have had WoS PMs disabled for years so they couldn't possibly PM Lee. That's convenient.

And I don't get the new Infoseek using old WoS data either, that's going to use the "new" database in a few days. So he released the new all singing, all dancing Infoseek the other week with what he admits now was out of date data.

Your magnum opus, that you've said for years is the great work of your time, taking five years with a world class dev team - and you release it with decades old data? Oh, and you've been assuring the community the data has been continually and manually updated behind the scenes over those years. And you release this wonder of the web world but it's not connecting to the latest DB yet?

Jesus...

[Vampyre]

Edit: Thinking about it, using an older database version isn't unreasonable, just a little... odd. So I retract my previous post.

[hikoki]

They feal cheated and alone. Let the rascals steal a bone.

[Rorthron]

I felt the need to respond:

https://worldofspectrum.org/forums/disc ... -db#latest