Wiki/Report of Meeting 2023-10-26

Report of Meeting 2023-10-26

Present: Art Anger, Ed Gottsman, Raul Miller, and Bob Therriault

Full transcripts of this meeting are now available at the bottom of this wiki page.

1) Ed gave an update on the work he had been doing on GitHub searches. GitHub had been glitching recently and Ed said that this is an example of issues that can occur for the user if the database is not stored locally on their machines. Bob relayed a question that Chris had about crawling the wiki site. Bob understood that the the wiki was being crawled by Ed to build an SQLite database that is then downloaded to users. Ed said that that is mostly correct except that his crawl also includes GitHub and the Forums. The Forums and the Wiki are crawled incrementally, while GitHub is crawled in its entirety because there is not a way of tracking incremental changes that Ed is aware of. Ed very much appreciates Chris' offer, but the consensus between Raul and Ed is that it would be extra work to save seconds a week. Having the user hit the MySQL database that Chris creates on the wiki introduces latency that does not occur with a downloaded SQLite database. SQLite also provides relevance ranking across the Wiki, Forums and GitHub.

2) Ed showed off his adjustments to the GitHub search which now searches the Jsoftware account and provides primitive glyph searching along with the line numbers of the places that the code occurs. Ed has also implemented a 'next character' view that tells the user what characters follow the code fragment that has been searched in decreasing frequency. The challenge is that J primitives are often more than one character. This provides a way to see common use patterns in code. Ed noted that the Fold conjunctions are not used in the J repository (not surprising because of its relatively recent reduction). Ed finds that this a really good way to find examples of how code is used and by exploring the common usage there is an educational advantage.

3) Bob suggested the glyphs might be entered in the same way that tryAPL does, with primitives entered in a complete form rather than character by character. https://tryapl.org/# Raul wondered if there was a way to alert the user that they had completed a glyph with colour coding of the text. The results are refined progressively as characters are added to the text input. The search is very fast and is using indexed search on a string of all of the code. A word formation version on the same string is possible. Raul pointed out that spaces are not always recognized and this is an issue for tokens such as ':' which are suffixes as well as standalone. Bob thought that it might be useful to see the text box separate glyphs by spaces. There seems to currently be an issue with whitespace having meaning. Raul pointed out that quoted strings and comments may require a different approach.

4) Bob thought that there is an opportunity to use this as a platform for labs. Raul and Ed felt that this is better considered as an exploration mechanism for intermediate users because the amount of work to create the labs would be substantial. Bob wondered if a form of annotation might be useful, but Ed pointed out that is more about the commenting of code rather than developing an educational platform. Bob suggested that a wiki page may be able to provide an easier access to the tool. Ed suggested that as much as possible he would like the user to know how to use the tool by the way the tool is built.

5) Raul pointed out that when he has fullscreen and does not have a close button. Ed was checking versions to correct this and will look into that to see that he is doing that properly. Another option is to include a close button on the JQt form.

For access to previous meeting reports https://code.jsoftware.com/wiki/Wiki_Development

If you would like to participate in the development of the J wiki please contact us on the general forum and we will get you an invitation to the next J wiki meeting held on Thursdays at 23:00 (UTC) Next meeting is November 2, 2023.

Transcript

Yeah.

So GitHub just started glitching randomly and I retried and retried and it glitched on my end.

GitHub just started glitching randomly and I retried and retried and it glitched on different files each time.

Just downloading our balls of J software projects.

And so I gave it an hour, went back, tried again and everything was fine.

It worked.

So this is an example of what I'm wanting to make sure that the crawl happens where I can see it rather than it being distributed out to many users.

That minor glitch could have turned into a field support nightmare if it had taken place in the field.

As it stood, it was trivial.

It was a non-problem.

So I very much admire Chris Burke's zero admin approach to search.

But I think if we're going to get the user experience that we're getting now, which is basically search is more like a video game than anything else, I think we're going to be stuck with admin, barring some sort of complete redesign of the architecture, which you could conceivably do.

Amazon Web Services, for example, supports streaming.

So you could do a streaming style instant video game user experience that had no local cache whatsoever.

But that's a significant effort.

I don't know anything about that stuff.

And the cost, the fees would probably start to add up pretty quickly.

So I think we may have resolved this last week.

I don't know.

But that experience on Sunday persuaded me that we're in the right place as far as admin goes.

So Chris, after reading my minutes of the last meeting, sent me an email asking me what crawling meant in my mind for when you're crawling the site.

And what I replied was my understanding, and you can correct me if I'm wrong, because I'm probably wrong, was that it was you're using your laptop to crawl the wiki site and build the SQ Lite database that you're storing on AWS.

Is that correct?

That's mostly correct.

It's not just the wiki.

It's also the forums and it's GitHub.

Right.

And it's mostly incremental.

So it doesn't scan the whole wiki every time.

It doesn't scan all of the forum posts every time.

It does scan all of GitHub because I don't think there's any good way of doing that incrementally.

Okay.

And Chris's question when I said that was he said he's already got a MySQL of the wiki on the site.

So do you have to do the crawl?

And he's very kind, and that's great.

Maybe not.

Maybe I could hit his MySQL database.

But I've got a working thing.

Yeah, yeah.

Okay.

I could re-implement it to use his working thing, but I've got a working thing.

So I could make that investment, but it's not obvious to me that I'd be ahead if I did that.

Well, I guess what he's thinking is it might be less data flow for you just to rather than to crawl even parts of the site because I guess his MySQL is continually updating.

Yeah, and that's great.

I've got a master database that I keep.

Yeah.

And every week I check the wiki edit history, and I grab the last, I think, 10 days worth of changed pages.

Yeah, okay.

And I update my master database.

Okay.

So I'm pretty sure that I'm effectively doing exactly what Chris is doing.

Right.

He's doing it continuously.

I'm doing it weekly, but I bet it's exactly the same thing.

And again, yeah, I could write a MySQL interface that would go after his database.

I could do that, but it's not clear to me what the return on that investment would be.

And just to be clear, I very much appreciate his offer, but I'm not seeing it.

Well, I guess what I'm wondering, and I don't know because this is just my level of understanding, this is a conversation, but I'm wondering whether he's thinking at least as far as the wiki goes, this wouldn't apply to the posts or the GitHub, but as far as the wiki goes, would that get around the problem of trying to keep the wiki updated more often if his database is already up to date?

But the cost of me to update my master database is minor because the number of pages that get changed in a week is quite small.

Yeah.

I mean, it's seconds to do it all.

So yes, there would be a little less data flow if I could hit his master database and didn't need to update my own, but only a little less.

And the amount of time required to do it is pretty small, and it's swamped by the time spent on other stuff, like, for example, scanning all the GitHub file tarballs, zipballs, whatever they call them.

Yeah.

No, the other areas I understand.

He's not doing those.

So yours is the only way to get that.

But we were talking about.

.

.

He could save me a few seconds every week if I made that investment.

Is it possible to have the search reference his MySQL database for the wiki?

It's a formatting layer in between.

The database is formatted differently than how its current code consumes it.

This is a case of if he just puts in a few days work, he could save minutes of time.

Every week.

Seconds of time every week.

Yeah, more likely.

Yeah.

So again, I appreciate Chris's interest and I appreciate his offer, but it all works.

Yeah, yeah.

No, no.

It doesn't get us around the problem of the wiki being up to the minute, up to date, right?

Right.

Because you still have to go look at it.

Yeah, but I do that once a week and I only need a weekly update.

I don't need a continuous update that doesn't buy me anything.

Because I'm only updating weekly.

But the user will be out, like at the end of six days, the user will be six days out of date for what the wiki is, right?

But that would be true anyway.

I'm not going to generate new.

.

.

Oh, I'm sorry.

I see what you're saying.

Yeah.

No, no.

I see what you're saying.

You're saying that the client, not the crawler, the client should go against the MySQL database.

Is that what you're saying?

Is that possible?

Yeah.

It's certainly possible, but it would slow things down dramatically.

I'd have network latency where now I don't have any.

Okay.

And that's because that communication has moved to an area where it would slow down, everybody would converge on it.

No, no, no, no.

It's not.

.

.

No, because the usage is quite modest.

We're not talking about database contention or network contention.

That certainly wouldn't be a problem.

We're talking about a search every few minutes, even if it's wildly successful.

What we're talking about is network hops, bandwidth, IO.

Right now, when you do a search, it hits a local SQLite database and comes back in under 100 milliseconds.

Right.

Because you're downloading, you're updating right onto their machine.

Right.

Gotcha.

Okay.

Now that makes more sense.

Now I got it.

That was the part that I was missing.

I'd forgotten you're downloading the whole database right to your machine.

That's why it's quick.

Yeah.

And actually, I wanted to show you something on that topic.

Hang on here.

Yeah.

Let me.

.

.

Can I share my screen?

Sure.

So.

.

.

So I want to start with an inadequacy, which is when you do a search in live search, and you select a file, the file comes up, but it doesn't scroll to the hit, and it doesn't highlight the hit.

Right.

Yep.

And that's.

.

.

I don't know a way around that.

I can't, as far as I know, talk to the web view in that fashion.

It's not just a matter of telling the web view to scroll.

I've got to.

.

.

I don't know how to do that.

It's possibly you can't do that.

But I don't know how to do that.

Period, paragraph.

The other problem that I found is that when you're doing it, when you do a search and you get results, looking at them is like playing Where's Waldo?

So slash co slash co, you have to go and find that in each line in the results set.

So I added an underline, an underscore.

I don't know if you can see it.

Yeah, no, that's neat.

So that's nice.

And then I realized that if you limited your search to .

IJS files, so you didn't do forum posts and you didn't do wiki pages, I actually implemented something that does exactly that.

It's called GitHub.

And it's GitHub code search.

It's only the J software account.

And it uses the phrase input text box.

And if you do a search, you get a slightly different format of results.

Because we are only hitting GitHub and because we're only hitting IJS files, there's a couple of things we can do.

We can give you the name of the repository.

We can give you the name of the file, which actually live search also does.

But we can also give you the line number where the hit occurred.

Nice.

And that is not something we can do with a forum post, at least not easily.

And certainly not with a wiki page.

Neither of those are consistently reliably line oriented.

They're more paragraph oriented.

So the result of that is if you are interested in the stack format that IJS hit, for example, that's line 159.

Scrolling is annoying, but it's not too bad.

At least you know what your destination is.

And it's fairly easy to get there and find line 159.

And sure enough, there's slash go slash go.

All right.

So that paragraph.

The other thing I did was something I started to talk about last time and wrote a note about a couple of weeks ago, which is this idea of exploring code space by looking at commonly occurring tokens or characters following a code fragment that you type in.

And I'm just starting to experiment with that.

And I would like to get reactions.

So the way it works, and the presentation isn't very good yet, but I've typed in slash go.

And I get my pages, which you would expect.

And I get hits, as I've described, with line numbers.

But underneath the page list, I get not tokens, and I'll talk about that, but characters that follow the code fragment slash go and the number of times they occur in the corpus.

So there's just thousands of dot IJS files from GitHub.

389 times tilde occurs.

And if you type tilde, you get all of those hits.

Yeah.

And then you get a slightly different set of next characters.

Or it could be very different set of next characters.

49 times tilde occurs again.

And you can go looking at those.

And here's where character and glyph or character and token becomes kind of a problem.

It's not just tilde tilde.

It's tilde tilde dot.

Right?

It's nub.

And so really, what you should have seen here was not tilde 49 times, but tilde dot, you know, 31 times or whatever it is.

I've got to figure out how to reconcile typing in characters versus getting results in terms of glyphs or tokens.

I haven't quite worked that out.

But tilde tilde dot.

And again, you can load files and go to line numbers just the way you normally can.

Nothing that doesn't change.

But the idea here is to try to make it possible to learn about common usages, about idioms.

Actually, I had an interesting experience.

I went looking for fold and discovered that fold does not seem to be used in production, at least not dot dot and not colon colon either.

But in general, if there's something that does get used, you know, plus over divided by pound sign, for example, or plus over the top.

Oh, no, that doesn't work.

Say, plus plus slash dot times.

Which would be matrix multiply, right?

Yeah, so you get a bunch of those.

So this is a way, I think, of learning about the language that we don't normally have.

And I mean, we can learn about the language by reading NUVOC.

But NUVOC doesn't really tell you or does tell you it does try.

But one thing I learned a lot about was adverse, which I've read about but never really understood.

So colon colon.

I had never seen this idiom before.

So to execute a command and return zero if the command fails for whatever reason.

That's actually quite interesting.

I've been using if dot else dot, which is stupid, you know, by comparison.

Now the problem, sorry, go ahead.

I was going to say, well, it works and it essentially works the same way.

It's just that this is a more tacit version of it.

Yeah, right.

Well, it's and it's less verbose, which is, you know, all to the good.

Now, again, the issue with tokens versus strings, a lot of the colon colon matches are actually fetches.

So left left brace colon colon.

Now, a very common occurrence after colon colon is right square bracket.

And so this is a this is a common usage that it's probably worth studying to understand what's going on here.

So it's a way of reading code, but reading it, I'm not sure how to express this.

I don't even know if it's terribly important, but reading code in the large, not focusing on a particular line of a particular file, but focusing on common usages across many files, many occurrences across lots of different applications.

And I think that's potentially educational, but I'm very new to this.

I'm not sure.

I don't know.

I've never tried to teach Jay.

And I'm obviously no longer a beginner, although I am only an amateur myself.

So I'm not really sure about my own reactions.

So that's all throw it open.

Lots of comments, questions, abuse.

Well, I've got I've got one.

Idea, and you were talking about about glyphs as opposed to characters.

Yeah.

Is it possible at some point to do essentially word formation like the colon semicolon on the string and pick it up that way?

The thing the thing you run into is.

Let's say plus insert at.

Yeah.

Really want to do this isn't impossible and what you suggest isn't impossible.

I'm just not quite sure how to think about it.

It's followed by a colon.

So there are 127 occurrences of plus insert at followed by a colon.

That's not right.

It's really what you've done is you've typed in half of the glyph.

Right.

And what I should be showing here is at colon.

And probably also at dot, although it looks like dot doesn't occur.

So I'm not quite sure how to present that because you've already typed the at.

So showing at colon sort of implies that it's.

A plus insert at at colon, which also isn't accurate.

It's not a technical problem.

I can do word formation.

It's strictly a presentation problem, I think.

But that only occurs for colons or dots, right?

Yeah, that's I think that's right.

So I would say that's right.

So when you get your result and if it's got a colon, then I would just put the colon in.

Oh, but what if?

Oh, I see.

But what if it's not?

What if he wants the dot?

Yeah, yeah, yeah.

Or what if it's.

Yeah, I see what you mean, because if it's followed by something else, then it would be, I guess, a top instead of at.

Yeah, right.

Or vice versa.

Yeah, yeah.

Yeah.

It could be a top, it could be at or it could be whatever that colon is.

I don't remember.

At colon is that at a top and whatever at dot is, I don't remember.

Yeah, at dot is agenda, I think.

Is agenda.

Yeah.

Okay.

Yeah, yeah.

It chooses amongst the evaluates.

We never, we never get that.

We never get plus insert agenda.

I'm sure you get, I'm sure you get.

I'm sure you get the at dot, though, at different points.

It just might not make sense for it.

I would say you'd probably get right parentheses at dot.

Because that's usually how it's going to.

That is quite common.

Yeah.

Yeah.

And the reason is, is because it's, then its next verb is going to be within parentheses.

Right.

The verb it's choosing.

And then the other place that would show up, it will be the tie condition.

Next to a left parentheses.

So your grav and then left parentheses, because that will be all your, all your gerunds basically, you're creating gerunds that you'll use the agenda to come off of.

Yeah.

So I just, I think it might be a learning tool, but I'm not sure.

I haven't, I haven't pushed this.

This is sitting on a branch.

I don't know whether I'm going to do it or not.

That's part of what this discussion is about.

If you were to make it a learning tool, what if instead of just typing in text, what you needed to do is select the glyphs, like off some, you know, like, so you didn't have an option just hitting at, you'd have to hit at colon at dot or on its own because there's separate inputs layout.

How do you mean wrong?

There's just the new block page itself has, has the, all the primitives laid out there.

Oh, I see.

You might select off the new, something like the new book page.

Although it would be in a reduced form.

So that would be, you know, you've seen, have you seen try APL?

No.

Okay.

If you go type in try APL.

org.

And then hit return.

Oh my.

See the line across the top.

That's there.

I certainly do.

Yeah.

How interesting.

And you can use those.

You can click on those to enter text into the, into the, into the area.

Yes.

I see that.

I see that.

But then there's also keyboard shortcuts you can use.

So you don't have to do that.

And I think it's meta or escape or something horrible like that.

I think it's back tick, but I haven't used it that way too much.

Right.

Oh, interesting.

How interesting.

But if you were just like, if you were just typing in the text, you would have to hit back.

Yeah.

If you were to have the list across the top of whatever that, whatever your interface was for that.

Yeah.

Then you would be able to, cause it's basically pull a glyph at a time.

Right.

There is something a little fiddly and annoying about that, about selecting, clicking.

Typing feels much more fluid as you're exploring.

But you're right.

I mean, the reason, the reason they're doing this is not that clicking on glyphs is so wonderful.

It's that the keyboard doesn't match the glyphs.

Yeah.

So this is sort of a desperation device.

Well, what this was, was there was, I guess this dialogue is running the server that's supplying this.

This was actually the thing that triggered us looking at, at JPlayground.

Oh, really?

Yeah.

And the difference between JPlayground and this is this is actually running off of a server and JPlayground is running off of your browser.

Right.

Now, the thing that we're trying to address here is how, how does the user convey that they want a string versus a token?

Is that what the underlying problem is?

That might be a good way to express it.

Yes.

Why don't we go with that and see what happens?

One thing that might be worth doing is if there's a, if the current thing being typed in is a token, colorize it.

So the user would at least have feedback whether it's a token or not.

But in the case of at and a top, you both would be tokens.

Right.

But they would also, when they're typing, when they're typing in, if they type at dot, they would see that it's the same, it's all one token.

So it's colorized.

So you're saying that the input field would take on additional formatting.

Yeah, now, that would require having to build your own input field.

You would no longer be able to use.

That's not a problem.

So, he said confidently.

So.

I know that feeling.

Yeah.

So.

We do these things not because they are easy, but because they appear to be easy.

I'm sure Kennedy wished he had said that.

I guess, as I see it, the problem isn't so much the input.

Because I know what I'm typing.

Problem is the interpretation of the input?

It's the presentation of the next thing, of the available next items.

I'll call them items because I don't want to call them tokens or characters.

What are the available next items and how do I present them?

So, for example, R is the next item.

God knows why.

But it shouldn't be.

Because it's just.

What's the concept of next item?

How do I know what the next item is?

What, how is that meaningful to me?

That list across the top looks gray.

So, next items are the second line.

And so, there are 2,295 lines in GitHub in the J software account where the at sign is followed by a dollar sign.

And if you type the dollar sign, you do that search.

And now your list of next characters, and in this interface, in this presentation, they are just characters, not tokens, changes.

And it turns out there are 74 lines where at sign dollar is followed by another at sign.

So, this again is, yeah, go ahead.

There's two issues I see here that you're struggling with behind the scenes.

One is keeping a relatively small universe of what possible next items are so that it's meaningful to count them.

You don't have a massive collation problem to pull the results together.

And the other one is having enough access to the database structures to have a place where those numbers can come from.

Okay.

I'm lost on the second one.

Let's stick with the first one for now.

Okay.

It's there are two sides of the same coin, basically.

All right.

It doesn't worry me that at sign is followed by a colon 5,480 times in the repository.

That's okay with me.

So, I don't think I'm worried about large numbers.

No, that's not the number that I was thinking about.

Oh, okay.

If you had what is the next token, then every single English word that follows a something in GitHub, every single variable name, every single, I guess, I don't know if comments would be considered tokens, but every single variable name at least would potentially be something that you'd have to have a tally on recorded somewhere that you could, you could pull that number back out when you needed it.

It's a lot of, whereas if it's characters, you only have, you know, a few hundred or so.

I take your point.

90.

What I should be able to do.

Well, so all of all of this, everything you're seeing here, all of this processing, this is not SQLite.

This is just an indexed string that I'm running the intervals primitive against, and it is fast as lightning.

I think I could do the same thing.

Using a special combination.

So I'm binding E.

to the, it's about a 15 megabyte string.

Okay, so you've already, you've already got this, this, you're pulling it out of the temporary result that used to build the display.

That's what you're saying.

Yes.

And I could do something similar with word formation, a word formation version of that same string.

Okay.

I think that would be a good idea because already you like if you have, there's a big difference between a space colon and a colon in J and a space dot and a dot.

Yeah.

And, and you're not showing that here.

And that's a significant issue.

Yep.

But I, but at the same time, well, I, yeah, I just, I'm not, I'm not sure how to reconcile the fact that I'm entering care.

Well, I am, I am sure now because you, you showed me try APL, but I enter characters and I get tokens back.

I have to think in terms of upcoming tokens.

I have an idea.

Maybe.

Oh yeah, please.

This is subtle, but in your actual text box, when you hit the, say you do at colon, then as soon as it's a recognizable glyph, the next thing it's going to, the next thing you enter is going to have a space between the colon and the next thing.

So you're separating the glyphs.

Yeah.

There's actually directions you can go here when we were, because, because, because tokens are composed of sequences of characters.

Sometimes there's two different kinds of suffixes that you got to consider.

One is, is a variation on the existing token that has a suffix, you know, inflection basically.

And the other one is the things that follow this current token that are not inflections.

Right.

Right.

Exactly.

That is the dilemma.

And, and that, and those are two different kinds of things.

And they should probably, if you're going to cover both, they should be displayed in a way that lets people see the distinct things.

Yes, exactly.

That is what I struggle with.

So in this case on the, on the text file, if you were to say, click on the O to be able to see the things that are followed by an O, in your phrase text, O would be separated from at colon by a space.

But, okay.

So it would be space O.

O.

Right.

After you did it, it would, it would separate them.

So you would know at colon is one thing and then O is the next thing.

Yeah.

And for the things that are colons and dots, there's a space dot and there's a dot without a space.

Yeah.

The two separate things that, that you need to need to display.

I mean, even if you stick with just letters, if you don't, if you don't, if you're not looking for the next token, you should at least make a distinction between a dot that has a space in front of it and a dot that does, that's part of it, that doesn't have a space in front of it because those are very different kinds of dots.

Yeah.

And the same with a colon.

Yes.

Dots and colons both.

Yeah.

Because the colon on it's- I hadn't focused on that.

Yeah.

The colon on its own with a space on either side is your definition of monadic versus diacritic.

It's a definition.

Yeah.

Yeah.

Whereas as dots and colons without spaces in front of them are inflection marks.

Yeah.

Right.

Oof.

So this is actually more complicated than I had realized.

Uh, okay.

Well, I guess the thing is- It's not that bad.

But it's- Yeah.

I mean, that's, just sticking with the current design and just, just making that distinction.

It'll be a few lines of code, but it's, it's, it's probably- No, no.

I'm, I'm, I'm struggling with presentation, not implementation.

Implementation doesn't, doesn't, at this point, doesn't scare me because I haven't thought about it.

Well, just for, for, for presentation, since there's only two characters here that are special like this, the dot and the colon, you can just mark off the first two things being the dot and the colon, and then maybe, maybe put a dividing line or a color change or something like that.

And then the rest of this stuff.

Ah.

Ah.

Hmm.

And they can be empty and collapse, I guess.

Is that in effect what I'm doing here with 5480 colons?

I'm not sure.

If, if you're ignoring spaces, then that's not what you're doing.

Yeah, I'm ignoring spaces.

I have bundled.

Yes, that's bad.

Right, right, right, right, right.

And then the other thing with the 5480, although it's the most popular with the colon, it's just as value, valid to have the dot, which is the 1152, and it's further down the list.

Yeah, well, the 584 will be split into two, into two different pieces, right?

Right.

Inflection to the, yeah, a leading freestanding colon.

Yeah, the same thing with the dot, the 1152 will be split into two separate pieces.

And actually your, your comment about the white space is exactly right, because it should be telling you how often the at on its own, and that would be white space following it, right?

Yeah, that's, there are two occurrences of colon.

He's making a third distinction here, which is, there's, there's colon, there's at colon, there's at dot, and there's at space.

And at space is also not represented here.

There's no, although technically, in fact, I'm actually, actually, maybe it would be it once you split out the col, once you split, once you split out the, the colon pairs and the dot pair, the dot, you know, the two kinds of dots, everything else is space, is space examples.

That's true.

Yeah, yeah.

Because there's only two, two inflections.

Yeah, it's the absence of an inflection when you get anything else.

Yeah, it's a lack of, yeah.

All right, well, I think I can, I think I can work with this.

Is there a way for you to retain white space when you're looking at your lines?

Because it looks to me like you're getting rid of white space.

I'm searching without white space.

I, I was collapsing inflections and freestanding colons and dots, and that's bad, and I need to fix that, and that will require a significant rethink.

Yeah.

Because mostly spaces don't matter.

I mean, if you, it's mostly spaces don't matter, and I was taking advantage of that.

I should call out one or two other special cases here to think about.

Please, yes.

Three, three, three special cases to think about.

Quoted strings.

If, if you're going to quote a string, the rules change.

Yeah.

Numbers, numbers is a slight change because you can have dots and numbers, but, and you can have, and numbers are kind of the same thing as what you got now.

It's probably not a big deal because numbers can both end in dot.

There's a special token for colons, and I wouldn't worry about that as a special case.

And same thing with, with, with, with names, which is the other thing.

So yeah, I guess strings are the only, the only special case we're worrying about.

Nope, strings and, strings and comments.

Comments, yeah.

If you want to comment.

I can, I can easily drop comments and strings.

That's not a problem.

Or I could leave them in, and you know, you just live with it.

It doesn't, it may not matter too much, but I, I can, I can easily omit them.

There's a global for removing comments from scripts, isn't there?

I mean, in the interpreter or in what context?

Well, I'm just thinking you could, you could run through your GitHub and remove all comments for what you're worth searching through, right?

Oh, it's easy enough to spot comments, drop leading spaces and well, you just drop everything after NB.

NB, yeah, yeah.

The line is only NB, you drop that line.

Of course, that means that you aren't going to be able to search on comment text when comment text might be useful.

Right?

Yeah.

I think it wouldn't disturb me too much to say, oh, you want to do that?

Do live search.

Don't, don't do the GitHub code search.

GitHub code search is mostly about looking for tacit code.

Fair enough.

And if you did do some sort of menu based, whether it was a pull down or a menu across the top, like try APL, then you really are limited to glyphs.

You can't put in variable names, for example.

And the other thing is there are, there are primitives that have multiple dots, colons and combinations of, right?

Like fetch or under infinite or whatever they call it.

Yeah.

Yeah.

Under, under dot, under, under colon indeterminate.

Yeah.

Yeah.

Yeah.

Yeah.

I'm going to need to do something with word formation.

The string search is not, is not appropriate.

Okay.

I'd like to take a step back.

Is this worth doing at all?

It looks fun to play with.

That's about all I can say.

I'll take that.

Absolutely.

Is it worth doing?

I think it could be a very powerful way for people to start doing it.

Go through the code and understand how it works.

But I think it would require people to write labs or some type of annotation that tells people what they're looking at.

I'm not following you.

Well, I'm thinking if you just go through this and if I'm a beginner and I start, you know, monkeying around and trying different characters and seeing what different things do, that is a powerful thing.

If I've got a certain level of expertise, I can start to figure it out on my own.

This is probably more intermediate than beginner.

Yeah.

Yeah.

But for a beginner, I would think you just need to, you know, you need to have a little bit of experience.

Yeah.

But for a beginner, I would think you just need to build some structure around this.

Not necessarily around this application, but to use this application with a lab for a beginner, I think would be very powerful.

I think of the beginner as being more of a Nuvoc or J.

Primer person.

I think the beginner, I think there is good beginner material.

I think I may agree with Raul that this feels more like, all right, I'm ready to code.

What are the standard usage patterns?

What are the idioms?

How do people do it?

What are the accepted ways of using hash colon, for example?

What are the standard ways of using it?

What's it good for?

I think I understand it, but do I really understand it?

And that feels like an intermediate level thing.

Okay.

But even for an intermediate level, I think it, again, I don't think it's the application.

I think it's the support of the context of the application, which might be a lab.

But what I would do at that point is, the lab is, how do people use slash colon?

You use this to pull out slash colon.

And then the lab would go through the different ways of what you're seeing, the different ways it's actually being used.

What is it doing here?

But that's a huge manual effort, right?

I have no objection to it other than that.

Yeah, no, no, it's a big effort, but it's what, if you were an intermediate trying to pick up, like if what you're saying, you can look at all the code, but in order to make it more effective, in other words, rather than having people randomly find bits of code and say, well, what did they do here?

What were they trying to do?

Again, often the code isn't really well commented.

So it's kind of hard to figure out what's going on.

And I think it's that annotation of what's going on when you see this one might be useful, but it, yeah, it's a lot of.

.

.

What you're arguing for is better comments in code.

I can only agree with you.

Yeah.

But that's a hard problem.

Yeah.

I come back to discovering how to use adverse properly.

Yeah.

I'm not, I wasn't ran, I was actually randomly typing in bits of code, but if I were curious about adverse, that wouldn't be random.

That would be me looking around and trying to understand adverse.

And some of the usages I might not get, some of them I would look at and say, oh yeah, that's obvious.

I knew that, but some of them might surprise me.

Yes.

And that surprise is valuable, I submit.

No, I think it is valuable.

And maybe what all I'm saying is the first step would be to put a wiki page in that just quickly explains that this is how you use this.

This is what you could go looking for.

This is actually going to take you to precise lines of J code that will use these glyphs.

If you want to know how to use these glyphs, you can see how they're used in J code.

Whether or not you understand it, you'll see how the glyph is used.

Right.

And that might be enough of a starting point to make this really more useful.

But I think it's.

.

.

I might be able to explain that here.

Yeah, absolutely.

And if I do a proper job of presentation, and this is what my first cut, I think I might be able to make that clearer in the tool itself.

I'm not sure how, but I think I can do a better job of presentation than I'm doing now.

But I take your point that more explanation is needed.

I would just like to see the explanation folded into the tool itself, rather than it's being a separate wiki page that I might or might not find and look at.

I agree with you completely there.

And what you're describing in terms of presentation, I think what you're describing is where the Tri-APL glyph line Yeah.

is closer to what you want there.

Yeah, yeah, yeah.

And that's not too bad, because what you could do is you could have all the main glyphs without the inflections, no colon, no dot.

And then when you hover, you get the colon and the dot versions as well.

Well, and you can see what they do on theirs.

They've got their tool tip show up.

Oh, yeah, look at that.

And they'll explain it basically to you.

It's a pretty good interface.

Oh, nifty.

Yeah.

Oh, all right.

I admire it.

It does not actually make me want to learn APL.

But I'm happy to steal it.

Somebody needs to do a steal APL.

org.

Well, and actually, that's what our J playground started out as.

And Adam basically said, yeah, go ahead.

Use it.

Like whatever works for you guys.

You need something that people don't have to download J to be able to play with it.

And that's when Will Gahate and Joe Bogner jumped in and created that.

But they went a different direction.

I'm just thinking that line might fit closer to what you want.

But don't hold yourself to that.

Something that encompasses that, maybe.

Yeah, I'll think about it some more.

Yeah.

OK, cool.

I do agree with Raul.

I think it would be very useful.

OK, well, I will continue with it then.

This is my neurosis, which is that I really like building stuff.

But the idea of doing product support leaves me cold.

So I'm using this as my excuse to defer general release.

But I will take another run at this and present next week if the meeting is going to take place.

And we'll see where we are.

And thank you both very much.

This was really helpful.

And the only thing about next week is I don't know for sure whether I'll be having a meeting.

Although I should know by, say, Tuesday of next week.

Oh, actually, no, next week I will be having a meeting.

No problem.

The week after that, I'm not sure about.

So yeah, next week's not a problem.

I just realized I took up most of the hour, which was not my intent.

Thank you for the discussion.

Not that I had a whole bunch to present.

Yeah, I didn't have really anything to present.

So it was a relief to me.

And then the only area that I did want to talk to you about was, you know, the discussion or the email I had with Chris asking about that.

And now I understand better why you want that to be able to be loaded onto the machine to be quick.

And that whole search interface is happening on the individual machine.

Yeah, and I'm more than happy to talk to Chris.

And also, it's probably the other thing that you can think about doing with Chris's approach is inverting the database every day, you know, doing incrementing it every day because it'll be on the host and the host wouldn't have to upload the file anywhere.

It could just be downloaded straight from the machine.

But the downside there is now the user is downloading the thing, the database every day.

Yeah.

Potentially.

Yeah.

Which you don't want to do that.

You don't want to require that.

Well, again, you wouldn't require it.

It would just be a button that says, you know, you can't update the database.

Well, that's.

.

.

Yeah, I've got a single.

.

.

Oh, go ahead.

Sorry.

If you're getting into that space, the whole concept of a stable database and an updates section for recent things that's coming in to start making more sense.

Yeah, that gets difficult to manage and fiddly.

I did think about that fairly hard when I was trying to figure out how to avoid 100 megabyte download and it can be done.

But just biting the bullet and saying, well, every couple of weeks, I'm going to download 100 megabytes is a lot easier to manage.

The other thing is I've got a single integrated database, one search and you get back relevance ranked results across forums, Wiki and GitHub.

No one's ranking.

And I'd lose that.

Yeah.

Yeah.

Yeah, that's on the SQLite people that they didn't think through the issues with continuous update of a large database.

I mean, this is sort of thing that APL people have been doing for decades.

No, it's not.

No, SQLite is more than capable of.

.

.

You can tell SQLite these eight documents have changed.

Here are new contents for them.

Yeah, but how do you distribute that?

Right.

I mean, you run into problems like, okay, who has which updates?

That kind of thing.

Where are you date-wise in terms of the update cycle?

Do you need to get the whole database?

You have just last week's worth of updates or two weeks or what?

Then your server has to be smarter and to take requests for the updates ever since the last two weeks or whatever.

It's just painful.

Putting it back onto the user, what about if you just on your update button, you just put in the date of the last update?

Right.

I've got to.

.

.

Oh, well, I mean, okay, what would that solve?

I don't say it's a bad idea, but what would it fix?

Well, if you were updated on a regular basis, the date of the last update could be yesterday.

And if you clicked on it, you'd get today's update.

If you clicked on it.

Right.

What that means is now your server has to be smarter.

Your server has to be able to take a request that says, give me everything that's changed since June 27th.

No, it just gives you everything.

Then what's the point of the date?

It tells you how far out of date you are.

Okay, but I still have to download 100 megabytes, right?

So how do I fix anything?

Well, no, but you've given the user the knowledge that they're.

.

.

If they say, you know, I don't have to be right up to date.

If it's the last two weeks, that's fine.

I'm not going to update.

Yeah, that is a good thing to do.

I should probably add the date, but I don't think it addresses any of the problems or issues that Chris has floated.

No, I agree, but that's because we've already assumed that the user is going to have that database on their machine for speed.

Yes.

Yeah.

Yes.

Speed and integrated results.

Integrated results.

That's why we're doing it that way.

But then I think what Chris is saying, why are you crawling the database once a week?

I've got a MySQL database ready to go.

And the answer is I am not crawling the database once a week.

I am grabbing incremental updates once a week, updating a master database that I keep privately and using that master database to generate the 100 megabyte file that users have the opportunity to download as frequently as once a week.

And that's the part of the picture that I was missing.

So that's on me for not understanding.

Yeah.

I used to crawl the whole of the key every single day.

I used to crawl the whole of the key every hour, every time.

I don't do that anymore.

Yeah.

Okay.

I would say this was actually really productive.

Oh, good.

Okay.

I hope Chris is not unhappy with me.

I'm more than delighted to talk to him.

No, no, no.

Yeah.

I think the tone I got from his emails back and forth was he felt like he'd been saying this and he wasn't expressing it properly.

He wasn't feeling that he had something he wanted to say and he was saying it, but he didn't know whether it was getting through.

It had not been getting through.

Yeah.

Yeah.

And then I think that's he's not upset by any means.

He just wants to, you know, that was what he was expressing and whether I, if I, if I, and so he asked me and I went, well, this is what I think.

And he goes, well, why aren't you doing this?

And I went, well, I don't know.

I'll ask him.

Yeah.

I probably have it wrong.

And I did have it wrong.

Yeah.

Basically, if, if there were a way to do network based 50 millisecond response time, I'd say great.

Yeah.

But we can't consistently do that.

No.

I have a question before we end this off.

Yeah.

Where's the close button on the J wiki viz viewer?

Cause I keep having to exit J to get it down.

Uh, Ro that sucks.

Um, it sounds to me as if the WD show P show, which takes as a parameter, either maximized or full screen wrong for your machine, which is one of the problems.

So I started checking the version of J to decide whether to use full screen or maximized.

And it sounds like I'm doing it wrong.

I'm using beta seven, 9.

5 beta seven right now.

Okay.

Whatever you're using 9.

5 beta seven.

All right.

Um, so if I'm using WDP show full screen for you, I should be using maximized.

And if I'm using maximized, I should be using full screen.

And I will dig into that because I've got a windows box I could experiment with.

But what happens is if you use the wrong one, you get a fully maximized window with no Chrome.

And the Chrome is where the close box is.

Maybe I can get, do it from the task bar.

Then I can't get, oh yeah, for sure.

Well, I actually know you'd be closing all of J.

There's a, is a, on the task bar, if I highlight over the J session, there's two little icon size windows that pops up and each of those has a closed and I can hit the close button on the J wiki biz.

But you shouldn't have to.

Right.

That works.

Shouldn't have to.

All right.

Um, all right.

I need to, I need to dig into that.

I, I wish I understood what was going on there.

Well, the experience I've got on, on the Mac and it's J 9.

4.

2, which, so it's the commercial version.

Um, I, I get the Chrome, so I just click on the red dot and it disappears.

Yeah.

Yeah.

That's, that's what's supposed to happen on all platforms, but clearly I'm doing something wrong.

So I will dig into that role.

Thank you very much.

That that's really obnoxious to have an application.

But cannot be closed without terminating something important.

Well, you're using the beta suck it up.

Well, I did.

I have been, I just, I asked.

But actually maybe you're right.

Maybe I should just put an explicit close button in the body of the window.

And because I'm not going to be, I may not be able to solve this.

That's entirely possible.

Well, and if a person wanted to have something maximized, there still should be a way to close it.

Right.

Yeah, I, that's a good question.

I don't.

On the Mac, if you maximize to eliminate Chrome, you can get the Chrome back by moving the cursor up to the top of the physical screen.

Yeah, you can't, I don't know that it's true.

On Windows, I don't know how you get your Chrome back under windows, but if you, you got to use a taskbar.

Yeah.

But if, if you did put a button that just said close.

Yeah, that might in fact be the answer.

That might be the answer.

No matter where you are, you can just close it.

That's right.

That's what I'm thinking is rather than solving this problem, I will sidestep it.

No, I think it solves the problem and it solves it in both cases.

Yeah.

Yeah.

There's nothing wrong with having two ways to close a window.

No.

Especially when one of them is the, you know, the understood approach to, to close windows.

It's, it's the, you know, the normal way of closing a window.

You happen to have another one.

It might be worth starting to throw menus onto, onto JWIKI Vista.

Hide all of the seldom used.

Tweaks.

Yeah, there are, there are starting to be a lot of controls.

That is certainly true.

And you can put venues on through Qt, right?

Oh, absolutely.

Yeah.

Yeah.

Yeah.

And Roel, you as a user with a more diminutive screen currently, at least for your working, would probably appreciate the menu approach.

Yeah.

I should probably invest money in a real lock, but have this, this machine was so cheap.

I just couldn't resist getting it.

I know the feeling.

All right.

I think we're at time if there were overtime, unless there's something else, I'll let you guys go.

Thank you so much.

I'm good.

We'll, we'll see you next week and see if you can get a, you know, like an ethernet connection to the modems and that'll clean up a lot of the issues, I think with the, with the skipping right now.

All right.

I'll have to huddle upstairs by the bookshelf and plug in directly.

But if you do that, then I think you won't, I mean, if you're doing a demo, you want it, you want it to be smoother, right?

I'm doing a demo.

Maybe I should do it locally.

Do a screen recording with an audio, with a voiceover.

Oh, sure.

Do it that way.

Yeah.

No, feel free and then send me the video if you want me to edit it.

Yeah.

Yeah.

That might in fact be the right.

And that one advantage of that is that the video wouldn't have video thumbnails.

Yeah.

The zoom stuff.

Although we can minimize it, we can't get rid of it completely.

Yeah.

Let me think about that.

That's a possibility, but not this week.

This week I'm going to work on Steel APL.

Steel APL.

Nice.

All right, Jeff.

Take care.

Bye-bye.

Be safe.

Bye-bye, Arthur.

Bye.

Take care.