I took a look at this in Finnish, and I have quite a bit of feedback. I’m not sure how strongly it should be given since you mentioned that this is not beta yet, so take from this what you will. I looked from the viewpoint of stumbling upon these guides when searching online. I’m writing this with the assumption that you plan on using ChatGPT for the content generation. There are several issues:

• Falsehoods in the code examples and text. All the Gleam examples are wrong (since there’s not much Gleam for ChatGPT to know about), and not in a way that it’s just old – the examples are syntax that has to my knowledge never existed in Gleam. Several examples in Elixir are also wrong, e.g. claiming that strings in Elixir are objects (and thus you can call "FOO".downcase), and that strings can be concatenated with + but it’s slow, and that’s why Elixir uses <>. In the introduction it said that “other languages using the Erlang VM are Ruby on Rails and Phoenix Framework”. The page about date parsing teaches how to use a nonexistent function. For a popular language such as JavaScript the code examples seemed to be quite accurate.
• Bad translations. When I think about it from a technological point of view, I’m actually surprised how well it works. But when I think about it as if I was a reader of a programming tutorial, it’s not good enough. Constantly having the wrong inflections makes it more difficult to understand the text, and sometimes it changes the meaning of the sentences. Several different words are used for the same concept such as “string” (which only has one acceptable translation). As an example, the name of the Elixir guide for upcasing a string is Merkitsevä nauha, which actually means “meaningful ribbon”. One page states “People who place bets may wonder what HTML analyzing means”. I don’t think the tech for automated translation is there yet; this would need a thorough review by human volunteers which might take the same amount of time as a translation in the first place.
• Disconnect between body text / footer links and code examples. For example in a C++ example, the footer linked to chrono, but the code example stored a date as three separate int variables and didn’t use the library at all. All in all the code example was quite useless. (The code also imported ctime but didn’t use it.)
• Hallucinated docs links. E.g. this page has all the links lead to 404 or not the content you’re looking for. Elixir doc links pointed to 1.12 in some places that I saw.
• Fluff. Every guide seems to have an introduction and a “deep dive” that are mostly worthless. Every one of them says something like “X is very important to learn” or “Historically X has been part of the computer science since the 1960s”, which reads very odd and is just unnecessary when it’s something like “HTML analyzing has been an integral part of creating and analyzing websites since the beginning.” (It then continues: “Before the invention of JavaScript, JavaScript could only be used to analyze static data.”) I think these introductions and conclusions should be dropped because they don’t offer anything of value.

Since you said this is nearly beta, I hope there is some way to address the above concerns, especially just the amount of misinformation. At the least I think each page should be manually reviewed before publishing.

The site itself looks nice and easy to navigate.

And this is my experience with the current LLMs. The output looks correct from further away. It’s very wordy and knows how to be convincing and confident. But when you get into the details like you need to do when you’re learning from a guide, it starts to fall down and needs manual attention.

Now on the other hand, I see you’ve both quickly made websites that work using LLMs. So there’s plenty of promise there.

I love using ChatGPT and Copilot. A year ago I would find myself getting weary writing the boilerplate for something I had done hundreds of times before. Now ChatGPT or Copilot (I am using Copilot with Emacs) can generate a really great starting point for me in seconds.

The generated code always looks right and it almost always has subtle problems.

I am not worried about becoming obsolete. I am just faster. The way I think of it is like building the pyramids. We used to throw human bodies at the problem until it was done. ML-driven tools are like bulldozers/cranes/diggers in this analogy. They can do more work in a day than I can do in a year but they are still useless without human influence.

I’ve been experimenting with Elixir code generation using ChatGPTs API (GPT4) a bit and put my results, scripts and benchmarks against advent of code problems here: GitHub - dominicletz/advent_of_chatgpt: Prototype for test based code generation with ChatGPT · GitHub

image544×329 38.2 KB

Also did a talk on that here in Berlin. Here are the slides: chatgpt_berlin_meetup - Google Präsentationen

I think ChatGPT5 is going to be really interesting, so far it seems mostly be useful for small error corrections, auto-completion and toy problems.