So because I have gone temporarily off my rocker, I'm working on an #xmpp client using #dotnet 10 and WPF, with artalk-xmpp for XMPP, libsignal for #OMEMO, and prism for screen reader output where needed. I'm only an evening into it, and I can already tell that OMEMO is going to be the source of complicated, inexplicable, and unreproducible bug reports for decades to come. It's still not as overcomplicated as #matrix, though.
@ppatel Please include xmpp anyway. Users deserve more than one accessible option. I’m not territorial and would happily offer testing and what help I can.
@fastfinge I'll consider including XMPP after I've pushed v1 of Matrix. I'll make it as feature-rich as possible. I think I'll be able to convert my Windows version to the store as well.
@ppatel I’m between jobs at the moment. I’m using this project to upskill and for resume juice. There are 37 xeps and 4 rfcs I need to support. Getting xmpp to done is a year of full time work for a single developer, even with AI.
@fastfinge I've been working on a Mail client + Matrix for about 8 months now between work with clients, major health issues, and life in general. I'm just about ready for alphas for each project in a week or two. I'm still supporting the Terminal Access for NVDA addon as well. I generally budget several hours of my time for reviewing code that AI comes up with during the day.
@fastfinge I used to have to do that with pre Opus 4.6. I still have to be precise with instructions. But, with skills, reinforcing red/green TDD, and code reviews, things are quite good. One of the other reasons why I use Rust is that the errors and warnings I get from the compiler are by far the best in the industry.
@ppatel@fastfinge I think the effectiveness of TDD with LLMs is overstated. TDD only works if the tests are exercising the right thing. Any human or LLM can write a body of passing tests that prove the code works but not that the code is correct.
@jscholes Of course. That goes without saying. That's one of the reasons why I use red/green testing combined with other things. Telling the system to deliberately fail at something to build a function helps a lot. Fuzz testing also helps catch quite a few bugs. Not abstracting too much code makes things easier to review. @fastfinge
@ppatel@fastfinge A few weeks back I had a tiny problem with a CLI tool I'm building. There were two blank lines between some sections of help output and while it wasn't gonna keep me up at night, I wanted there to be only one.
The LLM wrote a test that failed as expected, and then cracked on with a fix. Specifically, it determined there was no straight forward way to eliminate the extra line break and instead wrapped the entire output of the tool in a buffer. When the program was done, it removed the extra whitespace from the buffer's value before dumping the entire thing to the screen.
The tests were all green! The evil blank line was gone!
Of course, so were all of the helpful indications of progress the tool printed as it was running; it now appeared to hang and do nothing for several minutes.
@jscholes@ppatel@fastfinge Doug Langley told me to ask this question recently which has oddly helped a lot. 'What are you the least confident in, where this code is concerned?'
@FreakyFwoof@ppatel@fastfinge That's the same sort of argument used by people calling for confidence scores when an LLM imparts information. It doesn't work because a language model has no concept of confidence.
Even if it tells you it is 100 percent assured because of the training weights or research, it doesn't make it factually true in reality.
If I watched Fox News or read something on Grokipedia, I would be 100 percent certain that I'd done it. I would have much less confidence in the information itself.
@FreakyFwoof@jscholes@ppatel@fastfinge I think the reason it does work so well is that the LLMs are usually over confident. It's sort of a weird way of pushing back and making it look at the confidence of everything and giguring out what has the least matches. All I know is, after receiving this advice, when I have used it, it does yield some valid fixes and ideas.
@Doug73 Using all these processes takes these projects from vibe coding to AI-assisted coding. This is not to say that experience doesn't help. @FreakyFwoof@jscholes@fastfinge
@Doug73@FreakyFwoof@ppatel@fastfinge Questioning an LLM's assertions is good for both your brain and the quality of the thing you're working on. AI frequently suggests certain patterns are common practice because they show up a lot in one place.
My experience is that asking it to come up with those questionable things for you is drastically less effective. Not completely ineffectual; it would be wrong to assert that about a non-deterministic system. Just not as good as when you can question them yourself.
@jscholes When you work with these systems frequently enough, you also start understanding some of the vagueries and the reasoning patterns. I like to watch the so-called thinking patterns. It becomes easier to see how circular the reasoning could be. Recognizing patterns also helps future projects. You can give better instructions or codify your own knowledge. It becomes an exercise in optimizing the way you interact with the system. @Doug73@FreakyFwoof@fastfinge
That's a serious and non-prickly question; you've been very transparent about your inability to synthesise and understand the code for these projects. So how do you know that there was any sense of correctness or coherence in what the AI said it was unsure about?
I've used this technique. Much of the time, what I get back is a credible-sounding chain of thought about aspects of the code that are quite obvious to me.
@jscholes OK so for example with Sensor Readout I said what are you least confident about where the visual aspect is concerned? Jake daily drives SR so when I pushed out the change and asked him for feedback, he said it was much better all across the app and he was already happy with it previously. He didn't know I would ask that, and he had no input into the changes. Things like that. I can't think of anything else at the very moment but I know I have continued to ask that for the rest of my apps. @ppatel@fastfinge@Doug73
@jscholes I sort of use four questions as my own guides, not for the LLM. What is it for? What does it strengthen? What does it replace? What does it allow to be done poorly? That last one is my personal check—sort of a guardrail that I use when I'm evaluating code it generates. @FreakyFwoof@fastfinge
@jscholes Oh my. That failure sounds like a doozy. Mine was Terminal Access treating every window as a terminal. This came out of a test (pre-red/green). Wrong assertions are wrong assertions. @fastfinge
@ppatel Hell, just getting an e2e test harness set up took me two hours this morning: automating a local ijabberd deploy with docker and wsl. There’s no simpler way to handle that.
