What are your pain points, folks? Stuff that you hate doing or dealing with, or problems you can't find a good solution to? Stuff that other people might be frustrated with, too.
I'm looking for a way to make myself valuable to other people, as a way to both help people and also earn an income to feed my family in the process.
One thing I can do really well is create reliable software to automate rote tasks, generate financial/statistical/other reports, or calculate difficult solutions. Think it can't be done without LLMs? I might surprise you!
I can go into more detail about why all the options are bad if you want. But this is the sort of problem that eats years of your life, requires advanced mathematics (digital signal processing at a minimum), and advanced linguistics, on top of being a good systems-level programmer.
@fastfinge I just so happen to be an (unemployed) machine learning researcher by trade, with advanced mathematics, linguistics, and programming skills. Maybe not systems-level programming, but I could probably find someone who does that and work with them.
Given that the first two responses I've gotten were both about accessibility, there might be more of a market for this than you think, and also, it might make a good way to demo my skills even if it isn't paid work.
@hosford42 The reason I say systems-level programming is mostly because for a text to speech system used by a blind power user, you need to keep an eye on performance. If the system crashes and the computer stops talking, the only choice the user has is to hard reset. It would be running and speaking the entire time the computer is in use, so memory leaks and other inefficiencies are going to add up extremely quickly.
From what I can tell, the ideal is some sort of formant-based vocal tract model. Espeak sort of does this, but only for the voiced sounds. Plosives are generated from modeling recorded speech, so sound weird and overly harsh to most users, and I suspect this is where most of the complaints about espeak come from. A neural network or other sort of machine learning model could be useful to discover the best parameters and run the model, but not for generating audio itself, I don't think. This is because most modern LLM-based neural network models can't allow changing of pitch, speed, etc, as all of that comes from the training data.
Secondly, the phonemizer needs to be reproducible. What if, say, it mispronounces "Hermione". With most modern text to speech systems, this is hard to fix; the output is not always the same for any given input. So a correction like "her my oh nee" might work in some circumstances, but not others, because how the model decides to pronounce words and where it puts the emphasis are just a black box. The state of the art, here, remains Eloquence. But it uses no machine learning at all, just hundreds of thousands of hand-coded rules and formants. But, of course, it's closed source (and as far as anyone can tell the source has actually been lost since the early 2000's), so goodness knows what all those rules are.
@fastfinge Reading your linked article article and this reply, I get the sneaking suspicion that HDC (hyperdimensional computing) or other one- or few-shot learning methods that are designed to factor the model into independent components that can be quickly recomposed in new ways might be appropriate. The idea would be to, as you suggest, learn the values for these components using machine learning, but also the mapping between them and the sounds produced, so that each becomes separately tunable on the fly.
HDC has the added advantage that it is great for working with "fuzzy", human-interpretable rule representations, is typically extremely efficient compared to neural nets, and even meshes well with neural nets and gradient descent-based optimization.
Do you happen to have data of any sort that could be used for training?
@fastfinge I have some ideas too on how to ensure that text isn't omitted from the outputs. The trick here would be to require the representation to map one-to-one, using an autoencoder at the phoneme level to ensure information isn't lost, plus a one-to-one phoneme to sound generator to compose the final audio
@hosford42 Sadly, this is so far outside of my expertise and abilities it's not even funny. I have an excellent handle on what's needed, and the vague shape of the path forward, but actually doing any of it is way outside of my skillset. If it was anywhere near something I could do, I would have started already. :-)
@fastfinge my thinking is that, sure, I can build a thing, just like all those other folks, but you know the actual needs it would meet firsthand. That's tremendously valuable and can make the difference between something awesome and something completely useless
@hosford42 Absolutely yes to all of the above. I can think of another 10 people on Mastodon at minimum who are also ready and willing to help where ever they can. Just none of us with the skillset to do the actual work.
What you (and others who are interested) could do to help me right off the bat:
1. Make a list of the common issues, bugs, and failure modes you see in existing systems. Split hairs where you can on this, so I know exactly what issues to design around.
2. Make a list of the features you want, and include info about how important they are to you. Pay special attention to things that distinguish a good screen reader TTS from tools designed for sighted people.
You've already given me a lot of material on both of these, which is super helpful. I just want to make sure I'm getting a complete understanding so we are not surprised by a finished product that was subtly misaligned to your needs.
@hosford42 When it comes to requirements, in general, if it can work with both the SAPI5 and NVDA addons API, it will suit the requirements of speech dispatcher on Linux and the mac API's. The important thing is that most screen readers want to register indexes and callbacks. So, for example, if I press a key to stop the screen reader speaking, it needs to know exactly where the text to speech system stopped so that it can put the cursor in the right place. It also wants to know what the tts system is reading so it can decide when to advance the cursor, get new text from the application to send for speaking, etc. I really really really wish I had a better example of how that works in NVDA than this: github.com/fastfinge/eloquence_64/blob/master/eloquence.py
@hosford42 I wish it would. Unfortunately, that code is what we use to keep Eloquence alive in the 64-bit NVDA version. So it's awful, for dozens of reasons. This...is a bit clearer? Maybe? Anyway, it's the canonical example of how NVDA officially wants to interact with a text to speech system, written by the NVDA developers themselves. Any text to speech system useful for screen reader users needs to expose everything required for someone to write code like this. Not saying you could or should; there are dozens of blind folks who can do the job of integrating any text to speech system with all of the various API's on all the screen readers and platforms. But we have to have useful hooks to do it. github.com/nvaccess/nvda/blob/master/source/synthDrivers/espeak.py
@hosford42 The sourcecode for dectalk is out there. Unfortunately, It's...legally dubious at best. It was leaked by an employee back in the day, and now the copyright status of the code is so unclear that nobody can safely use it for anything, but also nobody can demonstrate clear enough ownership to submit a DMCA and get it taken off github. GNUspeech is also pretty close to what's needed, but it won't even compile without all the NeXT development tools, I don't think. So at best all that would be is a base for something else; modernizing it would probably effectively be a complete rewrite anyway.
It looks like both DECtalk and DECtalkMini are being actively maintained, with commits as recent as 1 to 2 months ago. I was hoping the copyright for the "mini" version would be unencumbered, but no such luck. It would have to be a re-implementation from scratch using this code as a guide. That's a lot easier than implementing a new system out of nothing, though.
@hosford42 I also have no idea about any associated IP or patents, though. Wouldn't whoever does it need to be able to prove they never saw the original code, just its outputs? Otherwise you're still infringing, aren't you? In this regard, it's probably actually a bad thing that the dectalk sourcecode is so widely available.
And most of the commits seem to be about just getting it to compile on modern systems with modern toolchains. I dread to think how unsafe closed-source C code written in 1998 is.
@fastfinge hmm, patents do complicate things. I could search and see if there are patents on it, I suppose. Or I could just try to implement from scratch.
@hosford42 If you're going to reimplement something, you might be better to go with gnuspeech, as it's known to be in the GPL. At the least, it gives you a vocal model to improve on, that was coded with open research in mind, rather than proprietary code probably written for job security.
@hosford42 Also, if you enjoy comparing modern AI efforts with older rule-based text to speech systems, and listening to the AI fail hard, this text is wonderful for that. As far as I'm aware not a single text to speech system, right up to the modern day, can read this one hundred percent correctly. github.com/mym-br/gnuspeech_sa/blob/master/the_chaos.txt
But eloquence gets the closest, gnuspeech second, espeak third, dectalk fourth, and every AI system I've tried a distant last.
1. it's 150 or so years old, so a few pronunciations have changed a bit 2. the pronunciations and spellings (and hence some of the apparent mismatches) are UK English, not US English.
At a minimum, you'll have to envision skipping "r"s after vowels at the ends of words for many of these to make sense. As for the rest, I recognized a few of those from past experience with older UK English (e.g. "clerk" with an "a" sound), but a couple left me scratching my head saying "that's how people actually said or spelled it then and there?"
@hosford42@fastfinge My favorite version of this sort of thing is the kind where the pronunciation doesn't rhyme, but the spelling and the meter make it seem like it should, for example:
The Fates told Odysseus "If you write to Penelope Make sure that you use A waterproof envelope."
@dpnash@hosford42 Right, but most text to speech systems have a UK English setting. And the mistakes they're making are on things much more basic than that. For example, far too many so-called state of the art AI TTS systems can't even pronounce "Susy", "plaid", "fuchsia", and "lieutenants".
@hosford42@dpnash Compare that with the version of GNU Speech released in 1995. It still messes up "tear" and "live". But once you get past the unnatural voice, it's far more precise. And once you get used to it, much much easier to listen to at an extremely high rate of speed (4x or more) all day. All text to speech advancement from "AI" is just the wow factor of "Wow, it sounds so human!" But pronunciation...you know, the important part of actually reading text...is either the same or worse. With five thousand times the resources.
@hosford42@dpnash For example, here's Eleven Labs, the billion dollar voice AI company that's supposed to replace all voice actors forever. I used the voice builder to specifically request received pronunciation. That was not at all what I got. Aside from that, notice the incorrect "tear", pronouncing "plaid" "played", having no idea that "victual" is pronounced "viddle", and a number of other mistakes. I reran it just now, to be as fair as possible. It has not improved.
@fastfinge@hosford42 what a beautiful text. As English is a foreign language to me, I can't even dream of reaching the TTS-models' levels. It would be fun to have speakers of different English dialects record this!
