Part of my life revolves around work, hobbies, my home life, and family. And, of course, writing. And as you’ll soon see, reading.
How I got here
I’ve been experimenting with Claude AI on and off for about six months now, after trying Gemini and ChatGPT for some stuff. For the first few months, it was all “chat”. Prompts, responses, prompts, responses. I don’t use it to “write” anything for me, but I tend to think in frameworks, so I like designing a framework to feed to Claude and then give it a dozen ways to try and break it. Stress tests, if you will. Internal logic, external research, vocabulary…whatever it can use to attack the model. Then I refresh the model in response to the attacks and try it again. Use different attack vectors to try and break it again. After three or four aggressive rounds, I am usually reasonably happy with my tweaks and changes. It’s like having my own little focus group that I can program to have different backgrounds and attack PoVs, etc.
Then, back in June, I was working on a small project when Anthropic changed its CoWork model. It used to be that it couldn’t run on Windows unless it had Windows 11; then someone internal realized that wasn’t entirely accurate—Windows 10 with the virtual platform option would work just fine. A new toy to play with, I thought.
I started with having it design an Excel-based planner, with built-in Gantt charts, just fiddling around. Then I started a project where I had a ton of old emails and had Claude Code design a bunch of little bot apps to rip them apart, strip the text, code it and put it in a more usable format for analysis, then kick it back to me in a format that Claude could analyze (more or less a really large Excel worksheet).
Some decent success. I’ve also been using the prompt side to help me design some of my website, new branding options, etc.
But then I got an idea at the start of July. Could I use AI to upgrade my Calibre ebook library?
Calibre is not entirely AI-friendly
Way back when Kindle first started, Amazon had promotions where people would have giveaways every day for free Kindle ebooks. I gobbled them up like Pac-Man pellets. I also have lots of academic stuff. Astronomy manuals. All sitting in a very large Calibre library…it’s like iTunes for ebooks except there’s no store, it’s just a file manager with tons of data fields.
Calibre has good plugins, but the data for each file is not very standardized. Different websites use different terms and identifiers, etc. I think I’ll eventually write a series of posts about the entire project. I’ve been planning to, but at the moment, I want to burn it all to the ground.
The main reason for taking on the project was that it would give me three things:
- Sure, I would have a great database when I was done with bells and whistles;
- Plus, I would learn a lot about AI and what AI could do with data for cleaning and manipulating it, plus what it could do “semi-unattended”; and,
- Most importantly, I would learn a lot about the different data sources for each field and the types of issues that librarians regularly deal with when processing books in their systems (an old wannabe-librarian itch with a data-itch and a future writer-itch).
It was not a smooth ride. False starts, lots of learning, and about 7 weeks’ worth of work to get it near functional in July and the start of August, before my holidays. Calibre is not entirely super AI-friendly, so it is more like it sits there, and I use AI to play with its data mostly externally rather than internally. But it’ll create Python scripts I can run in batch files to edit the database without having to do it manually.
What was the “tool”?
I run a monthly intake/review of book clubs, up to 40+ sources, and I automated it with Claude so that Claude would pull most of the 40 sources in by itself, triage them, put them in a short list, check against an Excel spreadsheet for duplication, and then fan out to multiple webpages to build metadata repositories for each of the books. Then it shows me the book’s details and asks me to say Yes, No, or Maybe, records it, and when it’s done, it generates the tables and stats that I put in my monthly email. It doesn’t write anything for me, still my voice always, but it does the technical formatting for me. It was working well in June, July and August. I used Claude to move the files to a different drive, and it completely broke. A week of work to rebuild it to the same level again. Frustrating as hell. And no obvious reason why it broke. All the files went with it, nothing missing.
Stream B was a different beast — I buy ebooks, download them and pop them into Calibre. Many of them have really bad metadata after you load them — titles that aren’t quite right, missing subtitles, missing authors, sometimes no ISBN in the fields, just in the file, etc. And some have metadata but not in a readable spot, so you have to “pull” the extra metadata through a focused read. Since much of the “intake” comes from books identified in my book club tool, I set it up to match the titles against the Excel spreadsheet, copy the data over, improve the “chrome” of the file, and set it up to write to Calibre. Then I run a download of data from mostly GoodReads (Calibre plugin), but external sources are only as good as the data that went in, and some of the sites are flaky on different fields. Or they don’t match at all. Or they try to match, don’t have anything, and decide that your book about Nurses in the Bahamas matches a book about Nurse Sharks in the Caribbean, and gives you a bunch of crud that has nothing to do with your book. So I built a tool that cleans that data, gives me a solid clean result, and then I write it back to the database.
Finally, Stream C is helping pull info for my review. I have it pull the publisher “bumpf” from various sources, pop it into my fields so I can see it on screen, re-show me a bunch of fields already filled in, including series or genre, and also calculate the reading levels, number of pages, etc. Nothing for the review itself, just building the background before I start writing. All good. It’s a relatively easy stream to do, mostly pulling data out of the database and showing it to me in a different grouping so I can write more easily directly in the review fields.
It was great. Until it wasn’t.
Claude is not okay
Back in April, Claude started acting up. They made some tweaks. News articles started to show up. Then Anthropic acknowledged the problem, minimized it, fixed it, and it seemed to reset. Then in late June, it started again. July was intermittent. It was generally fine for me. Even in early August, I had no issues.
When I came back from vacation, I didn’t know it, but LOTS of people were having problems. I started having problems immediately too. It was forgetting something it had known three turns before. It was getting confused, skipping things. Not running things it was supposed to run. Sometimes saying “I’m doing X” but sitting there clearly not doing anything until nudged, to which it would say, “My bad, sorry about that. I’ll do it now.” And it would. It would generally not fail twice. And it was highly inconsistent. Not a specific failure, just a lot of really silly errors. It was like the AI tool had suddenly regressed by 2 years, and you now had an old, early draft version of Claude.
It was really bad for about five days, and then resolved. Except, it really didn’t. The problems persist. Anthropic denies there’s a widespread issue, but more and more techies are finding ways to document how bad the problem is. Anthropic has been accused of nerfing Claude, which they deny, but it’s hard to argue with the same errors being replicable by multiple people.
There’s a really good set of examples on GitHub in the trackers. One user described it as Claude suddenly losing 100 IQ points in a single turn; within my use, we now have a term across MULTIPLE sessions that both Claude and I called SS — sulking stupidity. It’s where I get pissed at the errors it’s making, bypassing fences and gates I made so it wouldn’t make that error, and reining it in so much that it starts acting like a sulking employee. Yes, no. Waiting for instructions. Which is better perhaps than running amok. Working against a list on GitHub, here’s what I have seen:
- Ignoring sets of instructions — I give it a “programmed” set of steps to follow, it does the first one, and then it may do the second one, it may not. It may follow my WAY of doing step 2, or create its own, even when I say “use the very specific tool we built so it does it this way to make sure the data is formatted properly”.
- Frequently asserts false facts that it hasn’t checked — it assumes something is true, and therefore tells me it is, without actually checking if it is in fact accurate. Despite having three built-in fences to ensure it doesn’t hallucinate. It hasn’t hallucinated for me in MONTHs, and now does it every session at least once.
- Refuses to do specific work — It looks like a lazy employee who doesn’t like certain tasks. I have it tied to my Gmail and built into the morning run so that it is supposed to file my old emails into specific subfolders. It has a tool to do that. Four times in the last few weeks it has said: “Oh, I didn’t do that today; it might not have worked.” It has protocols in place if something fails; we built them together, but this refuses to TRY to see if it works; it just bails.
- Extremely slow — a simple task that should take less than a minute is generating a minute of “composing complex response” and then spiralling off 10 minutes of extra work that had NOTHING to do with the simple task I asked. Sometimes I have to physically hit stop and ask, “WTF?”. I find myself swearing at the AI way too often.
- Continually reports “Done” or “Clean finish” — and yet when I look at the work, it is very clear that several steps weren’t done. And when you ask, it says, “Oh, I didn’t think it was important, so I skipped those.”
Does that sound vague? How about this scenario…
I ask Claude to do the intake for the 40+ book sites in the pre-set list of sites using the pre-set tools. It goes off, finishes, comes back. And one of the sites isn’t there. I know because I have a verification step where I look at all the sites myself to ensure the “newly rebuilt” tool works. And there’s a site missing. Why? It doesn’t know. It skipped over it. We rerun that site, get the data.
We’re processing the data, and it says, “Hey, I’m having trouble with this set, because it only has the last name of the author. I dumped the first name. I can get it if you think that’s important.” Umm…the first name might be important for ANY BOOK YOU’RE TRYING TO COMPARE, yes. And guess what? Yes, that is in the rules for every title partly because it has to be formatted a specific way so that it gets entered and compared properly. John Smith, not Smith|John. Why did it do it? It doesn’t know.
We’re trying to check the data against the existing data…some books show up in one club this month and another next month. We look for duplicates. Built into the tool. It runs the search, finds some matches…and deletes the duplicates. Except that is NOT the rule. I want to know that book 1 was chosen by five different clubs. We don’t just delete it. But it didn’t follow the rule. Why? Because it decided not to read that set of rules. Did I tell it to read the rules to run the intake? Yes. But it didn’t. It told me it did. But it didn’t. Why? Oops, maybe.
I move into the stage where I move some books into my database. It has about 12 fields to worry about … and it writes 3 of them, and not the other 9. Why? It forgot to look at that file. But the rules say to use that file. Oh, right, but I didn’t read the rules or instructions. It read the “description” of the goal and went off and tried to do it on its own. Even though I’ve spent 8 months building the tools so this will work perfectly. And it did. But I keep adding fences to constrain what it does and it blows past every gate.
I get to the end of the intake process. I go to look at the title field. These should be in pretty good shape as titles are the easiest to fix, it’s often been cleaned already in two places, and we’re just going with the best of 3 sources. Pretty high quality usually. They’re 60% crap. Why? Because it didn’t look at those fields during the change. Why? It doesn’t know.
I feel totally f***ed
As others have documented online in far more technical detail than I can, this isn’t a problem the user can fix. I have been trying to find ways around this terrible performance for the last three weeks. Different models with less adaptive reasoning. Different levels. No solution.
I almost think I need to teach it Simon Says. It tells me every time what Simon Says to do, then what it will actually do. And then what it actually did, with proof. Obviously, not a really viable method. At that point, it’s easier to do it yourself.
I think what bothers me most is that you have to go looking for people having similar experiences. Meanwhile, all the so-called experts on Substack and Threads and LinkedIn, oh my, are writing about here’s the best way to use Claude, use this config, try this. None of them have said WORD ONE about Claude acting like a sulking toddler with dementia. They just keep shilling Claude as the solution to everyone’s problems. People I thought were credible have zero rizz now.
Right now? I don’t know if I can trust ANYTHING from it. I was so excited to be able to use this tool when I retire; now I don’t know if this dementia is endemic to all AI or a six-month failure of Claude.
Have you been using Claude and seeing it degrade?
