Tag Archives: website

The Fall of PolyWogg (part 1 / 4)

The PolyBlog
April 19 2021

I want to talk some more about my website, and if last September was version 5.0, then the version I had as of February of this year was probably about version 5.2. I had added a few extra features, added some functionality, and expanded the base. I had made extensive progress with my photo gallery, and overall, I hate to say it, but I was feeling pretty good about it. From January 2020 to February 2021, I had drastically upgraded a TON of stuff on the site, and I felt like it was generally under control.

Jinxing myself

If feeling like things are going well, then February was probably a karmic risk. I was spending a little extra time on my photo galleries, and I was starting to feel like I had almost reached critical mass. I had 2005, 2006, and 2007 on the site, plus “regular 2008” and most of “wedding 2008”. I hadn’t quite finished the wedding albums, but most things were working so it was only a matter of time until I did. I was about halfway through the wedding albums, with my eye starting to turn towards the honeymoon photos, and beyond.

Except I ran into a small problem with the galleries. When I went to upload photos, I basically would open an upload window in the NextGen Gallery plugin, upload all the photos, make a number of tweaks, upload the videos, make some more tweaks, save it all, and then share it to FB.

But the upload was throwing errors for some reason. It took a while to narrow down even what the error was about, but essentially it would start uploading, and regardless of the progress on any file in there, if any one file took more than 30s OR the whole process ran more than 2m, then the files would time out. Now you would think that would be easy to figure out where the problem was — obviously there was a server setting somewhere that was timing out on me. Yet when I worked with the Level 1 and 2 support people with the hosting provider, we could not adjust the settings to prevent it.

Of course, I had to work through standard initial responses. “It’s a plugin conflict” … nope, I already tried deactivating EVERYTHING else. “It’s a theme conflict” … nope, tried that too. “It’s a problem with the plugin itself” … except it was the same version as the month before, no updates, AND I have the premium version. No change in the plug-in, but suddenly my server wasn’t letting me upload consistently. We tried modifying a bunch of different variables, mostly to make it more efficient so it would complete before it timed out, but we couldn’t seem to stop the 30s and 2m timelines. The plugin developer gave me a work-around so that it would do one file at a time, no concurrent loads (the default is 6-8 at a time in segments), and it worked most of the time. Right up until a file reached 30s and then it would time out. The 2m limit was still active though.

Now, the simple solution normally would be to do much smaller batches OR upload using FTP, but both added a bunch of extra steps to the process, so I was still trying to find WHERE in the server I was timing out.

Enter the dragon

I don’t want to appear overly dramatic, but I don’t know how to avoid it. I want to be fair, but at the same time, someone screwed the pooch. With the multiple attempts to “fix” things, someone in Level 2 support got the bright idea that there was something wrong with the configuration of my WordPress install. That it was taking too much overhead, and that was why things were timing out. That’s completely unrelated, but whatever, let’s not quibble.

Anyway, they sent me an email on a Thursday morning that said basically, “Okay, I’ve run an optimization on your WordPress databases and activated compression. I know that you said you didn’t want compression on your site, but it will speed things up, and the uploads should be able to complete.” He included pics / screengrabs of the front end of my site to show that it was all working.

I got the email on my personal account while I was working, and my immediate thought was, “Oh, crap.”

First and foremost, I’ve run compression on my site and it screwed up a BUNCH of things. It took some effort to undo them. Now, I ran it at the plugin level, and he was running it at the server level, but it scared the crap out of me. If he had asked me, I would have initially said no, not a chance, but might have been able to be talked into it. Except compression on the site wouldn’t affect my uploads — that would make my site render faster for front-end facing, but it wouldn’t improve my uploads, would it? (Answer: No).

Secondly though, I was wondering about the optimization process. When you run optimization, it often asks you what you want to do as part of your optimization:

  1. Simple optimization, looking for deleted entries that need to be cleaned up, etc.;
  2. Deleting revision histories, i.e., if you made a change, the previous “x” number of versions of that post are still available if you have to revert to an earlier version;
  3. Deleting auto-save / auto-drafts, i.e. if you leave the post open in edit mode for awhile, while you’re doing something else, and don’t close properly, your last “auto-save” is sitting there;
  4. Emptying the trash;
  5. Removing old comments that are left unapproved;
  6. Removing old transients, i.e., unattached orphan bits of info;
  7. Removing ping-backs and trackbacks to other websites;
  8. Removing orphaned meta data for posts and comments; and,
  9. Removing orphaned relationship info.

Generally speaking, what you keep or delete can be “everything” or “nothing”, and every point in between. The first one (simple optimization) is generally done by everyone, without too many problems, as can removing orphaned meta data (#8), relationship info (#9), transients (#6) and the trash (#4). But the rest? That’s highly personalized. I tend to keep up to 4 revisions in my history on current files, just in case I screw something up and want to go back in time without having to do a restore from backup. Older stuff? Sure, no problem, but recent revisions? Some plugins set the cut-off at 2w, which is reasonable, but I had no idea what the support guy had run. Auto-saves? Old comments? I didn’t have anything in the pipeline at the time, but it wouldn’t be the first time in my website history if I had examples of both that I didn’t want to lose.

So I wasn’t exactly thrilled that they had run both compression and optimization without checking with me first. But, the front end was working still, so should be okay, right?

The best-laid backup plans

Later that night, I tried to login to my site and it wouldn’t let me in. I had to actually go in through the server settings, reset a plugin, and then login manually a different way. Odd. But I figured maybe the optimization had messed something up. No worries, all in, reset, all good. Except it wasn’t.

I noticed one of my pages on the front-end looked fine on the blog home page, but when you actually clicked on it to go to the main page, it threw errors. In fact, ALL of my posts were throwing an error when it was on the full page for the post. Umm…Then a few other glitches cropped up. Okay, something’s not right. Time to go to the backup and revert the current version.

As an aside, there are 4 types of backups generally for WordPress sites that are self-hosted. First, you can make a manual copy of the entire site and download it. It’s incredibly time-consuming on a large site, but you can do it. Second, you can run a plugin that will make a backup of the site and store a copy either on the server or send it to some sort of off-site cloud storage. Third, you can run an external software that will backup from the server to a third-party site. Or you can use server software to backup everything.

I had some older versions of the site fully downloaded, so that was always an option, just old. In addition, I ran a backup plugin that had some versions saved on the server. And the big option, the server software option, had daily full and incremental backups. I had run into glitches previously where I had to use the server backup offered by my hoster, and they have always worked well. Even though they screwed up my site Thursday morning, there was a full backup run and available as of 8:00 a.m. that morning. Perfect, I restored from there.

Except the restore only partially worked. It gave me a restore…from almost 14m before. None of my recent work in the last year had restored, nor any of the content. Huh? Okaaaaaay, how about Wednesday? Tuesday? Monday? Nope, none of them would restore properly. Nor would the full-download version of the backup or the plugin ones stored on the site. None of the backups would complete. F***.

I ended up dealing with a CSR who was actually decent, and he figured out that for some reason, the caching software that all of their servers were running, Litespeed, was interfering with the restore. In essence, it was telling the restore that the files were already there, so it wasn’t restoring all of them. Crap. That caching software can’t be disabled. It is at the full server level.

Between the CSR and myself, we managed to run a series of “manual” restores on the Thursday morning version and in the end, we got all of the data back. It wasn’t accomplished by the rules according to Hoyle, but it was done. Whew.

Enter the gremlins

The backup completed, and I seemingly had everything back, but then I started to notice some gremlins. I’d go to edit an old post, and the photos I had linked to wouldn’t show in the editor. If I did a preview, some of them showed, I would refresh, and everything would be fine. Okay, looks like a simple caching problem. Then I would come back to the page, and something else wouldn’t work. I tried editing again, and a paragraph would be “missing”.

It looked at first like it was just “gone”, but then I would notice that the block was still there, it was just NOT SHOWING the text. So I would click on it, switch to the HTML mode, and it would give me a really weird set of codes. Blocks that were the simplest blocks of all would suddenly have almost CSS-like styling codes embedded with them. Huh? Where did THEY come from? How are they merged with my HTML content? WTF?

For my photo galleries, I had been embedding them on pages (rather than as posts), so I was running a plugin that displayed the pages as “nested” trees. It makes it way easier to manage them than as part of the standard WP page interface, yet with the gremlins, the tree wasn’t working consistently. I couldn’t move pages up or down, or I could move one and then have to totally reload the page before trying to move another. WTF x 2?

And then I started noticing some other gremlins. A couple of key plugins that I use were not working / loading at all, I started to get errors in the admin screen, which I could dismiss, and then 20m later, I’d get a similar but just slightly different one.

Finally, I ran into a problem where none of my reusable blocks were loading. Okay, that’s a problem. I tried disabling a few plugins and reinstalling them, I even tried reinstalling the core WP files. Nada. The gremlins remained. It almost looked like I had malware or a virus working its way through the install, but I don’t think so.

I think it was just that the restore had not properly restored everything.

Which left me with a choice

I tried a bunch of things, no luck. No new options from the tech support people, although I was a little bit gun-shy about their help anyway. They were the ones who nuked things in the first place.

So I gradually came to the sad and sobering realization. I had 24y worth of data, and while I had it back, the website that I had crafted over the last 17y was basically now unreliable. The foundation was shot.

I knew I could rebuild, with a massive amount of work. Did I want to?

Continue reading at My existential angst as a dead blogger (part 2 / 4).

I’m finally back in business for my websites

The PolyBlog
April 19 2021

Well, it has been almost two months since the technical support people at my website hosting partner accidentally nuked my site. The changes they made screwed up the site and the backups wouldn’t restore properly, leaving me with a huge reliability problem for some of the content. Pages that would work one minute and crash the next, posts that wouldn’t display properly, some pages or posts that disappeared altogether yet were still “active” on the site. Glitches that were annoying and not something I could leave in the site nor was I able to simply “fix” them.

I had all my data, but I needed to nuke the account and rebuild everything entirely. I rebuilt two sites that I host for others, and the content of those was relatively easy with a few annoying features, but it was the main PolyWogg.ca site that was the biggest challenge. 200 pages, 1600 posts, 1.6M words, 400 comments…all needing to be reloaded. And, as I noted, some glitches were embedded in several posts that required me to essentially reload things manually.

Working around the clock

But after two weeks of existential angst and six weeks of working at least 4h/day (at least 10h/day on weekends), and about 260 hours of work in total, my sites are back in business.

Because I had to rebuild the sites, and I didn’t want people getting daily update notices, I had to “force unsubscribe” anyone who was subscribed to my original feed for PolyWogg. Back on March 6, I sent them all emails to say, “Well, the site is down, and I’ll email you again when it comes back up”, while redirecting people to Facebook or Twitter to find me for now or in the future. I’m a little sad that many of them won’t resubscribe, but that’s the nature of the beast when running websites.

Today, I re-invited everyone to rejoin, we’ll see how many do. I now have two feeds though, one for each website.

PolyWogg.ca which includes my more formal “products”, such as my HR Guide, reviews, recipes, and astronomy-related posts;

and

ThePolyBlog.ca which includes my more informal blog posts, such as posts on Life (experiences, family, humour and computer-related topics like the website itself); The Little Grey Cells (more thoughtful topics like learning and ideas, writing and publishing, quotes, and health and spiritualism); goals; and my annual reading challenges.

If anyone else wants to subscribe by email, here are the two links:

Of course, people may prefer to access my posts through Facebook (PolyWogg.ca) or via my Twitter feed (@ThePolyBlog), as I try to post notices of new content to both feeds. I had to “force stop” the feeds during the rebuild process too, albeit one seemed to occasionally update anyways on Facebook for a few posts, but everything is back live again.

Of course, the work is never done

I have a few areas that will require more work still, but that isn’t entirely due to the website torching. As I rebuilt, I took the opportunity to redirect some of my efforts in a bit different direction. As a result, some of the solutions are not totally compatible with the previous structure. My trivia efforts had to be reset, my photo gallery is moved over to Flickr, and I’m taking a whole new approach to my PolyWogg Guide to Astronomy. Most of that change though is that as I did the other changes, I figured out better ways to handle those options too. That’s a longer term project though, and I’ll get to it over time. I don’t need to relaunch with those today.

On the other hand, I’ve done pretty well. Both sites (PolyWogg.ca and ThePolyBlog) are running about 45 plugins in total, and about 20 or so are new ways to handle the content (replacing older plugins that I was running before). I’ll blog about that in a bit more detail as the week goes on. More importantly to the reader, I have 604 posts / 45 pages / 570K words on PolyWogg.ca. Plus about 230 comments, most related to my HR Guide. Meanwhile, ThePolyBlog has 879 posts / 14 active pages / 981K words, plus about 220 comments. I’ve also streamlined the categories on both sites quite a bit.

As the planet starts to emerge from the COVID pandemic, one of the phrases being used is “Build Back Better”. I’m not 100% certain that it’s perfectly re-built, but it’s definitely better than it was, with more extensive branding for sub-areas, particularly the PolyWogg “products” like my HR Guide.

And more importantly?

It’s all the way back, baby! Onward!

Update on my rebuilding of my websites

The PolyBlog
April 17 2021

Well, it is almost two months from the fateful day when I received the email from my hosting company’s tech support that they had run a compression option on my website AND an optimization plugin. Either one alone was enough to send me into shock as compression has NEVER worked well on my site with the plugins I run, and optimization runs by deleting orphan elements in your database. Except there’s really no such thing as a universal orphan — some plugins treat autosaves that you haven’t worked on in two weeks as orphans and wants to delete them.

Sure enough, it nuked my site. Restores from backups ran into conflicts with the caching software and the result was that everything did NOT restore perfectly. Some posts would look fine one day and the next would crash when loading; some plugins on the backend wouldn’t load at all, and then an hour later worked fine. I suspect, but cannot be sure, that the fault was a combination of incomplete restores AND the caching software at the server level. In the end, the only way to rebuild in a way that was entirely reliable was to nuke everything and restore generally from zero. I had all the data, but the rebuilding still required me to pretty much update each plugin first AND to open and resave each post to make sure the data was complete. In several cases, I found glitches in the middle of posts — parts that DIDN’T restore properly — which supports my theory that the restore didn’t quite work perfectly.

Anyway, I’ve been heavy in the weeds for sometime, averaging 3-4h a night on the rebuild process, and I’m making progress. Here’s the status:

AstropontiacMCSIncPolyWoggThePolyBlogTotal
Active plugins12124847119
Total plugins13134948123
%92%92%98%98%97%
Active pages12541765
Total pages1256544126
%100%100%63%16%52%
Active posts——6014191020
Total posts——6048981502
%——~100%47%68%
Active pages+posts1256424261085
Total pages+posts1256699421628
%100%100%96%45%67%
Overall
(1/3 plugins,
2/3 content)		97%97%97%63%77%

It feels good to be over 77% done, although it’s hard to tell if the weighting is correct. I estimated 1/3 to get the plugins right, and 2/3 to rebuild the content overall. If I reweight it 50/50, it goes to 82% overall. If I reweighted the content higher, perhaps a 10/90 split, it would change to 70%. Soooo, I’m somewhere between 70-82% I guess. The stuff I’m doing now isn’t long per post, but it is REALLY mindless.

Dumb website…

The PolyBlog
February 8 2021

I like running my blog, but I confess, I’m not a big fan of the upkeep behind the scenes. Basic stuff is fine, it goes with the territory. I could pay someone to maintain my site, but since my site does not and will not ever generate revenue, I need to keep the costs as low as I can.

But some of the overhead is of my own making, sure. Or perhaps even a result of my own ambition. I’m not satisfied with using some site like Amazon Photos or Flickr, I want my own site to host my pics.

And so I have some extra admin headaches managing a large collection of photos and galleries. Like the fact that I noticed that some of my Galleries had inconsistent structures to the weburl. What does that mean? It means that the right setup is supposed to be, say for my seventh gallery of my wedding collection from 2008:

  • URL: http://www.polywogg.ca/pandagallery/yr2008/w07-bachelor party
  • Page title: 2008-W07 Bachelor Party
  • Gallery: 2008-W07 Bachelor Party

The URL had yr2008/2008-w07 (2008 listed twice), the page title was fine, and the gallery in some cases was messed up and said 2008-2g Bachelor Party because that was the old filenaming I did, and it hardcoded it into the directory structures. Is it a problem?

Yes and no. It starts off no but quickly becomes a yes when I start linking to things, and suddenly something that should link just fine, instead has a really weird file structure to it and a picture doesn’t load when it should. I was ducking and covering, letting it go, until I hit a snag tonight reading where the WordPress environment is going and realizing my little booboos are going to become bigger booboos later and it is better to fix them now.

Okay, it’s annoying, but an hour or two of quality control has me having corrected some past mistakes and I’m good to go again. Except I’m not.

I’ve got a gremlin. I don’t know what’s causing it, I don’t know when it started, I just know that for some reason, a gremlin is visiting my website.

Here’s the deal. For my gallery, I basically say “CREATE NEW GALLERY”, type in a gallery name, press enter, and then drag and drop all my photos for that gallery into the browser window. I press UPLOAD, and it puts everything onto the server and into the gallery for me. Easy peasy lemon squeasy.

It has worked this way for years. And now it doesn’t.

For some reason, my website has decided to time out in the middle of file uploads. Oh, no, not consistently. Just enough to be a PITA. It gives me an error message, saying out of 21 photos only 16 uploaded. So I note the five that it says didn’t upload. I reupload those. And it rejects one as already uploaded. Hmm…Okay, whatever, it took the four. So I go to the gallery and it says I have 24 photos. Wait, what? I had 16, I added 4 new ones, I should have 20 perhaps or 21 if the duplicate was an error. But how did I get 3 more than I started with?

Oh, right. Not all of those errors were actual errors. In fact, of the 5 it said it didn’t upload, it actually uploaded 4 of them. So I was only missing 1, right? Nope. Another two didn’t upload. No error, should have worked, but when I look at the photo, it is blank or incomplete. Umm, okay.

Plus, for some weird and wonderful world, one of the pics that didn’t upload, and isn’t in the directory, did upload the second time as it should have, but for some reason and some how, WP decided it was a partial duplicate. Not enough to block it, but enough of a duplicate to rename my filename to something like 1_image_xxxx.jpg.

Son of a firetruck.

Deep breath, dive into the rabbit hole.

Generally speaking, a problem with uploading pictures is USUALLY a problem with certain settings. And there are sites that list what the most common problems are for that error, and 8 different ways to fix them. Except none of those 8 ways apply to me. I know, because I had another problem back in December that WAS of that type and I fixed it. Or so I thought.

The app that is giving me errors is one I actually paid for, including support, so I’ve reached out to them for official suggestions. They gave me two solutions, I tried both, seemed okay, thought I was sorted, but tonight, it started throwing errors again.

Okay, debug time. What is the first step? Deactivate a bunch of other plugins to see if they’re conflicting. Yes, it works without them. Great, slowly reactivate them one by one until you find the one that’s a problem. It isn’t usually necessary to literally go 1×1, I find 5×5 is a good enough sort technique, reminiscent of advanced sorts in programming. I got about halfway through my list of plugins and hit a conflict. It stopped working. Great, that’s the problem.

Oh wait, I put another 5 in with no problem and then the next 5 conflicted again. Wait, there’s more than one conflicting? That’s unusual. Particularly for the narrow area I’m having the error in. Most plugins would never conflict with that. Weird. Okay two conflicts.

No, wait, a third. And a fourth? WTF? A fifth and a sixth? That is NOT possible. The plugin works normally, it’s a simple upload feature. It’s used by hundreds of THOUSANDS of people around the world. I know it works. WTF????

Okay, so it isn’t a real conflict. Maybe it’s a load issue? Like if I load 20 plugins, the server doesn’t want to run 20 anymore? I have 40 in total, mostly light load. They should run fine.

Sigh. Okay, I’m out of options. I reached out to the plugin again to see if they have other suggestions, and to my official server support to see if they know what would be causing it to time out after 30s when every variable in the system is at least 60s and I’m not over any obvious memory load. Hmm…

Why is this a big problem? Because it means I can’t easily upload my pictures right now.

Don’t get me wrong, there are other ways to put them on the server, including the most obvious one is to do a side-load equivalent to upload them using better software directly to the server and then have the app “copy” them from the server into the gallery. It adds about four steps to my workload to do all that, and it’s a totally separate PITA.

I can do it, but I sure don’t want to. I need to fix the original problem.

The plugin support did mention they have a beta app coming soon which would potentially indirectly fix my problem, it would just potentially slow the upload to a crawl. I CAN do that, I don’t know if I WANT to do that.

Did I mention that some days I think it’s a dumb website?

Is a photo website battle ever won?

The PolyBlog
January 24 2021

If you have archivist tendencies, combined with strong analytical props, and a digital bent, digital photo management is the field for you, my friend. As an amateur or professional, you too can find new and interesting rabbit holes to explore.

For me, I have wanted to put some of my photos online since 2005 when we bought our first digital camera. Actually, a little before that, as we had films developed and they threw in digital prints for a $1 more sometimes. And I’ve had a website since back to the dark ages before that, with the natural thought, “Could I share them on there?”.

Round 1 – Basic HTML site

Initially, round 1 of my attempt was to code my own little gallery website. I was doing all of my photos in custom FrontPage designs, and I uploaded two or three albums in HTML code. It worked, it was functional, but it wasn’t very easy to manage. More like “dump them here and you can see them”. I also wasn’t particularly sure if I had a good process behind the scenes, but when there’s only a couple of hundred, the supply side of uniformity in file management gives way to the demand side of ease of management. Throw them in a folder, call it done. Upload a few, call it done.

Eventually, I upgraded the side to a content management system, and photos were NOT an easy inclusion. Sure, I *could* include them, but it was very manual AND hard to manage all the photos on the site. They didn’t go in a subfolder, they just went in one big directory online called “media” or “images”. Not very satisfactory even with a couple of hundred images. If I uploaded another year’s worth? Meh.

Sure, I could do it through FB, but I don’t really like FB that much, and I really don’t like having all my stuff there. Plus lots of people in the family who would/could/might want to view the images aren’t ON Facebook. Or at least most of them weren’t at the time. Over time, that edge has dulled a bit, but still, it’s an issue.

Round 2 – Photo hosting site

Eventually, I decided I needed a REAL solution for online. I had enough photos that I wanted to upload that I went for an online photo site, and reviewed a bunch of sites. In the end, I went all-in on Smug Mug. It was great. I could choose a theme, I could have subdirectories. I had to manually add all my labels and descriptions, uploading my initial pics for the second time plus about 4y worth of photos, but at least I could do it. Cross-linking to my website wasn’t easy peasy, lemon squeezy, but I could do it. Sort of. More like “good enough” rather than “good”.

Except that because of the volume and use I was needing, I needed a paid account. It was only about $100 a year overall, but it was always a bit grudging payment. Here I was, paying for Smug Mug to host my pics, when I was ALREADY paying for my own site. Does that make sense to anyone? Easier than running my own photo gallery, most of the time, sure, but on principle, it annoyed me.

Round 3.1 – PolyWogg.ca

And about 6-7 years ago, the principle got to me. I wanted it on my own site. No content rules, no limitations, my own site. So I moved it to my polywogg.ca account. Great. I ran a gallery called Coppermine initially, got it going, wasn’t totally happy with it, but managed to upload a year or two. Not bad. I considered it “round 3” for online.

But it wasn’t working quite the way I wanted it to, I struggled here and there. Eventually, I decided I needed a different solution, and opted for a photo gallery called Piwigo. It had a lot of power, extra extensions, themes and plugins, like any good online community eco-system. And it handled all of my photos REALLY well. Video was still a bit of a challenge, but I could make it work. Probably.

I didn’t quite get the chance to find out. I ran into some problems about that time with an old hoster, and moved to my third hoster of my online career. I lasted about 18m with that one before they really started screwing me around. I was almost to the point where I was considering calling in a lawyer if the amounts weren’t so small. Mostly I just wanted to smack them around. Really terrible business practices and even worse support. Like them modifying my site without telling me, my finding their changes, their denying it despite the logs showing they had done it, so they deleted the logs, and my support tickets, and then deleting my complaint files (all the same support people) so the bosses wouldn’t find out what they had done. Eventually it blew up on them entirely, and a lot of people went public for awhile before the whole unit was fired and supposedly new people hired, but by that time, I was long gone.

I had moved to a medium-sized company in Canada, my current hoster, and within days of moving, I knew I felt at home. There had been a long, lingering problem on the old site, I was convinced it was a server configuration issue but had no idea how to solve it and the support people denied there were any issues. Two days after I moved to the new host, their support group reached out to me, noting the misconfiguration was likely affecting my site performance and suggesting a fix, if I was okay with it. For what I was doing, there was a small reconfiguration required, and they were proactively helping me solve it. Nice.

I reinstalled Piwigo, spent about a year getting it all up and running the way I wanted it to (after all the other changes I made to my main website were taken care of), and I consider that round 3.2. I started uploading photos. Again.

For the first year of photos, this was the fifth time uploading them somewhere (once in HTML, once in SmugMug, once in Coppermine, and now twice in Piwigo on two different hosters). Some metadata transferred, some did not. Sigh.

Round 4.1 – WordPress

I know I’m anal, but this decision really wasn’t mine. Not exactly.

You see, my site has unlimited storage and unlimited bandwidth (within the general setup of the site for speed and servers), BUT one thing that almost all small hosting packages have in common is a small note in the fine print. A limit on “inodes”.

If you don’t know what an inode is, you’re not alone, and most people who have hosting packages never even notice it. It’s basically a “file marker” in the server that tells it where to find a folder or a file. Like an index card system in a library or your old file allocation table in Windows.

For my hosting package, I am authorized up to 200K inodes. Which sounds like a lot. I have unlimited space, but for inodes, I can have say 1 folder with 199,999 files in it, or 100K folders with only 1 file in it. Neither are likely scenarios but here’s the catch. When you install WordPress, with all its little files for the core, themes and plugins, it takes about 10K files and folders. Piwigo takes about 5K all on its own. I also run two other installations of WP on my site (for other sites), and I used to have 3. Which meant just based on “installed” software, I had 35K worth of my 200K inodes already taken up.

Still, lots of room, and I cut one installation when I merged PolyBlog with PolyWogg. Back to 25K in inodes, 175K left. Plenty of space, right? Except Piwigo has a really nasty habit of generating other sized photos. So let’s say I upload 10K photos. That’s 10K inodes. Initially.

Then Piwigo generates a thumbnal (+10K), a small image (+10K), a medium image (+10K), a large image (+10K), and the original image (~0 extra). So 10K worth of images generates 50K in inodes. Umm…that’s not good. In fact, with EVERYTHING running at one point, I was up to 145K/200K used. Yikes.

Now, I can reconfigure Piwigo not to do that, and I did. I got it down to a smaller number, but the way it does it, it will always generally be twice the number of inodes.

Okay, so I had it down to a smaller functioning site, all good, right?

Well, not exactly. I still had to keep maintaining the site for admin, including improved security, etc. Plus, it isn’t exactly the most robust of software packages. I found a few things that had to be coded manually to fix, and while we found solutions (or rather the community experts helped me figure it out), it was kind of like hacking the code to make it do what I wanted. Satisfying and unsatisfying at the same time.

Round 4.2 – New WordPress versions

In the meantime, WordPress was continually evolving. It moved forward several iterations and then finally a full version upgrade, and more and more, the Piwigo solution wasn’t really integrating very well, Which is a bit of a problem.

I am, primarily, a blogger. While I have a huge site, most of the content is in pages I wrote as blog entries like this one. And I want to include more photos. Even if it is only, “Hey, here’s this photo I took yesterday at the tulip festival” before I tell some story about the experience. Yet the more WP evolved, the harder it was to integrate the photos from the site. I did it a bit manually for awhile too, passing up on some malfunctioning automated tools, but it was far from satisfying.

What I REALLY wanted was what I had wanted from the beginning. One site, one solution.

I dug back into all the photo galleries that had existed from the dawn of time, or at least it seemed like it. I found dozens that were popular and in heavy use. Some were really cool. I limited myself to those that were still compatible with the new versions of WordPress, but it was still a long list.

And almost all of them had a recurring problem. The same one I had way back at the beginning…they all use the media library as their default save location, which means by default, all the photos are stored in the same place as where you store your site header, featured images, etc. It’s nice that it’s all in one spot, but it is kind of like throwing all your books in one room and saying you have a library. No organization, no easy searching, just a long list of images to find the one you want.

The most popular one of all is one called NextGen. It has been around for years, made by Imagely, and one of the reasons people use it, other than robustness, is that it has a totally separate file structure. That presents good and bad features, but the biggest “pro” is that all of your media is stored separately. Your core media library remains untouched. One “con” is that it doesn’t handle video.

But since none of the others can handle video either, I gave it a go. Again. Sure, I say again, because I had tried it 2-3 other times previously. I always wanted all my stuff in WP, and every time I considred Smug Mug or Piwigo, I looked to see lightly if I could find a good solution in WP, and NextGen was always on the list. I could never get it to work properly.

I don’t know exactly why, but it would NEVER work right. So I’d move on, frustrated.

This time I tried it, and it worked. Out of the box, day one, first light. It just worked. What’s different from the last time? A new version of WordPress, which is significant. And I’m on an entirely different hoster that is properly configured. Does that make the difference? I don’t know. I just know it works.

Holy crap. It worked. I could integrate my GALLERY within my MAIN SITE. Holy snicker doodles.

I started uploading. I got 2005-2008 uploaded, and I hit a small wall. My site design wasn’t quite right.

Round 5 – PolyWogg 5.0

I redesigned major parts of my site in the last year. Fixed a bunch of inconsistencies, tweaked some other settings, added whole new sections. And each time I made a change, I kind of said, “Okay, I’ll figure out later how my photos fit into this new site.” I kept pushing it off.

I needed the “words” to work before I figured out how the photo and videos would work. Or if they even would.

That’s no small issue. While there are huge advantages to having everything in one site, my site has grown. It’s quite large. It has a LOT of moving parts. And the more I push in certain areas, the more I expand my content, the less functional it seemed having everything together.

A few weeks ago, I had to bite the bullet and decide. Was it going back to having two (or more) sites for PolyWogg content, maybe one for my HR guide, a separate one for regular blogging, a separate one perhaps for photos? In the end, I reframed the question. What were the REAL obstacles in having it all on one site?

The final analysis brought me to two pain points:

  1. Navigation
  2. Branding

Content management wasn’t the real issue. It was that I have a lot of content that I want to group together but branding it doesn’t really work with my standard “PolyWogg” headers. And navigation amongst the sub areas is too hard when you only have one pull-down menu for that category with a lot of sub-sub-sub-menus.

Again, as with all things in web developer, there was another option besides a separate site. I could, in theory, have separate headers for my different content as well as separate menus. There would be one master menu for the site, but once you got into more granular areas, you would move to a wholly different menu too.

Except I had tried this on multiple occasions, my theme is SUPPOSED to be able to do this, and I’ve never been able to get it to work. I’ve tried other plugins, nada. But this is what I WANTED. Maybe I could bang my head for a few weeks and see if I could cobble together a solution.

Okay, step 1, reach out for theme support. See if they had suggestions as to which other plugins would work well with the theme to do exactly what I wanted. Or tell me how to make it work with the theme. I’ve had some luck with them in the last year tweaking my theme, so I was willing to give it a go. I posted my question, aaaaand I crashed their site.

I’m not kidding. I literally crashed their support site. They fixed it and went, “Huh, what happened?”. I told them I had been posting a question, they double-checked the log, and sure enough, it was my account that killed it. My account is somehow corrupted (they don’t know how or why), and my posting killed them. They’ve tweaked it so that I can’t do it again, but my acct is still messed up somehow. I can use it, but well, I get some weird screens that others don’t get. No worries, I’ll survive.

Except in the meantime I figured I would see how far I could get on my own down this rabbit hole. I went to my theme. I enabled the features. I went to my test page, switched the header to the proper one, no change. Yep, I remember that outcome. Went back to another sub-page, made some more tweaks, misread an option, set it, reset, now NO header. WTF? Oh. Oops, misread it. Okay, reset that option, found two others that seemed to make sense that I haven’t noticed before, might be new, retested. And my header changed. All of my branding changed for that one sub-page with 2 minutes worth of work. Holy smokes.

Okay, don’t get cocky, I thought. I went to the menu area. I tried to create a new menu, copying over my old one. Told it not to put it ANYWHERE, just a dummy menu. Went in and deleted some stuff just so I could see that it was different. Went to the page that had the new header and told it to show the new menu on that page, not the regular menu. Reloaded. BAM! All of my navigation was changed for that one sub-page with 2 minutes more work.

OMG.

I did it. Exactly the way it is supposed to work, and I’m 95% sure, exactly the way I had tried it on previous occasions. But I don’t care now. It works.

Which meant I could keep my single site. Which means no separate setup. Nothing to stop me from using my existing site. All I had to do was decide on a consistent format to my layout and design for that sub-area that wouldn’t bite me in the butt later.

Because I’m not talking some small site. The average site in WordPress for people using other galleries is maybe 1000 photos. Sites that run full WooCommerce and sell products frequently don’t have more than 1000 items in their site. For me? We average 2000 photos a year, of which I post about a third. We broke a thousand mid-way into year 2, I’m over 2000 by the end of year 4, and I haven’t even included all the photos from our wedding events that year. Including the honeymoon section which is huge.

Long term, I’m estimating somewhere around 20K photos just to get caught up to now, although that may top 25K. I know professional photographers who don’t have that many. AND I haven’t even got to what I want to do for astrophotography images.

Rebooting the gallery

Since I had already uploaded the photos for 2005-2008, I didn’t have to do much to “fix” those galleries. I renamed a bunch, I changed the look and feel from an old template being deprecated to a new one, tweaked some inconsistencies here and there, and added a new video section that works really well, so I’m generally “good to go”.

Previously when I played with the first 4 years worth of gallery, I had to spend a lot of time getting them up and running. Maybe one gallery a day. I just did 32 galleries in about four days, one year per day, generally about 2 hours work while I was editing other things.

I’ve even managed to get past my previous point of progress (2005-2008), completed all the old galleries for the wedding, and I’m finally back into the truly “new” ones for being part of WordPress. I had reached 2011 at one point with SmugMug, I think, but I’m pretty happy with my early rebuild. I have a full workflow figured out, complete with Mylio as my software, and it is giving me the confidence that I have finally “turned the corner” on my go to solution. Four years down, thirteen to go, albeit the next thirteen won’t be anywhere near as fast. And alas, 2009-2011 is redoing old work. At least I’m doing it properly now.

Just don’t ask me about astrophotography yet. I don’t know HOW I’m going to organize that stuff.

Overall though, apparently the “seventh time was the charm”…I have won a decisive battle, but the war rages on.