Computers – Page 20 – The PolyBlog

Installing Linux on a Netbook

March 4 2017

I have an old netbook, an ACER ASPIRE ONE. Not the most powerful of tools, and it is almost ten years old. It worked at the time for what I wanted it for — a simple laptop to use at a coffee shop, do some basic wordprocessing, maybe some web surfing, a bit of email. No graphics, no games, no media really, mostly just a portable wordprocessor.

Back in the late 90s, I was convinced there was a market for this type of product. So I went looking for it. Including a trip to Toronto to see if I could find a stripped-down laptop, not too large, preferably without a giant CD-ROM adding bulk and heft to something that didn’t need it. I tried a bunch, but all were either too big or WAY too expensive (I’m looking at you Sony!). In the end, I gave up. I tried upgrading my palm pilot to have a keyboard too, but the screen was just too small to be worth it. I ordered a wordprocessor-like device from Toronto to try it out, and for the cost of shipping, they let me try it with an option to fully return it. It was okay, but the screen was only a few lines, and while the weight and size were perfect, the power just wasn’t there.

I even exchanged emails at the time with David Pogue who was the tech writer for the NY Times to see if he knew of any devices out there that fit the bill. Nope, there weren’t any. There just wasn’t enough market to make it worthwhile to a producer — the market was heading towards desktop replacements and anything else was falling by the wayside.

Fast-forward almost ten years, and there were netbooks flooding the market. Small, compact, ran Windows, exactly what I had been looking for previously. More or less. Speed was still an issue, but the chip (the Atom processor) that made netbooks possible by reducing battery drain was also the reason for a limit to the speed and user power. But I wanted one, the price wasn’t bad, and it would finally let me be able to take notes in class (I think I was still doing my MPA at the time), surf at the coffee shop, take it with me on weekends, etc. In the end, it was good, but from time to time, I struggled to be truly “functional” because of the speed. Sometimes even scrolling through large Word docs or a PDF would almost feel like it was grinding to a halt. My needs shifted again, it was okay but not great, and the slow speed was challenging me when I could just do the same task later on the big computer at home, so instead, I would just read when I was in a coffee shop. As I said, the netbook was good, but not great, and I eventually stopped using it.

The big change that killed it was my smartphone, and later, my tablet. Suddenly, I could surf easily, quickly, and even if I couldn’t type long docs, email and surfing were WAY better on it than trying to struggle on the netbook. I have a keyboard for the tablet too, although I rarely use it.

Awhile back, I tripped over the netbook when I was doing some clean-up, and although it works, I wondered if perhaps I should add it to the recyclables sometime. Or dump it on ebay and let someone have a toy. Then, not too long ago, I was going through some news items for a tech feed I get, and there was this simple yet glorious heading — “Installing Linux on a Netbook”. I thought at first it was someone doing it as a “toy” project, no real functionality to it, more like doing it just to see if they could do it. Since I have plans to get into Raspberry PI stuff, I thought I’d read and see what it was like as a project. Imagine my surprise to find out people have done lots of this stuff already, and not just as a toy, but to strip away the Microsoft bloat and to have a real, live, functioning Linux machine that boots rapidly and runs quickly because there is no overhead.

Sure, there have been challenges in the past. Missing drivers is the obvious one…if the system isn’t widely used, Linux drivers may not exist for the pieces of hardware you’re working with. And honestly, I’d be dead in the water if that was the case. I know so little about drivers and hardware, but give me the software to configure, and I’ll make a lot of it dance. Even way back in the 90s, I was running a Windows 3.1 machine long past when everyone else had killed theirs, and it was still more than meeting my needs.

But if others had met those challenges, and we’re running Linux on a functioning netbook, and considered it actually “good”, was it worth a try? Apparently it was.

I read a few other articles, and as with all things Linux, realized the first question was going to be the same as with any machine — choose your flavour of Linux. Lubuntu looked like a working model for a number of people, including among people who were writing comparative articles with the pros and cons of each. While many of the subtle differences seemed lost on me, not being an aficionado of desktop clients in Linux, I did notice that the authors themselves had ended up their search with Lubuntu and liked it. A nice combination of power and simplicity. Puppy Linux was my second choice as an “easy for newbies” version, but I thought I would start with Lubuntu.

The steps to install are pretty straightforward in general:

Install Linux ISO on a flash drive using your main computer;
Boot your netbook from the flash drive;
Choose install;
Follow the prompts and menus.

It seemed simple enough, so I started working on Step 1.

INSTALL LINUX ISO ON A FLASH DRIVE

Kind of like the old floppy disk days of Windows or booting from a CD now, you put a bootable operating system on the drive, tell your netbook to boot from the drive instead of the hard disk, and Bob’s your uncle. Sort of.

First, you need to get it on the flash drive. You can do a lot of it manually, but a bunch of articles mentioned “Universal USB Installer” as their tool of choice, and since it seemed to automate Linux installs, I thought it sounded good to me. Free download, started the app, got it going, and it gives four questions to start:

Which distro you want…I’d already chosen Lubuntu, so easy enough;
Where the ISO file is…wait a minute, I thought it automated things? Yes, but you have to download the ISO file separately. Not what I had read, but whatever, easy enough to download myself. Grabbed the torrent version, just over 5 minutes of transfer time, no biggie, clicked back to tell the installer where the file was, and continued;
Which drive your flash drive is…lots of warnings about getting it right, but well, you are FORMATTING A DRIVE, so you always want to get that right anyway; and,
Set a persistent file size…umm, WTF does that mean? No clue. Well, I mean I do, I just don’t know if the considerations mean anything significant with Linux/Lubuntu. It says it’s optional, so I went with the default setting.

Didn’t take too long to install, just over a minute I think, with another 30 seconds in there to format the flash drive, a 4GB version I was using. I kind of expected that there would be some sort of “congrats, now test this or do that” message when it was over, but there wasn’t. Just “Done”, and when you click it, it closes. Umm, okay.

BOOT YOUR NETBOOK FROM THE FLASH DRIVE

This step seems to freak some people out for some reason, but I have no idea why. Have they never seen a bios message on their main PC before? Never had to tweak a hardware setting? Anyways, I had no concerns about this step. Which is why I went “Huh?” when it didn’t work. I went into the bios, it told me to reorder the boot priorities to change it from IDE first to Flash drive first, I did that (even though my screen says press F5 to go up and F6 to go down, and it’s actually the reverse, but I moved USB FDD to the top) and booted it up. Nope, ran Windows again. Hmm. Double-checked the priorities, nope, it accepted the changes, hmm. Looked online, found a reference that said to put USB HDD first. Except my flash drive isn’t a hard drive, it’s a flash drive. Wouldn’t FDD go first? That’s what the other articles said. Nope, it was right, it should be HDD first. Changed it to try it and everything booted fine.

CHOOSE INSTALL

This part was a bit of a surprise. There were six or seven options, including testing various hardware pieces, trying out Lubuntu without installing, etc. Nice choices. But I was already reasonably committed, others had tested things before me, so I went for the full install.

That’s where things started getting weird. I chose to try to connect to the internet to download some stuff as it went. Except I had two problems. First, it didn’t want to connect to my router for some reason. The router was there, it asked for the password, tried to connect, and nada. I’m not entirely sure of the backward compatibility of the new router but would have thought it should handle this okay. Second, I started having power problems. It was clear what was happening — after about 60 seconds of not pressing the keys, the system would go into sleep mode. I don’t remember having such draconian defaults for my power management, but it was possible near the end. Really annoying. I tried skipping it, and Lubuntu booted, but I wasn’t sure I had everything.

I played with the Lubuntu install just enough to find power management, reset everything to what I thought was more reasonable, and restarted the install, but I connected to my phone this time as a mobile hotspot — hoping that it would all work since most of the installs are small. Either that or I’m going to get warnings this month about being over my data limit. 🙂

I told it to install, pull updates as it went, and install third party software so that I would have all the basics to start with when it was done installing. And I got another surprise. It asked me if I wanted to dual-boot the machine and keep Windows as an option. At first blush, I was about to say no, and then thought, “Well, why not?”. I lose disk drive space, but I’m not talking huge media needs anyway, and if I need it, I have portable USB drives I can use not to mention lots of flash drives. And, most of the time, I just store to the cloud or email files to myself. I reset partitions, told it to proceed, and it gave me 105GB for Windows and 55GB for Lubuntu. I’m sure that’s more than enough for a small linux install, and I get to keep my Windows option in case I ever need it for something else, or someone else wants to use it without trying Lubuntu.

It took a while, completed all the install and the post-install cleanup, and other than entering language choices, keyboard layouts, local time and my name/password, it just bopped along until it finished. I rebooted, and this time I had 4-5 options, the first being Lubuntu and the last being to boot in Windows. It waited for my choice for a few seconds and when I didn’t do anything, it went to Linux by default (kind of like what happens on a SAFE reboot of Windows). The other three choices are one for advanced options for Ubuntu and two for different memory tests.

But midway through the boot, it died on me. Not crashed, I mean the power manager kicked in again and shut down. I thought it would be something in the bios settings, as it looks like it is over-riding the Operating System settings somehow. But I checked the bios and there are no settings to help. The desktop settings work exactly right. And if I press the power button, it resumes just fine. But it’s annoying and I’ll have to do some searching to get that figured out.

But other than that? It’s fully installed. It took me a little over an hour, and that included a full download of the various files so I could run the installer to create the Flash drive. Not a bad hour’s worth. Now I have a netbook that is charging, and tomorrow I’ll play with it.

I have five things on my to-do list:

Fix the power management time-out during boot;
Test the Windows boot;
Get it to connect to my network;
Install any additional software I need; and,
Browse in Firefox to see if I can get it accessing Gmail, etc.

I have to say, that was way easier than I expected.

Calculating the scope of my ebook addiction

January 24 2017

My first e-reading device was a Palm Pilot. I had an early Palm III for a short while, a cast-off as I recall, and then I got one through work and it was the Tungsten. A beautiful device, and I tried everything on it, including reading an ebook. Something old, free, likely from the Gutenberg Project. It was neat, but not something to write home about. I killed my Tungsten in a freak accident at a hockey game involving a folding guest chair, a coat pocket, and a crunching sound as I sat back down after cheering for a goal. I still remember the feeling later that night when I went to put the Tungsten on charge and saw the destroyed screen. I eventually moved on somewhat reluctantly to a combined Palm Pilot / phone (a rudimentary smart phone) called the Treo, but it was far too small to read on and I never tried.

I was still a purist. I liked paper and I have the basement full of books to prove it. Successive moves in 1997 and 1998 didn’t kill that purity but three more in 2004, 2007, and 2011 did put a damper on my paper enthusiasm. I love my books, and if I had a place to put them out all on nice shelves, I would. But I don’t have a library like that, and honestly, I don’t want to use up the space in the house to do that. It’s just not worth it considering many tend to be “read once, shelve forever”. They are on storage shelves in the basement now, and my wife was mildly concerned about the previous rate of accumulation, but the real motivator for me to change showed up around the time of the last move.

I bought a Kindle 3 — not touchscreen, just side buttons and a keyboard, and only monochrome. Very much like the early Palm III if I’m honest. But the e-ink is glorious to read. I boost the font size a bit, not quite “large print”, but I’m getting older and I quite like the reduced strain. I confess that some print books have had such small type, I tossed them quickly back on the shelf and bought an e-version instead.

Since the K3, I have become a device-agnostic e-whore. I’ll read on anything, anywhere, anytime. A couple of times when I’ve wondered if I would like a series or not, I’ve even downloaded the first volume onto my desktop, particularly when there are sales or promos, and then read the entire thing on my main computer just because I got engrossed or my other devices were charging. Not often, but 2 or 3 times. I’ve read on the Tungsten, the K3, three different tablets, at least three different phones, a laptop and a netbook.

I know, I know, many of you might say, “Never! Paper or die!”. But that’s not the test for me, because I am all about the content. I like to lose myself in the story, and if the story is good, I don’t care what format it is. Podcast, TV show, movie, animated, live theatre, magazines, I don’t care. I want to get lost in the story.

In my most arrogant days, I think the e-book partially appeals to me because it is faster. I don’t just mean that I can order a book and download in seconds, which is a factor, but that I also can read faster. I can turn pages faster. You might not think that is significant, physically, but mentally for me it seems a lot like experiences with old typewriters and early word processors. The QWERTY layout that is popular for typing was designed to prevent people from going too fast — the keys would hit each other. So it had to be fast enough to make it worthwhile, but not too fast and crash. Early wordprocessing had the opposite challenge. If it took longer than about half a second (can’t remember the actual threshold now) for the character to appear on the screen after the key was pressed, typists would stop to see if it had gone through. Their brain processed the key press and needed to see the character appear right afterward or it would stop and wait for it to appear. For me, the K3 was perfect…I could turn the page fast enough that there was no chance of me “leaving the story”.

I have left a story many times with books, particularly at the end of chapters, simply from the time it takes to manually turn the page, complete with all the sensory input that goes with it. I can feel myself stopping even for a split-second and pulling myself briefly out of the story. With the e-ink, the refresh is almost instantaneous. I am a very fast reader, and that matters to me because I read so fast.

For example, one time I was reading the novelization of one of the Spiderman movies. I finished it in just under 2 hours, about the same length of time as the movie runs. It was like watching it spool on the screen before me, just like a movie, only it was just my imagination. A totally immersive experience. Oddly enough though, that one was on paper.

But I’ve had it happen while reading e-books a lot more often — I just zip along at lightning speed. Which makes up for an odd fact — I can’t skim read on my Kindle. If I’m trying to digest some non-fiction stuff really fast for work, for example, I know how to skim read / almost-speed-read to get through the salient facts. Relax my eyes, focus on the top half of the text line, skip words that are often long adjectives, focus on verbs and nouns. I can’t do it for long texts, maybe a few pages before I start to gloss over.

But sometimes when I’m reading a novel in paper, and the author for some reason decides to drop two pages of exposition or description into an active scene, my brain goes on auto-pilot skimming forward a paragraph or two until the action starts again. It happens, particularly with new release debut authors. Yet I can’t do it on the e-ink devices or even tablets or phones. Just not the right font, I think, or maybe I just don’t see enough of the text before I have to skip to the next screen. Either way, it doesn’t work. But the speed of screen refreshes is way faster than turning pages in a paper book and keeps me reading.

The last six years with the Kindle match the statistical profile of many an e-book reader with a new device. It starts off hot and heavy — one of Amazon’s busiest download days in recent years has been Christmas day itself or Boxing Day…people with new Kindles or other devices have them all charged and ready to go, and they start downloading books for the first time.

In 2011, one of the biggest “unique features” of Amazon was the daily deals on e-books. Lots of authors putting books on promo for four or five days at a time, often for 99 cents, or just as often, many giving away book 1 of a series for free. Kind of like drug dealers giving samples to hook clients. And there was a cottage industry that was born with it…e-zines that advertised the deals. Now the market is flooded, which might sound like a good thing, but really is just info overload.

Yet myself, like many an avid reader, couldn’t say no to free books. A free guidebook for Web HTML? Sure, I’ll take that. I do webpages. A new mystery novel with a librarian as the detective? Sign me up and I’ll download right now. Cool. A new series of basic guides to a variety of topics from property law to biology, from world history to a Korean cookbook? Sure, it’s free, I’ll DL it. And I did. Over the last five years, about 850 books from Amazon. I estimate I probably bought maybe 50-75 of those, almost all except 2 or 3 were deeply discounted, and the rest were freebies. Why did I download them? Cuz they were free, and it was like crack to a reader. And they don’t take up space in my house. If I don’t want it, I’ll delete it. Maybe it will be good, and I am a voracious reader for any subject matter.

I also made the mistake of reading about the Gutenberg Project. For those who haven’t heard of it, it is basically an old book preservation project run as crowd-sourcing for books that are past their copyright period and long out of print. Lots of countries have different copyright periods, so one country might have 25 years, another 50, another 75, etc. Beyond that period, except where copyrights have been extended by other legal means, the books are now in the public domain. Of course, they didn’t have e-books 50 years ago, which means someone scans the old book and uploads it. Often they have sophisticated scanners that can scan whole books at once, even turning pages, and save as a PDF-like file.

Then the crowdsourcing comes in — anyone can join, read a page of some book, and “fix” the optical character recognition. Because of font issues, the computer might read a “the” as “be”…so you see on your screen the JPG or PDF version side-by-side with a raw text box that shows what the computer thinks is the right text. You read the image, adjust any of the text that needs to be adjusted (like a copy-editor or proofer) and say “save”. That puts that page into a larger quality control process where a Level 2 editor looks at the page and reads your text and approves it or not. Once you have “proven” reliable in your edits, you too can become a Level 2 editor or be given a harder book or your edits might even bypass Level 2 and go straight to Level 3. Level 3 looks at things like a compiled text where your page 1 and someone else’s page 2, and someone else’s page 3 are all merged together into pages 1-3. Depending on the project in each country, there may be one person at the end who reads the whole book and makes sure there are no obvious errors. Just reading it, not comparing it to the original text. Some of the edits are consistency issues…for example, did you capitalize a word that the book didn’t because you think it should be capitalized whereas someone else was literal? And when it is done and added to the inventory, any user who finds an error can flag it for an update.

You don’t have to be an editor to look at completed books though, it was just how I got sucked in. I loved the idea, partly as I worked in a library when I was in university, and the idea of books being lost to the ages is somewhat horrifying, matched with the beautiful, low-cost, crowd-sourcing of preservation by simple readers instead of a large bureaucracy. Even if you do get involved, it isn’t necessarily time-consuming. Sure, like any “hobby”, there are dedicated nutjobs where it becomes their life. But you can edit for a few minutes any time you have free space in your calendar.

And then the unthinkable happened. I discovered that they had their ENTIRE collection downloadable as DVD copies. 1000s of books on disk with a simple download. I resisted for awhile. Browsing. Being selective. There’s a lot of stuff in there I’ll never read. And then one day, for no apparent trigger, I cracked. I just downloaded the whole collection and put it in Calibre.

You would think that was enough. And it generally has been. An e-book overdose to scare me straight. But it’s been made worse by bad cyber management on my desktop. Because of some computer problems over the years, a lot of files that I have on my machine have gotten duplicated into multiple directories. For example, a collection of photos from a trip might have been saved as 2012 – Newfoundland and another copy, backed up on another disk, said Newfoundland – 2012. Not knowing which was the “good” set, I saved both for future “clean-up” and rationalization.

E-book files suffered the same fate. Multiple times. Plus I didn’t exactly know how to organize my library very well in Calibre (an e-book library management program). So I would import collection X into one library with a separate library in another. But I’d only get so far and then get sidetracked with other priorities. Which would mean I had a partially sorted library, often with 2 or 3 copies of the same file. Add in multiple e-book formats one time where I stupidly told it to create a PDF, EPUB and a MOBI copy of everything, and my library went crazy. Keeping them all as separate entries in the library.

As part of my goals for the year, I decided I wanted to read more and part of that required me to create a better set-up for Calibre with my libraries. And I discovered the clean-up problem was far worse than I imagined:

67,293 files
53.5 GB of space
25,469 titles

Ook.

I suspect that at least 75% of the 25K titles are actually duplicates or format variations under separate listings, so that leaves me with 6000 or so actual titles. Deleting Gutenberg stuff takes me down at least two thirds of that, so 2000 or so titles of itnerest, with about 1200 being non-fiction titles that are possibly throw-aways. Call it 800 titles to actually process, of which about half are ones that are basically free replacements for titles I have in paper.

So I have about 400 titles to be read that are half-way decent, possibly in three formats – EPUB, Mobi, PDF, and possibly, AZW (Amazon format).

Okay, that’s still quite the addiction. Not rehab country just yet, but still. 🙂 My goal is to have the library vastly cleaned up by June. I just have to find ways within Calibre to better eliminate duplicate titles that just happen to have separate formats or even the same file.

Looking for data visualization tools in all the wrong places…

January 9 2017

So I confess that once upon a time, I paid for some of my living expenses through the fact that I knew how to work a computer for things like graphics and formatting. No, seriously, back in the early 90s, it made me stand out from other people, the fact that I knew how to work Lotus 1-2-3, Word and WordPerfect, dBase IV and, drumroll please, Harvard Graphics. At the time, I think it was version 3.0, and while people could do very basic graphs in Lotus, the real trick was to choose a variety of simple graphs in Harvard Graphics, add your data, and voila, you were good to go. Back in ’93, for my first job at DFAIT, we did a presentation for Cabinet where all the graphs were done in HG, in colour, and there were about 30 of them. Blew the Cabinet people away, colour slides. In government. Bear in mind that most departments had barely switched over from DOS to Windows, and we gave them a 60-page deck on the future of the Asia Pacific region. Bound.

Now, of course, most people do their graphs in Excel or Powerpoint. And I wanted to do one this week. I have 29 or so “goals” that I am tracking against “Level 1” progress, and I wanted some sort of cheap visualization of how I am doing towards reaching level 1. I started in Powerpoint, which really just replicates the software functionality of Excel, and while it was decent enough, I’m not a graph tweaking expert anymore. I used to be able to get HG get up and dance and look amazing…compared to the Excel graph I did, I feel like I regressed in technology.

So, I was curious. Are there any other data visualization tools out there that people use easily and, more importantly, are free? My friend Aliza mentioned Google Charts and I confess that I had heard it mentioned in passing, but since I had little use for it, I had never looked at it.

I should step back for a second though. I *did* experiment with a WordPress plugin or two a few months back, and well, I was never too satisfied with the result. I would love a simple visualization tool that keeps my data in WordPress, doesn’t add a lot of overhead, and has lots of tweakable options. The experience reminded me a lot of that saying about things being cheap, fast, or good, and you can only pick two. I never quite found what I wanted.

But Google Charts is pretty darn near it. You basically code a function in javascript, and while that may seem a bit more daunting for some, you do get nice code snippets to just copy and paste. Not as intuitive though when it comes to customizing the look and feel…a single page with all of the options would be great, but they are spread across multiple themes. I was using a DIFF Chart, and it was the best I saw…overlapping a bar chart on a previous bar chart, or in my case, putting a light colour to show the “goal” and a darker colour to show actual progress. In the end, though, it was a struggle to get it to format properly and to fit properly as an inline-block within WordPress. It was decent, lots of power, but it renders itself each time it is called, and it wasn’t quick. Plus, for some reason, it was adding a big gap at the top and the bottom of the page. I liked the embedding, but the rest was too much.

I moved on to a variety of other options:

Zing Chart — highly complex, and the options were not as intuitively laid out as Googel Charts;
Tableau — this is only available for download, and I haven’t tried it yet;
PowerView — a Microsoft option, but not available in the version of Excel I was running;
D3.js — A live data tool, I couldn’t even claim to understand how it worked or what I was supposed to do;
Fusion charts — looks pretty solid, but linking to it from WordPress wasn’t obvious, and thus a dead end;
Visual.IS — in addition to having to create an account, it had strong links to paid versions;
High charts — the type of chart I wanted wasn’t there;
Chartist — good mentions of it elsewhere, but the website is just on GitHub, and not as friendly to access;
Piktochart — although it requires an account to be created, the real challenge was that it is also aimed at infographics more so than simple charts, and for some reason, it starts with asking you about infographics vs. presentation vs. printable, i.e. you’re choosing your design and function before you choose anything about what you want to create and put into the presentation, etc.;
Ubiq — data visualization running with MySQL, and I was hoping to avoid a database element;
Vizydrop — pulls data from a datafile (CSV JSON or Excel), which I was also hoping to avoid, but may reconsider;
Plot.ly — no clue what it did as it just kept crashing;
Datamatic — uses Google Drive, which doesn’t sound problematic, except that GD is blocked by our firewall at work, so visualizations wouldn’t work if I wanted to put them on my screen as a reminder; and,
Hohli — I really liked this one, but it seems to be geared towards replacing the interface (or lack thereof) for Google Charts, yet with 1-time creation (i.e. couldn’t seem to save or edit/re-use later).

Which put me back in the world of Excel. I did a graph, and it looks fine. I have room for future expansion of data, which could be useful (kind of stacking option for Level 2). Not exactly quite what I was hoping for in terms of quality, but maybe that is just me needing to spend more time with Excel to make it pretty. I would LOVE to be able to recreate the DIFF chart look and feel of Google Charts. But it met my need for the day (to update on my progress on goals in Goals – Weekly roundup (#2017-01)).

WordPress, missed schedule, and why I don't run my own server

January 4 2017

The other night, I was browsing my posts list in my admin panel on the website and noticed something odd — my scheduled post for yesterday morning didn’t actually post. And right next to it in the internal WordPress admin panel was a nice little red warning called “missed schedule”. I had seen this before and thought it was just a glitch, no biggie — just click publish now and it goes live. No problem with the post, it just didn’t go live when it should have. I don’t often schedule future posts when I’m writing detailed posts, although I might change the time stamp to 8:00 or 9:00 in the morning even if I’m writing it at 1:00 in the afternoon, like now. Just a small quirk I do to manage an indexing protocol that checks the time stamp, no real reason.

However, from time to time, I will write a bunch of posts all at once, and I want them spread out over several days. So I “schedule” them for future publication, usually 9:00 in the morning. So, the question was, “Why didn’t they post?”.

Running around the WordPress community

As with most glitches with WordPress, one is often not the first person to encounter it. Simply googling “WordPress Missed Schedule” brought up a whole series of people who had the same problem, posed the question online and had people respond. In most cases, their problem was slightly different in config once I scratched the surface, or the solutions were a bit different than what would seem to fit my situation. However, they did send me looking into several plugins.

I tried a plugin called WP CRONTROL that wasn’t a fix so much as a way to glean some additional diagnostic information. Namely that the small “glitch” was bigger than I thought — my CRON file was not running, and there were problems going back to August. Hundreds of behind-the-scenes happen on your website without your having to control them directly, little bots that can run from time to time, and your website software or the server software has ways to schedule them. Basically the same as your CALENDAR on your computer keeping track of reminders, etc., or even your regular WINDOWS update that does certain things at regular intervals. In this case, WordPress has a file that keeps track of all things it has to do (its own internal to-do list) called WP-CRON. I started trying to clean out the long list of expired entries but it would take forever.

So I added another cron manager called ADVANCED DATABASE CLEANER. Sure, I could have logged into the PHP admin area of my website and manually cleaned things in my database, but why would I do that when this plugin would do it for me?

Down the rabbit hole

I went down a rabbit hole after that. Lots of little tips and tricks everywhere, but few that were specific enough to help me, nor even in a good logical fashion. More like eight hundred people playing hunches and trying something on their server, some with luck, some with none. I jumped the queue while I was working through some of the options and started chatting with the Server Customer Support Rep, but honestly, their Tier 1 people are more about sales than they are tech support. They do okay, but this is way beyond them.

What I often find helpful though is that explaining the problem to them seems to crystallize it better in my own mind, and often triggers thoughts to try x or y that hadn’t occurred to me earlier or that I dismissed too soon. And it gave me a structure for understanding how to solve the problem.

A. GET YOUR DIAGNOSTICS

As I mentioned above, I used WP CRONTROL to see the list of cron activities and ADVANCED DB CLEANER to fix it a bit. WP CRONTROL also told me there was a 403 error (forbidden) on the WP-CRON file, but it wasn’t very convincing since some of the other errors might be causing it.

After going around the pole a few times, I found a great little tool called WP CRON STATUS CHECKER. It only has a few hundred downloads/installs, and it is a great little tool. The plugin doesn’t have a lot of info with it, but it adds a widget in your dashboard that tells you the status of your CRON files. It DOES nothing to fix problems but it does a quick test to see if WP-CRON can even run, and if not, tells you the status (403, etc.). I wish I had found this first. It told me for sure part of the problem was that WP-CRON was not running. And once fixed, you can deactivate it and leave it off. If you leave it activated, it will run on its own every 24 hours to check, but my site isn’t busy enough that I need that level of monitoring.

B. CHECK .HTACCESS

HTML sites have a small file in the directories called “HT access” that controls how visitors can access the individual files in the directory and subdirectory and is the second line of defense against people doing silly things like writing files and overwriting certain files on your site. On sites like WordPress, it’s the equivalent of a master security file that controls a lot of the subsequent access rights.

Some sites said to see if .HTACCESS was corrupted, but I was pretty sure mine wasn’t (I have other plugins that check the status of HTACCESS regularly, they would throw flags if it was corrupted).

What the sites FAILED to mention was that it isn’t just simple CORRUPTION that could be screwing up WP-CRON, it could be the actual proper setup is messing with it. I should have thought of that immediately when I saw the earlier 403 FORBIDDEN message in WP CRONTROL, but when I saw it again in the WP CRON STATUS CHECKER, I knew I had found my problem. Unfortunately, I did this step sixth or seventh instead of second!

The simplest way to see if .HTACCESS is your problem is to temporarily rename it something like .HTACCESS_TEMP or .HTACCESSXX. Then check the status again — if it was .HTACCESS causing the problem, the status will go green when you check after renaming it. If it is still red, move on to other options.

C. CHECK SECURITY PLUGINS

I run WordFence and iThemes, and of all the types of plugins, security ones are the most likely to cause access issues like above and conflicts with other plugins or activities. So, I temporarily deactivated them and then checked my other settings. Nope, still broken, so that was out.

D. CHECK OTHER PLUGINS

Anyone who has had a problem with WordPress, or really any CMS ever, they know that the simplest way to look for a conflict is to just deactivate all your plugins. Considering some people are running TONS of plugins, this isn’t often the favorite solution, but it’s better than turning them off and on one at a time. I deactivated them all, checked status, still broken.

E. CHECK FILE PERMISSIONS FOR WP-CRON

Of course, lots of people would say to check this one first — if WP-CRON is the problem, the “obvious” first step could be to check the file permissions on that specific file to see if the settings are wrong. Good logic, but since WP sets them in batches, the reality is that other files would likely have the wrong settings too and would have thrown red flags much earlier. Remember that this is a low-level file that only flags errors when it DOESN’T run.

For me, I temporarily changed the setting to 777, full read/write/execute and checked status again. Still broken.

F. CHECK FOLDER PERMISSIONS FOR WP ROOT DIRECTORY

I can’t say this would have occurred to me on my own. If WordPress is running, the folder must be set pretty close to right, or the red flags would be borderline catastrophic for failure. Never the less, it was still worth a try — again, I changed to full 777 rights, no change, still broken.

Why did I do all those things?

If all those had failed, I would have eliminated the three most likely internal problems:

corruption of access files;
access / permissions to the file or folder; and/or
conflict from security or other plugins.

That leaves you three other options that I’ve seen:

Try a plugin called WP MISSED SCHEDULE. It isn’t on the main WP install directory, you have to do it manually from GITHUB. I’m personally leery of installing plugins that don’t reside in the WP repository, and on top of that, the installation instructions did not seem awesome for clarity. Lots of people have tried it and had it fail; others did it and said it worked; others said, “huh?”. I also feel like if the problem was any of the first five, this plugin would not fix any of those issues.
Check with your host to see if there is a server config issue. Lots of people have said very simply, “Oh, it’s a config issue, contact your server”, but that is a bit too reactionary. If it is any of the above issues, the server people won’t know how to fix the problem either and unless you’re paying a lot for access to their tech support personnel, they’ll say, “Sorry, that’s a software issue, you’re on your own”. I used mine to get the ticket started, but I had resolved the diagnostic phase before we were done creating the ticket and I had figured out it was an .HTACCESS issue which they don’t help with anyway.
Bypass WP-CRON and create a server cron file to do it at the server level. Yep, to me that’s as advanced as it sounds. If you google, you’ll see lots of people who have written why they don’t use WP-CRON, why they turn it off, etc. Almost all of them are hardcore techies who are totally comfortable running LINUX, the type who would thumb their nose at building a Raspberry Pi project as “toys for kids”. If you go this route, and you have access to CPANEL setup, it really is a single line to tell it what to run and when, but I confess it didn’t work for me, and I didn’t want to mess around too much in that level of direct intervention

You might note that I’ve also left out the nuclear option of reinstalling WordPress from scratch. Honestly, that is pretty low on the list of probable solutions if you have done an upgrade from a previously working version and haven’t mucked around with the configuration too much recently. It *should* work, and if it doesn’t, something else is likely configured wrong, not your basic WP install. Always worth a shot of course, and many people would say to start with that in the first place. Who would say to do that? Those who did everything else, failed, tried a fresh install and it solved their problem and now they preach purity of install. 🙂

So what does this have to do with running my own server?

All of these problems? And I’m only talking about a SMALL software config issue that sent me way down the rabbit hole. If I was running my own server, I would have another 20 possible causes, all having to do with either server configuration or potentially even a hardware issue. One of the “saving” graces to paying someone else to carry the load of managing / running a server with a large hoster is that the cost/benefit ratio to me is WAY in my favour, particularly the number of times a problem like this has frustrated the hell out of me. I don’t need that added stress in my life. And if others are running the same install just fine on the big server, I know it is MY problem, not a technical screw-up that I did in the server setup. Worth it’s weight in gold.

As for me, now that I know it’s an HTACCESS challenge, I can push it to the back burner. I can work around it for now, and at some point, I’ll fix it. For now, I’m leaving it alone.

If you ended up here while searching for a solution, feel free to let me know how it’s going in the comments!

Version 3.3 of my websites: the software side

August 18 2016

As I mentioned in my earlier post this week (Version 3.3 of my websites: the technical side of things), I have tried various applications to “run” the website. Early on, I used Microsoft FrontPage. Then I tried some out of the box HTML and ASP applications with names like GPEASY or EZ107. I eventually went bigger and tried various content management system (CMS) applications like Drupal, Joomla and even MediaWiki. None of them were quite right for me.

Some people have opted for commercial platforms like Blogger or WordPress.com and those were all viable options. But I always came back to running my own site with my own set of installs. I started small, I went bigger and bigger, I went up to a WordPress install, I dropped back, and then finally, I went all in on WordPress.

I would be the first to confess that it is WAY more power than I need. People run full company sites with this thing, including full e-commerce capabilities or thousands of users with forums and discussion groups.

Me? I need a blogging platform, comments, and just a bit of capacity to show videos and images. Occasionally I get fancy and run a poll or something, but mostly it is just me blogging and the sound of a dry lonely wind blowing across my front page, maybe with digital tumbleweeds rolling along. It is not high traffic. One of my favorite Pearls Before Swine cartoons suggests the one character’s blog would have more traffic if he wrote it in crayon on a bathroom wall, and that is probably true for my site. Once in awhile I touch a nerve and garner an uptick in viewers, particularly if I’m talking about HR or development. Other than that? Mostly me prattling on into the digital ghostlands of the internet.

Yet I still want that power of WordPress. As I mentioned in the previous post, I tried separating my various musing into two camps for awhile but my approach to my site was still pretty diffuse. Blogs I would write with a specific angle in mind on one site often would end up fitting better on the other site under a different heading. Not a big deal to move, just illustrative that my approach to my separation into two blogs was not very concrete.

And when I rearranged my setup into a single blog, I thought, “YES! This is what it is all about. Everything I write in one place. Awesome!”

However, I realized over time that it wasn’t quite as awesome as I had hoped. I post in bursts and spurts, often on a single theme. Someone looking for HR info doesn’t particularly care about the latest quote I might share or what’s going on in my personal life (sob). And the balance didn’t seem quite right at times, particularly when I started adding quotes or humour. So I started thinking again about separating the blog entries into two groups.

The first group is what you see here. The more personal side of my musings:

Posts about family or just daily experiences…the truly personal side of my life;
In the same vein, but a little less “directly personal”, some of my approach to life like goal-setting, learning including photography, and spiritualism;
A science and technology theme, in that I will be writing more about astronomy (a hobby) and computers (mostly about this site, but not always); and,
A general set of posts around humour, quotes, recipes, and a catch-all around “ideas” (often involving principles and current news items).

The second group of blog entries is over on my other site (polywogg.ca), although at the time of writing, it isn’t really properly formatted and set up yet. For these ones, the groupings are not quite as defined yet either, but they are my more professional topics as a writer:

Of course, there is my main theme around Human Resources, and more specifically, my HR guide to federal competitions for jobs…I’ve considered even making this a theme for another single purpose site, but I don’t need to do that yet, and I’ve included the proceedings from an HR conference I helped organize with some friends from university;
Key insights that I can write about, since I can’t write about some aspects of my work life, are around the role of the civil service, governance, and international development;
I like writing reviews, and people seem to value them on Amazon and elsewhere, so I wanted to collect my book reviews all in one spot, along with musings on reading in general, and eventually expanding so that all my movie reviews, music reviews, and regular TV reviews are together too;
I read voraciously at times about the world of publishing, and I have some views I want to share regarding publishing, marketing, pricing, and even the role of libraries in society; and,
Finally, what probably pulls all of this theme together, I want a place to post some of my own writing — shortstories, maybe some novel excerpts, some non-fiction guides, etc.

The funny thing is that deciding all of that still left me with a major choice.

A choice of branding

Which site would be “PolyWogg” and which site would be “ThePolyBlog”? Technically, they are both PolyWogg, as they are both my musings, just on different topics. And I could certainly tell myself that if the first site had entries about my personal life and family, and I’m “PolyWogg”, it made sense that PolyWogg would be the one that was more personal.

Yet ThePolyBlog is a far more informal title than PolyWogg, and if I used the personal for PolyWogg, and put the HR stuff on ThePolyBlog, well, the guide isn’t really blog entries. Plus it would then make sense to call it “ThePolyBlog’s guide to …”, and that doesn’t sound right to me at all.

In the end it doesn’t matter of course, or at least not to any one but me, but it would be a major pain in the patootie to change it around later, so I wanted to try and get it right the first (well, actually the SECOND) time.

In the end, my wife cast the deciding vote. She felt that ThePolyBlog was more informal too, and PolyWogg did seem to be more about my writing normally. So the Blog title goes to the personal informal stuff (site 1 above) and PolyWogg goes to slightly more formal “public writing” (site 2 above).

I have to confess, I was a bit surprised by how easy everything divided up once the decision was made.

Making the move

Of course, making the move was more complicated than saying “site 1 is this”, “site 2 is that”. It was all on a GreenGeeks platform under PolyWogg.ca. I needed all that content moved to WHC.CA, and I needed it split.

In the end, I settled on a two-step move. First and foremost, I migrated EVERYTHING to ThePolyBlog temporarily. WordPress has some tools that help you move everything from one place to another, including the install, but since I couldn’t be sure the GreenGeeks site wasn’t all screwed up from the myriad of changes, I decided I would do a fresh install and just move the content. I had hoped to have both PolyWogg sites (old and new) running simultaneously and bop back and forth from one to the other while the new remained in sandbox mode, but that didn’t quite work out the way I had hoped (I would have had to play constantly with my HOSTS file, too much trouble).

So I did OLD POLYWOGG and NEW THEPOLYBLOG setups, got everything all moved over, tested, all the content fine, and then I deleted the old PolyWogg site, moved to a fresh install at NEW POLYWOGG, *and* what was really cool? I then just copied the NEW POLYBLOG content over into NEW POLYWOGG and started deleting the stuff I didn’t need from each.

I even got to play with plugins as I went to make sure everything was relatively identical. In the end, my goal is that the back-end will be relatively identical, it is just the content and names that will change. “My branding” will be relatively uniform across the two, just mild differences in colours and menus, etc.

Setting up the new WordPress sites

One of my goals in setting up the new sites was to completely clean up a bunch of old plugins that I might not need anymore while streamlining certain bells and whistles too.

As a small digression, I’m often amazed by the number of people who blog, “Hey, I use this great plugin called ACME WIDGETS”. Which is great that they want to promote something they like, but it also starts giving out information to potential hackers about how your site is set up, what it is running, etc. I break this rule myself as I say I’m running WordPress, but that isn’t that big a surprise — while it doesn’t say it on my site, anyone doing a VIEW SOURCE on any page would see WordPress references pretty fast. But I don’t feel like I should make it easy for anyone to tell them things they don’t need to know, so while I will talk about plugins, I won’t talk about their names, just about the functions I’m looking for…

Obviously, one of the first decisions people make in their design is what theme to run. Lots abound, and most add functions I don’t need — sliders, magazine setups, etc. I’m a pretty vanilla guy whose posts revolve around words. A lot of words. I was using a theme that I subscribed to a long time ago, upgraded to pro, upgraded to a new version, upgraded to the pro version, now it is available in regular free mode and another pro version. The regular free mode is more than good enough for my uses. Not quite as “clean” looking as the previous one, but lower load time. It will easily meet my needs. And free is good.
The second piece people need to think about is protection. No glove, no love, and that applies to websites too. Spam protectors, firewalls, login blockers, ongoing monitors, backups, and my personal favorite? Changing the default login page so that the bots can’t even find it to try and log in. Of course, if you screw up or forget where you hid it, you can get locked out too. Oops.
My next set of tools are around beefing up the look and feel from the default theme options to add a few bells and whistles. For me, this is mostly around adding some custom widgets, some better page navigation, maybe a tool for polls and charts, and even ones around helping identify other posts that are like the current one to help encourage people to stay on your site longer.
Of course, if you don’t write anything, there’s nothing to see. So I added some basic tools for improving the writing and editing experience, adding in shortcodes to simplify certain functions, making it easy to clone or copy posts into a new post, and even controlling how quotes look/work and adjusting how many revisions to keep in the database.
One of the hottest categories in the plugin world is social media, but I keep mine pretty basic — some simple sharing links, something to help with printing, adding in some contact forms to make it easy to contact me, improving the way links are done, and just for fun, tweaking integration with major sites like Twitter and Facebook.
I added some functionality to one site to allow me to more easily facilitate downloads of some documents, but mostly I am more about the back-end — statistics, word counts, checking for broken links.
Lastly, I add a few plugins that are usually not active, but help me with certain functions you have to do from time to time like managing a database, optimizing setups, or in the case of the “big move”, duplicating and importing another WordPress site.

The agony of defeat, the thrill of victory

One of the reasons the “move” went well is that I made a painful decision a few months ago. I finally accepted that part of the frustration of the last few years with my site has been constantly trying to integrate my large photo and video gallery into the same workspace. Sure, WHC offers galleries too, and like most hosters, block video hosting (you can import links, but the files have to be stored somewhere else). In the end, I bit the bullet.

I moved my photos and videos to SmugMug. It has been the best decision I ever made. Of course, if/when SmugMug goes under, I’m screwed, but in the meantime, it is working flawlessly. Videos, pictures, all together, easily controlled, nice layout. It just works. And allowed me then to concentrate on addressing more gaps in years processed than in figuring out which plugin or theme wasn’t working. I wish I had it working well in a self-hosted site, but it was just sucking too much time to get it right. We recently had a birthday party for a 90 year old member of our family, and I put a whack of photos all up on the site, ran the app, and voila, instant slide show for the party. Worked great.

For the first time in several years of trying, I feel like I’m organized for processing photos.

However, with the pain of the move, and the thrill of it working, I then moved into the renewed pain of fixing my WordPress site for photos. I know what you’re thinking, “Wait, you put everything on SmugMug, didn’t you?”. Of course, I did.

But some of my blogs LINK to those photos and use them in posts. This means all the old links for any photo in my OLD SITE which pointed to a page on my self-hosted site now point to dead links — I had to re-add all my photos back into my blog entries.

Mostly these were entries where, for example, I talked about my honeymoon, and what I was putting in the blog were samples of the things we saw as we went. Almost like a trip blog with pics added. I tried a couple of plugins that were designed to make it easier to link, but it wasn’t required. After testing several methods, I realized that the link available on SmugMug integrated seamlessly into WordPress just by pasting a certain form of the link. It was almost instantaneously PERFECT. Really happy with it. What I thought would take possibly weeks to fix was all done in a couple of hours, and not very strenuous either — most of that time was re-reading the original blog!

Where am I now?

This site is basically done for layout and configuration. Which is why I’m blogging again. I didn’t bother blogging on the old site since back in May, partly as I didn’t know if I was going to be able to keep the content. Now that I’m up and running again, I’d like to get the word count on this site up to the 250K mark this year. Particularly now that I’ve fixed a small formatting glitch as two conflicting plugins tried to format the same text. Bye-bye to one plugin.

Now, on to fixing ThePolyBlog, a bit more challenging for some of the plugin tweaks.

The PolyBlog

My view from the lilypads

Category Archives: Computers