Rad Geek’s Projects: FeedWordPress

FeedWordPress version 0.993 is now available for download.

There are a few new features that I am in the midst of working on for an upcoming release of FeedWordPress, but I have released version 0.993 now in order to resolve the critical compatability issue with WordPress 2.5.1. I am still doing compatibility testing to see whether there are any kinks in compatibility with WordPress 2.5.x, but upgrading to this release should eliminate the fatal error that prevented 2.5.1 users from accessing the Syndication Options and the feed settings pages from within the WordPress pages. There are some small bug fixes and the beginning groundwork for some features that will become more fleshed out in the upcoming, more feature-rich release, which aren’t worth going into in detail; besides those, here is what’s new since FeedWordPress 0.992:

• WORDPRESS 2.5.1 COMPATIBILITY: FeedWordPress should now be compatible with WordPress 2.5.1.

• WORDPRESS 2.5 INTERFACE IMPROVEMENTS: FeedWordPress’s Dashboard interface has undergone several cosmetic changes that should help it integrate better with the WordPress Dashboard interface in WordPress version 2.5.x.

• SYNDICATED POSTS CAN BE MARKED AS “PENDING REVIEW”: WordPress 2.3 users can now take advantage of WordPress’s new “Pending Review” features for incoming syndicated posts. Posts marked as “Pending Review” are not published immediately, but are marked as ready to be reviewed by an Administrator or Editor, who can then choose to publish the post or hold it back. If you want to review syndicated posts from a particular feed, or from all feeds, before they are posted, then use Syndication –> Syndicated Sites –> Edit or Syndication –> Options to change the settings for handling new posts.

• AWARE OF NEW URI FOR del.icio.us FEEDS: Previous releases of FeedWordPress already automatically split del.icio.us tags up appropriately appropriately when generating categories. (del.icio.us feeds smoosh all the tags into a single <dc:subject> element, separated by spaces; FeedWordPress un-smooshes them into multiple categories by separating them at whitespace.) Unfortunately, del.icio.us recently broke the existing behavior by changing host names for their feeds from del.icio.us to feeds.delicious.com. Version 0.993 accounts for the new host name and un-breaks the tag splitting.

If you have put off upgrading to WordPress 2.5.1 due to this bug, and plan to upgrade after installing FeedWordPress 0.993, please remember that after you upgrade WordPress, you will need to reinstall the FeedWordPress MagpieRSS upgrades in order to keep your feed parsing from getting broken.

Enjoy! As I mentioned, I’m actively working on a release, probably due sometime before the end of the month, including bug fixes and a few significant new features, so let me know about any ongoing issues that you may still have.

I wanted to post a quick note because I have received several reports of a serious compatability issue between FeedWordPress and the most recent public release of WordPress, version 2.5.1. Although FWP is broadly compatible with the initial release of WordPress 2.5, the recent update apparently eliminates an interface function that FeedWordPress used to display a box for selecting categories in the Syndication options and the feed settings pages. Since the function no longer exists in WordPress 2.5.1, it means that a fatal PHP error (shown either by a prematurely cut-off display, or in the form of a printed PHP error message) will be triggered if you attempt to use either of these pages in the WordPress Dashboard. (As far as I know, syndication will continue to work fine with WordPress 2.5.1; the error will prevent you from changing configuration settings from within the WordPress Dashboard.)

This problem should, hopefully, be something that it won’t be too hard to fix once I am able to sit down and work on it. Unfortunately, the release came after I had left to visit family in Alabama, so I can’t get to making the fix until I return home early in the upcoming week. Once I do, I’ll release an immediate compatability fix, and then return to work on a more thorough update, which I hope to release sometime during the month of May.

I’ll check back in with y’all once I’ve gotten home.

WordPress 2.5 was recently released, and as a result many FeedWordPress users have upgraded their blogs to the latest version of WordPress. I am currently in the process of testing for any compatability issues between WordPress 2.5 and the development version of FeedWordPress (0.993a); if I notice any definite problems, then I will make them high-priority bug fixes and try to push out the release of 0.993 as quickly as possible. (That probably means either tonight, or some time around the end of the month, depending on when I find any problems that I may find.) If you have tried using FeedWordPress with WordPress 2.5, either in version 0.992 or in the current trunk development version, and have noticed any problems since the upgrade that aren’t fixed by what I’m about to suggest, then please feel free to report them in the comments here or to me by e-mail, as you prefer. The most helpful bug reports are those that state, in as much detail as possible, (1) what precisely is going wrong, (2) under what conditions, (3) with what version of FeedWordPress, (4) under what version of PHP, and, if the problem is with syndicating posts, then (5) with which feeds at which specific URIs. If you are getting symptoms of a fatal error (either a printed error message or a blank screen where a page should be), then you can also help me out a lot by copying and pasting the contents of the error message into your report, or, if you have a blank screen, checking the bottom of your web server’s error logs to see if there is a PHP error report down there, and, if so, copying and pasting that.

That said, one of the most common sources of error reports when new versions of WordPress are released come not from a real compatability issue, but rather from the fact that, if you’re not careful, upgrading your copy of WordPress will downgrade your copy of MagpieRSS from the newer version shipped with FWP to the very old and busted version that WordPress continues, for whatever reason, to ship with new releases of WordPress.

Diagnosis

Here are the most common symptoms of this problem:

• Some feeds (notably, feeds produced by Blogger and other Atom 1.0 feeds) stop syndicating post contents. You get the headline of the post and nothing else.
• Some feeds (notably, those produced by blogs hosted at WordPress.com (!)) start appearing with just the capital letter A as the content of the post.
• Categories stop being properly syndicated. Everything is placed in Uncategorized or in bizarre, mashed-up categories (only one per post) that seem to contain several category names.
• Podcast attachments are no longer syndicated.

And so on, and so forth. If you notice these problems with your feeds just after you’ve upgraded your copy of WordPress, it’s probably because you need to re-install the MagpieRSS upgrade.

Cure

Here’s how you do that. In the FeedWordPress plugin directory (wp-content/plugins/feedwordpress/, relative to your WordPress installation), there is a directory called MagpieRSS-upgrade, which contains, or at least at one point contained, two files, rss.php and rss-functions.php. If you still have these files, you need to copy them to your WordPress wp-includes directory, where they will overwrite the older version of MagpieRSS that ships with WordPress. If you do not (because, for example, you moved them rather than copying them when you first installed FeedWordPress), then you can get new copies of these files by downloading the latest version of FeedWordPress, and extracting these two files from the archive.

Etiology and pognosis for the patient

The reason that this happens is that every installation of WordPress includes a very old version of MagpieRSS, the library that FeedWordPress uses to parse the feeds that it syndicates. As of the 2.5 release, WordPress still ships with a package derived from MagpieRSS 0.51, which is the same version it shipped with when I started work on FeedWordPress three years ago. This version of MagpieRSS is adequate for what WordPress needs it to do (basically, fetch headlines for the Dashboard from a select few feeds), but it was already outdated three years ago, and it is especially outdated now–it could not handle multiple categories; it could not handle enclosures, it could not translate feeds in alternate encodings; and, importantly, it cannot correctly handle Atom 1.0 feeds (now the default for Blogger feeds) or feeds with MediaRSS extensions (now the default for WordPress.com feeds). Unfortunately, since there is a version of MagpieRSS, which is loaded every time you load WordPress, it is hard to drop in a newer version which can do these things without causing errors from the collision in function and class names.

The solution I settled on was the bundled MagpieRSS upgrades, which in the past I sometimes described as optional, but which now really are mandatory if you hope to do any serious syndication in the modern environment. Users can avoid collisions by copying the upgrade so that it just overwrites the older version in wp-includes. Problem solved for the time being.

But the downside of this solution is that every time an upgrade of WordPress comes out, it comes out with older versions of the MagpieRSS package included, and when you overwrite all the files in wp-includes with the newer files from the WordPress release, one of the things you overwrite is your upgraded copy of rss.php. Meaning that, unless you remember to re-upgrade MagpieRSS every time you upgrade WordPress (something which is easy enough to forget), it breaks your syndication until you remember, or I remind you, to re-do the upgrade.

I frankly consider this a design flaw in FeedWordPress, but it’s not a flaw that is easy for me to fix. I am considering different ways of getting around it, and honestly the most likely solution at this point is probably simply to abandon MagpieRSS and package another feed parsing package (such as SimplePie) in the feedwordpress plugin directory, where upgrades to the WordPress core code cannot interfere with it. But doing that will involve pretty dramatically refactoring some of FeedWordPress’s internal workings, and that may take a while. In the meantime, if you have a working aggregator, you should probably apply this quick fix and see how many of your problems it solves.

FeedWordPress 0.992 is now available for download.

Since I’ve had to spend time either traveling or working on other projects, it’s been longer than I would have liked since the last update of FeedWordPress. This release is a rather limited one: it fixes one outstanding bug and adds one major new feature:

1. BUG RELATED TO URIS CONTAINING AMPERSAND CHARACTERS FIXED: A bug in WordPress 2.x’s handling of URIs in Blogroll links created problems for updating any feeds whose URIs included an ampersand character, such as Google News RSS feeds and other feeds that have multiple parameters passed through HTTP GET. If you experienced this bug, the most likely effect was that FeedWordPress simply would not import new posts from a feed when instructred to do so, returning a “0 new posts” response. In other cases, it might lead to unpredictable results from feed updates, such as importing posts which were not contained in the feed being syndicated, but which did appear elsewhere on the same website. This bug has, hopefully, been resolved, by correcting for the bug in WordPress.

2. NEW FEATURE - AUTHOR RE-MAPPING: I’ve been promising this one to my e-mail correspondents for a couple of months now; and I’m happy to announce that I’ve been able to polish up the preliminary implementation that I was working on late last year and make it ready for general consumption. FeedWordPress now offers a new feature in the site-wide Syndication Options (Syndication –> Options), and in the settings for each syndicated feed (Syndication –> Syndicated Sites –> Edit). You can now create re-mapping rules to determine how author names in a feed are translated into usernames within the WordPress database.

   Traditionally, what FeedWordPress has done, when adding a post, is to take the author information from the feed and search the WordPress database to determine whether there is a username with the same name or e-mail address. If it found a match, then the new post would be assigned to that user. If there was no match, then FeedWordPress would consult the settings for the feed, or the global default settings, for what to do with an unfamiliar author. These could be set by the administrator from within the WordPress Dashbard, with three options: (1) create a new user account with the same name as the author, and assign the new post to that new user; (2) filter out the post rather than adding it to the database; or (3) assign the new post to a default user (user #1 in the WordPress database, i.e. the site administrator).

   With the new re-mapping feature, you have considerably more control over what FeedWordPress does with post authorship data from the feed. In the settings for each syndicated feed (Syndication –> Syndicated Sites –> Edit), you can now define rules that tell FeedWordPress what to do with each particular author name on that feed — to filter out all posts by that author, or to assign posts by that author to any username that you like. You can also tell FeedWordPress what to do with usernames that aren’t listed in your rules, and which don’t match any of the users already in the WordPress database — to filter out the posts, or to assign them to any username you like, or to create a new username to match the new author, or to follow the default setting for new author names on all feeds. The site-wide default setting can be changed using Options –> Syndication.

   So, for example, one of my friends runs a FeedWordPress aggregator site that syndicates http://praxeology.net/blog/feed/. The problem is that, like an increasing number of WordPress blogs out there, all the posts on this blog are attributed to Administrator. For those visiting the blog directly, it’s clear that Administrator is the blogger; but when someone aggregates the blog on another website, the naming now mistakenly implies that the Administrator of the aggregator website is the author of the post. (It also means that if two or more blogs both attribute their posts to Administrator, the aggregator site will mistakenly treat all of the posts from all of those blogs as being by the same author.) With the new re-mapping feature, you could now define a rule that would attribute posts by Administrator from the praxeology.net feed to a different username — in this case, Roderick Long. Problem solved. Huzzah!

One more note before I go. I regret that I haven’t been able to develop FeedWordPress more actively than I currently am developing it, or to spend as much time handling and responding to bug reports as I would like. I originally created FeedWordPress for my own use, and made it available to others in the hope that it would be useful, so the best guarantee of getting a feature added or a bug identified and quickly fixed has always been whether it’s one that I personally encountered, or one of my friends encountered, in the course of using it. But I’m really very pleased with how much uptake and interest there has been for FeedWordPress. Ideally I would like to devote much more time to FeedWordPress development and support than I currently do. But I earn my living freelance, by the hour or by the project, and I do have to pay my rent every month, so the only way that I can keep up with this on more than a limited and casual basis is if the donations from FeedWordPress users allow me to free up the time needed to work on FeedWordPress, rather than spending the same time looking for paying gigs.

So, if you would like to see more regular upgrades and bugfixes, and more rapid replies to your support questions, I’d urge you to consider how much ongoing development and support for FWP is worth to you, and consider making a contribution through the project donation jar at http://projects.radgeek.com/feedwordpress/. It’s in your hands, and anything you can offer will help. (If I had $5 for everyone who ever sent me a FeedWordPress tech support question, I’d be flush for the next several years….)

Thanks,
-C

FeedWordPress 0.991 is now available for download.

Version 0.991 fixes some bugs that were present in version 0.99, adds a couple of minor features, reorganizes the file structure of the archive, and adds one very significant change. Let’s start with the significant change, which I am very happy to announce:

• WORDPRESS MU COMPATABILITY: FeedWordPress should now be compatible with recent releases of WordPress MU. Once FeedWordPress is made available as a plugin, each individual blog can choose to activate FeedWordPress and syndicate content from its own set of contributors.

And now on to the bug fixes:

• BUG RELATED TO INTERNATIONAL CHARACTERS IN AUTHOR NAMES FIXED: Due to a subtle incompatability between the way that FeedWordPress generated new user information, and the way that WordPress 2.0 and later added new authors to the database, FeedWordPress might end up creating duplicate authors, or throwing a critical error message, when it encountered authors whose names included international characters. This incompatability has now been fixed; hopefully, authors with international characters in their names should now be handled properly.

• media:content BUG IN MAGPIERSS FIXED: A bug in MagpieRSS’s handling of namespaced elements has been fixed. Among other things, this bug caused items containing a Yahoo MediaRSS <media:content> element (such as many of the feeds produced by wordpress.com) to be represented incorrectly, with only a capital “A” where the content of the post should have been. Feeds containing <media:content> elements should now be syndicated correctly.

• MAGPIE WARNINGS NO LONGER DISPLAYED BY DEFAULT: A number of MagpieRSS warnings or error messages that were displayed when performing an automatic update are no longer displayed, unless debugging parameters have been explicitly enabled.

Some minor bugs in the administrative interface have also been fixed.

I have received some feature requests concerning the new updating system that I introduced in FeedWordPress 0.99. Under the old system, there was a (needlessly complicated) system for having FeedWordPress poll its feeds for updates, which could be activated with a cron script on a regular schedule, thus keeping FeedWordPress up to date. In version 0.99, I replaced this with a much simpler system, in which FeedWordPress would automatically poll its feeds, according to a certain schedule, when users visited your website. (Of course, a cron script could still be used to make sure that this was activated on a regular schedule.) However, some users wrote me that they preferred that visitors not have to wait for the polling to finish before the page loads. So, I’ve added an HTTP parameter which gives you the option. If you prefer not to use WordPress’s cron-less automatic updates, you can now use the update_feedwordpress parameter instead. Here’s how it works:

• Leave Automatic Updates turned off. This will ensure that FeedWordPress polls for updates only when you tell it to do so.

• Add a line something like the following to your crontab:

  */15 * * * * curl --silent http://www.zyx.com/blog/?update_feedwordpress=1
  

  … replacing http://www.zyx.com/blog/ with the address of the blog where FeedWordPress is installed. This line will instruct FeedWordPress to poll for updates once every 15 minutes. If you have left Automatic Updates off, it will only check for updates on the 15 minute interval.

If you prefer the cron-less update method, then that’s still available; just go to Options –> Syndication and turn Automatic Updates on.

Finally, a word about the re-organization of the archive. FeedWordPress is now listed in, and hosted by, the kind folks at the WordPress Plugin Directory. In line with their style guide about how to organize archives, I have moved the main plugin files — feedwordpress.php and syndication-options.php — to the root directory of the archive, and I have relocated the MagpieRSS upgrade to a subdirectory called, creatively enough, MagpieRSS-upgrade. However, please remember that in order to take advantage of the MagpieRSS upgrade — which you will definitely want to take advantage of, unless you don’t care about Atom 1.0 feeds, proper handling of multiple categories, or several bug fixes — you must still copy the contents of the MagpieRSS-upgrade directory to the wp-includes folder of your WordPress installation.

Enjoy, and have a wonderful Thanksgiving, y’all.

If I hadn’t moved across the country recently, I probably never would have discovered a bug in how FeedWordPress handles the timestamps of syndicated posts. Fortunately, I did, and this bug is fixed in the recently-released version 0.99.

To explain what I mean, I’ll have to back up a bit, first.

There are a couple things that FeedWordPress uses date and time information for:

1. Dating new posts: When a new post comes in over a feed, FeedWordPress uses the date and time information reported by feeds to date posts appropriately in the WordPress database. This means parsing date and time information and putting the resulting timestamps into the database.

2. Checking for updates to existing posts: After a post has been imported into the WordPress database, FeedWordPress will keep checking to see whether the feed reports any updates to that post. This means comparing the last-updated timestamp on the feed to the last-updated timestamp in the database to see which version is newer.

The problem that I noticed came about because FeedWordPress was trying to do something sensible and easy to handle time zones when it was doing all this. Date/time handling in WordPress is fairly easy, once you understand what to do, but it is certainly anything but sensible, and therein lies the problem.

Part of the problem is, as usual, the stack of programs on top of which FeedWordPress has to sit. In order to get dates from the feed, FeedWordPress has to process two different human-readable date formats. RSS feeds use RFC 822 (or something like it), which FeedWordPress can convert to a Unix timestamp using the PHP strtotime() function. Atom feeds use the W3C DateTime Format, which FeedWordPress can convert to a Unix timestamp using a custom function provided by MagpieRSS, called parse_w3cdtf(). Unix timestamps supposedly do not vary by time zone: a Unix timestamp for a particular time is defined as the number of seconds between that time and 12:00 midnight on January 1, 1970 Greenwich Mean Time. So both strtotime() and parse_w3cdtf() make use of the time zone data provided by the format that they handle, and use it to convert the date-time information to a timestamp based on GMT.

So far so good. Now, when converting a syndicated item into a post for the WordPress database, FeedWordPress has to generate four different timestamps: post_date, which gives the date and time that the post was first published in the local time zone; post_date_gmt, which gives the date and time that the post was first published in Greenwich Mean Time; post_modified, which gives the local date and time that the post was last modified; and post_modified_gmt, which gives the GMT date and time that the post was last modified. These fields are stored in the WordPress database as MySQL DATETIME objects. WordPress later converts them back into Unix timestamps whenever it is necessary for formatting purposes. Here is how I did this in version 0.981:

$post['post_date'] = date('Y-m-d H:i:s',
    (!is_null($post['epoch']['issued'])
    ? $post['epoch']['issued']
    : $post['epoch']['modified']));
$post['post_modified'] = date('Y-m-d H:i:s',
    $post['epoch']['modified']);
$post['post_date_gmt'] = gmdate('Y-m-d H:i:s',
    (!is_null($post['epoch']['issued'])
    ? $post['epoch']['issued']
    : $post['epoch']['modified']));
$post['post_modified_gmt'] = gmdate('Y-m-d H:i:s',
    $post['epoch']['modified']);

The $post['epoch'] array contains the Unix timestamps that FeedWordPress got from the feed. The PHP date() function formats a Unix timestamp relative to the local time zone. The gmdate() function formats it relative to Greenwich Mean Time. Given that I need local and Greenwich representations of the date and time that the post was first published, and of the date and time that the post was last updated, this seems like the sensible thing to do. But in spite of (or because of) its seeming so sensible, this way of generating post timestamps introduces a subtle bug. More on that later.

When determining whether or not a post has been updated, FeedWordPress uses two items of information: the last-updated date and time provided by the feed, and the last-updated date and time stored in the database. In order to get these two dates into a common format so that they can be compared, FeedWordPress uses either strtotime() or parse_w3cdtf() to convert the feed’s last-updated date and time to a Unix timestamp, and it uses the MySQL function UNIX_TIMESTAMP() to get a Unix timestamp from the last-updated date and time in the database. So here is how I did this in version 0.981:

    $guid = $post['guid'];
    $result = $wpdb->get_row("
    SELECT id, guid, UNIX_TIMESTAMP(post_modified) AS modified
    FROM $wpdb->posts WHERE guid='$guid'
    ");

    if (!$result) :
        $freshness = 2; // New content
    elseif ($post['epoch']['modified'] > $result->modified) :
        $freshness = 1; // Updated content
    else :
        $freshness = 0;
    endif;

strtotime() and parse_w3cdtf() take account of time zone data; UNIX_TIMESTAMP() presumes that the time is in the local time zone. So, I just need to feed it the DATETIME object that’s in the local time zone — post_modified instead of post_modified_gmt, right? Wrong. Again, in spite of (or because of) its seeming so sensible, this way of comparing timestamps introduces a subtle bug.

Here’s what was wrong with both of these sensible steps: in any given WordPress installation, there are three potentially distinct local time zones that you have to consider:

1. The local time zone of the web server on which WordPress is running (usually set by the web server’s administrator);

2. The local time zone of the MySQL server that provides the WordPress database (usually set by the MySQL server’s administrator);

3. The local time zone set by the user in WordPress’s General Options page (set by the blog owner);

If any of these differ from each other, then the mismatch could cause problems for the way that older versions of FeedWordPress handled dates.

When the PHP date and time functions convert back and forth between human-readable formats and Unix time stamps, they do so relative to the web server’s local time zone. When MySQL converts a DATETIME object to a Unix timestamp, it does so relative to the MySQL server’s local time zone. When WordPress prepares dates and times for storage in the database, or processes them for sorting and displaying posts, it does so relative to WordPress’s local time zone, as set under General Options.

So, when FeedWordPress used the PHP date and time functions to generate post_date and post_modified, it used the wrong time zone — WordPress expects these to be local times in the time zone set under General Options, but the PHP functions use the web server’s local time zone. If the user has set a different time zone from the web server’s default time zone, then this date and time information will be incorrect. In order to get the correct time, we need to get the time zone information from the WordPress database, and then m