FeedWordPress 0.91(posted on 09 April 2005)

Update 2007-11-21: FeedWordPress 0.91 is now out of date. You can download the latest release — 0.991 at the time of this writing — from the project homepage.

FeedWordPress 0.91 is now available for download.

0.91 is a release that squashes a few bugs and adds a few useful features. Really, it adds enough useful features that it’s worth an increment of 0.1 rather than 0.01, but it’s not where I want FeedWordPress to be for version 1.0 yet, and it certainly hasn’t regressed to 0.10, so we’ll just call it 0.91 and grit our teeth at the vagaries of the decimal system for the time being.

Give it a whirl, and don’t hesitate to contact me with any questions, comments, applause, brickbats, etc. I’ll be updating the documentation shortly to reflect the changes since 0.9. In the meantime, here’s the highlight reel:

Features Added

  1. Category aliases: many times, when you have an aggregation website with many contributors, you will end up with several post categories that mean more or less the same thing but have slightly names (for example, Feminist Blogs ended up with “Feminism,” “Feminism, sexism, etc.”, “feministy stuff,” “Gender Issues”, “Women’s Issues”, and so on). You can now designate “aliases” for a category so that posts with their category set to either the actual name of the category or to any of its aliases will be placed in that category. To add an “alias” for a category, (1) log in to the WordPress Dashboard and (2) go to Manage –> Categories, (3) click the “Edit” link for the category that you want to add the alias for, and (4) add a line like the following, on a line by itself, to the “Description” field:

    a.k.a.: Other category name

    (Of course, you should substitute the actual alias for “Other category name”.)

  2. Hard-coded titles and taglines: By default, FeedWordPress automatically updates the Link Name and Link Description fields on a syndicated feed using the title and the tagline or description contained in the newsfeed. (So, for example, if a contributor changes the title of her blog, the new title will automatically be reflected in your “Contributors” section the next time that FeedWordPress updates its content.) If you want to stop FeedWordPress from automatically updating these fields for a particular feed (for example, because you want to use an abbreviated title or tagline for reasons of space), you can now do so by (1) going to Links –> Syndicated, (2) clicking the “Edit” link for the feed that you want to change the title or tagline for, (3) entering your hard-coded title and/or tagline in the Link Name and/or Link Description field, then (4) adding a line like this to the Link Notes section:

     hardcode name: yes

    Or a line like this:

     hardcode description: yes

    hardcode name will ensure that FeedWordPress uses your hand-entered Link Name; hardcode description will ensure that FeedWordPress uses your hand-entered Link Description; adding both lines will ensure that FeedWordPress uses both your hand-entered Link Name and your hand-entered Link Description.

  3. Ignoring syndicated categories: By default, FeedWordPress places syndicated posts in the same categories that the newsfeed they were syndicated from places them in (so, for example, if one of your contributors placed a post in the “News” category on her blog, FeedWordPress would place the syndicated post in the “News” category on your blog; if the “News” category did not yet exist, FeedWordPress would create it automatically). If you want FeedWordPress to place posts from a particular feed in only the categories you manually set, and ignore the categories on the syndication feed, you can now do so by (1) going to Links –> Syndicated, (2) clicking the “Edit” link for the feed that you want to change, then (3) adding a line like this to the Link Notes section:

     hardcode categories: yes

Bug fixes

  1. If you got an error message like this when you tried to add a new feed for syndication:

      Fatal error: Call to undefined function: curl_init() in /.../feedwordpress.php on line 1353

    … you can fix the problem by upgrading to FeedWordPress 0.91. FeedWordPress no longer depends on the cURL PHP extension. It now uses the Snoopy module, which comes packaged with all WordPress 1.5 installations, to fetch data from any URI that it needs to fetch data from.

  2. Manual editing of feed settings using the Link Notes section could introduce a couple of bugs under some web browsers–for example, by tacking a strange character (a carriage return, or "\r", specifically). This has been fixed: carriage returns (and, indeed, any trailing whitespace) will no longer screw up your settings.

    Thanks to Ray Lischner for pointing this bug out.

  3. If you used FeedWordPress to syndicate posts on a blog that you also posted to directly through the WordPress Dashboard, you may have noticed that FeedWordPress 0.9 disabled WordPress’s automatic formatting filter. (So if FeedWordPress was activated, any posts that you made directly into WordPress wouldn’t have <p>...</p> tags wrapped around paragraphs unless you hand-coded them in.) This was not so much a bug as a regrettable side-effect of the way I initially handled the problem of keeping WordPress’s filters from mangling the HTML on syndicated posts. In any case, it is now fixed: FeedWordPress stops the normal formatting filters from working only on syndicated posts (which are already in HTML), and not on any posts or pages that you posted directly to your site.

    Thanks to Alun for reminding me to fix this.

  4. If a newsfeed provides an excerpt or summary for a particular post, FeedWordPress 0.91 now uses it as the excerpt for the post on your aggregator site. (In FeedWordPress 0.8 and 0.9, the excerpt was discarded.) If no excerpt is provided, FeedWordPress will generate one from the post content.

Issues not yet fixed

This is a fallen world, and unfortunately there are a few bugs on the docket that have not yet been squashed:

  1. Blank page with no options under Options –> Syndication: This is a bug with WordPress 1.5, not with FeedWordPress. It results from a bug in WordPress 1.5’s Dashboard that makes it possible for the custom Options pages that plugins define to collide with one another. You may be able to solve the problem by deactivating all plugins except for FeedWordPress. (You can then re-activate plugins, one at a time, to see which one causes the collision.) Alternatively, you download an updated version of wp-admin/menu-header.php and upload it to the wp-admin directory of your WordPress installation. This bug will by fixed in WordPress 1.5.1.

  2. Some feeds with alternative character sets get mangled: FeedWordPress mostly handles alternative character sets fairly well, but its ability to do so depends on support from (1) MagpieRSS, (2) PHP, and (3) in PHP 4.x, the iconv library. This can make the whole affair pretty complicated, and at least one user has reported trouble (with incorrect conversion of Cyrillic characters encoded in windows-1251). This has not yet been fixed, because the multiple layers of software involved make it a non-trivial problem to solve. In the meantime, if you have been getting incorrect conversions of the character set on one of your feeds, you can work around the problem by restoring the version of wp-includes/rss-functions.php which comes with WordPress 1.5, instead of using the optional upgrade included with FeedWordPress.

posted by Rad Geek @ 2:30 pm

Replies to FeedWordPress 0.91 (7 so far…) Syndication feed

  1. Jon ("the Jester") replied:

    Hey! Thank you so much for providing a plugin to replicate/replace PlanetPlanet. You have completely saved my life!!! :)

    The only thing I’m trying to figure out is if there’s a way I can put for “the_title” some sort of [No Title] thing if a person posted without a title. This happens to be a case for a friend using LiveJournal, and while it’s a minor problem cosmetically, it’d be nice to fill it with something like a [No Title].

    Otherwise thank you again and again for the plugin. You’ve made life easier for a lot of us. :)

    posted 12 April 2005 at 3:18 pm

  2. Jeremy@theppn.org replied:

    I was in the process of doing exactly what this plugin does, you have saved me hours of work and I really like some of your design decisions.

    one thing I didnt like though was that the feeds mod date changed the real post mod date in wp, so I fiddled around a bit and added to the list of post variables

    $post['post_date'] = date(’Y-m-d H:i:s’, $post['epoch']['issued']);
$post['post_modified'] = date(’Y-m-d H:i:s’, time() );
$post['syndication_modified'] = $post['epoch']['modified'] ;
$post['post_date_gmt'] = gmdate(’Y-m-d H:i:s’, $post['epoch']['issued']);
$post['post_modified_gmt'] = gmdate(’Y-m-d H:i:s’, time() );

    which obviously I then checked freshness with, get_post_meta ( $result->id, 'syndication_modified', true );

    one of my feeds though doesnt include the date with its rss feeds, so would constantly be updating in the db, this morning I added this to the freshness check, after checking the date. (the strip slashes is because the site also has its feed badly encoding so I check with and without slashes stripped too.) I should clean up the code but I thought I would pass it back upstream sooner rather than later.

    if (!$result):
    $freshness = 2; // New content
elseif ($post['epoch']['modified'] > $old_mod_time ): // was $result-&gt;modified
    // some feeds do not provide a date at all, so we need to compare content to see if there really was an update or not.
    if ( strlen ( $result->post_title ) != strlen ( stripslashes ( $post['post_title'] ) ) ) {
        if ( strlen ( $result->post_title ) != strlen ( $post['post_title'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } elseif ( strlen ( $result->post_content ) != strlen ( stripslashes ( $post['post_content'] ) ) ) {
        if ( strlen ( $result->post_content ) != strlen ( $post['post_content'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } elseif ( strlen ( $result->post_excerpt ) != strlen ( stripslashes ( $post['post_excerpt'] ) ) ) {
        if ( strlen ( $result->post_excerpt ) != strlen ( $post['post_excerpt'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } else {
        $freshness = 0;
    }
else:
    posted 14 April 2005 at 10:34 pm

  3. peter replied:

    Great script! almost perfect :)

    I’m using yahoo news and I have the problem that some articles keep getting added every day.

    It would be great if there were a check for duplicate or similar entries.

    I’ve looked at the code, but haven’t been able to figure out how to do this.

    posted 20 April 2005 at 3:37 pm

  4. peter replied:

    I think I’ve found something useful in the following php functions:

    http://www.php.net/manual/en/function.similar-text.php

    http://www.php.net/manual/en/function.levenshtein.php

    I’ll try those in combination with the code jeremy posted above and see if that solves the problem of multiple articles on the same event.

    posted 20 April 2005 at 4:49 pm

  5. Rad Geek replied:

    I’m using yahoo news and I have the problem that some articles keep getting added every day.

    It would be great if there were a check for duplicate or similar entries.

    I’ve looked at the code, but haven’t been able to figure out how to do this.

    I could be mistaken, but my suspicion is that the problem with Yahoo! is that their feeds are broken. Some big sites remain notorious offenders on incorrect or nonexistant use of the Atom element or the RSS 2 element, which FeedWordPress uses to determine which posts have been syndicated and which have not.

    Different aggregator authors have dealt with this problem in different ways. I prefer not to deal with it (directly) because most attempts to solve the duplication problems that broken feeds cause end up complicating the code without producing reliably good results. However, if the use you want to put FeedWordPress to demands that you use broken feeds and try to strip out duplicates, then this seems like a natural place to use a syndicated_item filter (where you could check the current item against the database according to whatever criteria of similarity you want to use, and then return NULL if it seems to be a duplicate or pass the item through if it does not.

    Here’s a rather simple-minded example of how you might do this:

    <?php
/*
Plugin Name: Feed Filters
Description: Filters incoming posts
Author: Charles Johnson
Version: 2005.04.20
Author URI: http://www.radgeek.com/
*/

add_filter(’syndicated_item’, ‘filter_out_duplicates’);

// Don’t add posts with duplicate content.
function filter_out_duplicates ($item) {
    if (isset($item['content']['encoded']) and $item['content']['encoded']):
        $content = $item['content']['encoded'];
    else:
        $content = $item['description'];
    endif;

    // If capitalization or spacing are all that’s different, it’s probably
    // a duplicate.
    $content = $wpdb->escape(trim(strtolower($content)));

    $id = $wpdb->get_var(”
    SELECT ID FROM $wpdb->post
    WHERE
        TRIM(LOWER(post_content)) = ‘$content’
    “);
    if (is_null($id)) : $item = NULL; endif;
    return $item;
}
?>

    You’d put this code in its own PHP module, then install that module as a WordPress plugin like any other (copy it to wp-content/plugins and activate it from the Dashboard). Note that this is a preliminary attempt which I HAVE NOT TESTED, so I can’t guarantee that I haven’t made some kind of boneheaded error. Depending on the sources you are pulling the feeds from, you will very probably want somewhat more fine-grained logic than this for detecting duplicates, but it seems like this is the sort of start you would want to get off to.

    posted 20 April 2005 at 7:41 pm

  6. andromeda strain replied:

    The Feed Filters plugin doesn’t work. I know this may be annoying for you but using Yahoo! News as a source of feeds this duplicate problem appears continually. Ie. if you define some post categories, each referred to a keyword search in yahoo, when an article results in two queries it will be posted two times, one in each category, insted of say one time only with two categories assignments…

    posted 9 May 2005 at 1:18 pm

  7. bdblogs replied:

    I have installed feedwordpress, which takes several rss feeds and displays their posts ordered by date. it works fine.Feedwordpress only brings me the parsed data from the posts. how can I get the TITLE/URL of the weblog as well?

    posted 13 June 2006 at 6:50 am

Sponsorship