I’m using yahoo news and I have the problem that some articles keep getting added every day.

It would be great if there were a check for duplicate or similar entries.

I’ve looked at the code, but haven’t been able to figure out how to do this.

I could be mistaken, but my suspicion is that the problem with Yahoo! is that their feeds are broken. Some big sites remain notorious offenders on incorrect or nonexistant use of the Atom element or the RSS 2 element, which FeedWordPress uses to determine which posts have been syndicated and which have not.

Different aggregator authors have dealt with this problem in different ways. I prefer not to deal with it (directly) because most attempts to solve the duplication problems that broken feeds cause end up complicating the code without producing reliably good results. However, if the use you want to put FeedWordPress to demands that you use broken feeds and try to strip out duplicates, then this seems like a natural place to use a syndicated_item filter (where you could check the current item against the database according to whatever criteria of similarity you want to use, and then return NULL if it seems to be a duplicate or pass the item through if it does not.

Here’s a rather simple-minded example of how you might do this:

<?php
/*
Plugin Name: Feed Filters
Description: Filters incoming posts
Author: Charles Johnson
Version: 2005.04.20
Author URI: http://www.radgeek.com/
*/

add_filter(’syndicated_item’, ‘filter_out_duplicates’);

// Don’t add posts with duplicate content.
function filter_out_duplicates ($item) {
    if (isset($item['content']['encoded']) and $item['content']['encoded']):
        $content = $item['content']['encoded'];
    else:
        $content = $item['description'];
    endif;

    // If capitalization or spacing are all that’s different, it’s probably
    // a duplicate.
    $content = $wpdb->escape(trim(strtolower($content)));

    $id = $wpdb->get_var(”
    SELECT ID FROM $wpdb->post
    WHERE
        TRIM(LOWER(post_content)) = ‘$content’
    “);
    if (is_null($id)) : $item = NULL; endif;
    return $item;
}
?>

You’d put this code in its own PHP module, then install that module as a WordPress plugin like any other (copy it to wp-content/plugins and activate it from the Dashboard). Note that this is a preliminary attempt which I HAVE NOT TESTED, so I can’t guarantee that I haven’t made some kind of boneheaded error. Depending on the sources you are pulling the feeds from, you will very probably want somewhat more fine-grained logic than this for detecting duplicates, but it seems like this is the sort of start you would want to get off to.

http://www.php.net/manual/en/function.similar-text.php

http://www.php.net/manual/en/function.levenshtein.php

I’ll try those in combination with the code jeremy posted above and see if that solves the problem of multiple articles on the same event.

I’m using yahoo news and I have the problem that some articles keep getting added every day.

It would be great if there were a check for duplicate or similar entries.

I’ve looked at the code, but haven’t been able to figure out how to do this.

one thing I didnt like though was that the feeds mod date changed the real post mod date in wp, so I fiddled around a bit and added to the list of post variables

$post['post_date'] = date(’Y-m-d H:i:s’, $post['epoch']['issued']);
$post['post_modified'] = date(’Y-m-d H:i:s’, time() );
$post['syndication_modified'] = $post['epoch']['modified'] ;
$post['post_date_gmt'] = gmdate(’Y-m-d H:i:s’, $post['epoch']['issued']);
$post['post_modified_gmt'] = gmdate(’Y-m-d H:i:s’, time() );

which obviously I then checked freshness with, get_post_meta ( $result->id, 'syndication_modified', true );

one of my feeds though doesnt include the date with its rss feeds, so would constantly be updating in the db, this morning I added this to the freshness check, after checking the date. (the strip slashes is because the site also has its feed badly encoding so I check with and without slashes stripped too.) I should clean up the code but I thought I would pass it back upstream sooner rather than later.

if (!$result):
    $freshness = 2; // New content
elseif ($post['epoch']['modified'] > $old_mod_time ): // was $result->modified
    // some feeds do not provide a date at all, so we need to compare content to see if there really was an update or not.
    if ( strlen ( $result->post_title ) != strlen ( stripslashes ( $post['post_title'] ) ) ) {
        if ( strlen ( $result->post_title ) != strlen ( $post['post_title'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } elseif ( strlen ( $result->post_content ) != strlen ( stripslashes ( $post['post_content'] ) ) ) {
        if ( strlen ( $result->post_content ) != strlen ( $post['post_content'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } elseif ( strlen ( $result->post_excerpt ) != strlen ( stripslashes ( $post['post_excerpt'] ) ) ) {
        if ( strlen ( $result->post_excerpt ) != strlen ( $post['post_excerpt'] ) ) {
            $freshness = 1;
        } else {
            $freshness = 0;
        }
    } else {
        $freshness = 0;
    }
else:

The only thing I’m trying to figure out is if there’s a way I can put for “the_title” some sort of [No Title] thing if a person posted without a title. This happens to be a case for a friend using LiveJournal, and while it’s a minor problem cosmetically, it’d be nice to fill it with something like a [No Title].

Otherwise thank you again and again for the plugin. You’ve made life easier for a lot of us. :)