Hi Abe-

I'm noticing a problem that seems to be specific to Scripting News and
Slashdot.  It appears that older, unread messages from those two sites
are getting overwritten when I connect to get new messages.  For
example, I currently have only 10 Slashdot messages and 3 from
Scripting News.  I *know* I'm not that current on either of them.  In
addition, there are no "deleted" messages showing up from either feed.

Yesterday afternoon I noticed that these two sites were missing from
my IMAP view, so I went back to the web interface and did an edit/save
on each of them.  It occurred to me that these are possibly the sites
that caused my previous connection failure and "IMAP SELECT" errors.
The edit/save brought them back, but they appeared as though they had
been added for the first time.

I had chalked it up to the edit/save operation, but this morning's
messages overwrote yesterday afternoon's without any edit/save on my
part, so there's something else going on here.  Any ideas?

-Matt

Matt,

The short answer: Yes, those messages are getting overwritten.  But Hep
isn't overwriting them - the Slashdot/Scripting News admins are :-)

For the long answer, I have to talk a bit about Hep's architecture.

Hep does two things:  First it establishes connections to resources on a
network.  Second, it provides access to those resources through
different protocols.  Hep uses my Yarn libary[1] to do the first thing.
   Yarn does all the work of connecting to a resouce and allowing an
application to work with that interface through a standard API.

So Hep ends up with a whole bunch of connections to various resources.
 From there, Hep does the work of making those resources available
through different protocols.  Hep doesn't have to care about what kind
of resource it's working with, since Yarn makes RSS, Maildir, the
MetaWeblog API, etc. all look the same.  Hep just translates between
IMAP (or whatever protocol you're connecting with) and yarn's
ResourceView API.

Therefore when you connect to Hep through IMAP, each mailbox represents
a connection to a specific resource.  Your "Slashdot" mailbox actually
represents a connection to the RSS file living at
http://slashdot.org/slashdot.rss.  Hep is dynamically translating
between IMAP and RSS/HTTP, so you see the current messages contained in
that RSS feed. (Although it might not be up-to-the-minute current, since
the files do get cached).

This is actually a key feature of Hep's architecture - real-time
translation between messaging systems.  It would certainly be possible
just to grab the RSS feed once an hour, convert it to a set of email
messages, and dump them into a mailbox. (That's how Hep 0.3 worked,
actually.)  But that architecture would be a lot less powerful, and it
wouldn't be nearly as easy to add support for new non-email protocols.

Having said all that, I totally understand where you're coming from on
not wanting to lose old messages.  It should be possible to have an
option where you say "Keep a copy of messages even when they're deleted
from the source".  If I do this, though, I'm going to do it right:
implemented at the Yarn level, and working for all protocols and
formats, not just RSS.  It should be possible just to set the
..archiveDeletedMessages bit on a ResourceView, and then work with it
from there without caring which messages really exist on the resource
and which have been archived locally.

It'll probably be a few weeks before I'm able to look into writing that
code.  In the meantime, if anybody is interested in starting to hack on
Yarn, you're welcome to grab the latest code from
http://fettig.net/svn/yarn.

That's probably more information than you were looking for, Matt - I
hope I answered your question!

Abe

[1] http://fettig.net/projects/yarn

Abe-

Thank you for the excellent and detailed explanation.  Although I am
playing the role of the whiny and demanding user here, I do a
significant amount of development at my day job in Perl, SAS and other
environments.  I appreciate having these details in hand so I can
better understand the code when I can get around to learning it.
Although I have yet to "dive into Python," this project is giving me
another reason to do so.

Your explanation also helps me to understand better the issue with the
folder tree rebuilding and why hep's IMAP doesn't support flagging
messages.  Messages can't be modified if they aren't stored anywhere
on hep's server.  I think I was confused initially because, to me,
IMAP implies that I am managing resources dedicated to me on the
server.  I have now reoriented my view of hep toward a more "universal
translator" angle.  I'll have to look into yarn a bit to further
refine that understanding.

By all means, make sure you take the time to do things right.  It is
your duty to guard the integrity of hep's architecture, not to bow to
the whims of some crackpot user like myself.  If there is no place for
this sort of statefulness in your vision of hep, you should also make
that call, too.

In more practical terms, what this means for hep users in the near
future is that if they want to retain messages, they need to use the
POP interface or a modified IMAP reader that retains messages even
when they're "removed from the server."

Am I right in thinking that this also implies that if I'm not
"checking mail" via hep regularly and an RSS server such as Slashdot
"overwrites" the older messages in their feed I could be missing
messages?  I guess the way to get the behavior of other RSS readers,
in this case, would be to ensure my mail reader is set to retrieve via
POP every hour.

I'll have to experiment a bit to see if I can get the behavior I want
without hacking hep.  One thought is to configure an IMAP server of my
own (which I have been considering anyway) and use some scripts to
call up hep periodically to populate the server's folders.  One
advantage to this approach is that I could use an industrial-strength
IMAP implementation and my regular e-mail could sit in folders right
alongside RSS messages delivered by hep.

-Matt

Hi Matt,

I'm glad that explanation helped.  I haven't written very much about the
architecture of Hep, so I was glad you asked a question that gave me
reason to do so.

Actually, flagging messages can and will be supported.  For each
resource you're connected to, Yarn keeps a local file containing
metadata - which messages you've seen, the time you first recieved each
message, etc.  This is how Hep's IMAP server keeps track of whether
you've seen a message or not.  It should only take a couple lines of
code to support saving the 'flagged' attribute, but I just haven't had
the time to do it yet.

Wow, you really are an experienced developer! :-)  I appreciate that,
but in this case I think your instincts as a user are correct.  There
should be a way to keep messages around even when the disappear from the
source.  I've been thinking about the best way to make this happen, and
I'll let you know when I have a plan that I'm happy with.

That's right.

While I certainly don't want to _require_ you to do stuff like that,
_allowing_ you to is what Hep is all about.  The data isn't locked in to
any one protocol, so you're free to copy it into other systems, convert
it to different formats, etc.

Abe