Great Circle Associates List-Managers
(February 2001)

Indexed By Date: [Previous] [Next] Indexed By Thread: [Previous] [Next]

Subject: Re: how to make mail lists scale well
From: Chuq Von Rospach <chuqui @ plaidworks . com>
Date: Fri, 09 Feb 2001 23:31:55 -0800
To: James M Galvin <galvin @ acm . org>
Cc: List-Managers <list-managers @ greatcircle . com>
In-reply-to: <Pine . BSF . 4 . 21 . 0102091633360 . 5603-100000 @ two . elistx . com>
User-agent: Microsoft-Entourage/9.0.2509

On 2/9/01 1:50 PM, "James M Galvin" <galvin @
 acm .
 org> wrote:

>   I've more or less come to the conclusion that the discussion list
>   via email is a bad technology. It's just until recently, there
>   really weren't better technologies -- and if all you have are
>   hammers, everything looks like a nail.
> I hope you don't really mean this quite as literally as it is stated.  I
> think it's short-sighted to suggest that mailing lists are universally
> bad technology for discussion lists.

I wouldn't use the word universally. Small lists, tightly focussed lists,
low volume lists -- they all work fine in e-mail. The word I'm probably
searching for is predictable. The more predictable list content is (within
reason), the more people will get used to the list in their box, although
the busier a list is, the more people will opt out and do without rather
than put up with it. Digests or no, I simply can't stay on the MySQL or PHP
lists, for instance, because they are too busy for me. They're things I
really wish were in a different form where I could either adjust my
granularity or adjust how pushy the content delivery was (or both).

> I prefer mailing lists over all other related technologies.  The
> principal reason for this is that it is push technology and I really
> like having everything in my local environment.

And there's nothing wrong with that. However, there's a huge variability
from person to person just how much they're willing to have pushed before it
starts bothering them, and that's a big problem with mail lists. Those of us
with high tolerances and/or smart filtering systems in the client have
trouble understanding those that don't -- but it's a valid issue that has to
be kept in consideration. Simply saying "learn to filter your mail" isn't
the answer. 

> In my opinion no technology deals exceptionally well with the problem of
> a large group of people contributing to a discussion.

Basically, I agree. That's what I'm trying to work towards, though.
Optimally, something that works well for small groups but scales as needed
without radical shifts in technology or operation (administrative or user).
And then, on Tuesday...

My comment that "mail lists suck" is both true and not-true. There are many
cases where it's clearly not-true, but I think for the general case it IS
true. But it's a tough argument to make, because we all are used to mail
lists. It's the old "everything is a nail" thing again -- we're so used to
nailing things that we can't conceive of what it would be to own a
screwdriver. It's not a GREAT technology for this, but it's a technology
we're COMFORTABLE with. And the changes I'm suggesting are fairly radical
shifts in mindthink, and inertia likes to win.

> Developers of Internet technologies know very well that the only real
> problem to solve is "scalability".

Scalability is to some degree what I really do for a living these days. The
base technologies are known -- how to subscribe, how to unsubscribe, how to
deliver a piece of email. But -- how to handle 30,000 subscribes a day, 7
days a week, how to deliver 4 million pieces of email, all fully customized,
in 12 languages based on user preference, in 8 hours -- that's
scalability... (grin).

How to allow 50 people to post email to each other is easy. How to allow
5000 to do it is a challenge. How to handle 50,000? You won't do it with a
mail list....

> but how about we reach to a finer level of granularity.

That's a huge area of exploration for me today.

> Another solution I have seen work is to throttle or otherwise limit the
> messages that can be submitted to the list.

A useful thing if you have one guy posting 50 messages a day (like, um,
me!). But what if you have a discussion list that's grown from 5,000 to
12,000 users, and you've gone from 200 users posting 2.5 messages a week to
600 users posting 4 messages a week on average? Nless you want sitewide
throttles that dribble the messages out, what do you do? And when it grows
to 18,000 users? 

Actually, now that I think about it, one other option would be to give a
user the ability to throttle themselves -- a limit where they can say "only
send me N messages a day". Whether you hold or throw out, I dunno. Maybe
that's configurable.

Actually, in a project I'm currently trying to design, we're adding a
feature to guarantee people don't get more than N emails over a period of
time, to avoid overloading folks with stuff. It's an interesting issue --
you want to communicate, they want you to communicate, but you don't want to
get them feeling like you've moved in and put your feet on the coffee

But if lists are subject to topical bursts, I'm not sure any of these really
solve the problem. Slowing message delivery defeats topicalism, and that
defeats the underlying purpose of the list, no?

One option I've thought about is the "cel" concept. Blame John Le Carre, but
you build a network of N users (big number). Rather than every message going
to every person, split them into cels, where only stuff sent by someone in
that cel gets sent to members of that cel. Virtually -- we're talking about
that really big sports bar with lots of different rooms. You don't expect to
hear every conversation in every room in real life, no? But virtually --
people have problems with that. But for a big, busy list? Or is this really
a bunch of smaller lists umbrellaed under a meta list?

See, here we go mutating the system to attempt to handle the content again.
At what point does it stop being a mail list and start being something else?

Chuq Von Rospach, Internet Gnome <>
[<chuqui @
 plaidworks .
 com> = <me @
 chuqui .
 com> = <chuq @
 apple .
Yes, yes, I've finally finished my home page. Lucky you.

I tried to get a life once, but they were out of stock.

Indexed By Date Previous: Re: Lists and society
From: SRE <eckert @ climber . org>
Next: Re: Quasi-meta-discussion
From: Chuq Von Rospach <chuqui @ plaidworks . com>
Indexed By Thread Previous: Re: how to make mail lists scale well
From: James M Galvin <galvin @ acm . org>
Next: Re: how to make mail lists scale well
From: Mikael Hansen <meh @ dnai . com>

Search Internet Search