TTL in DNS part 3

When we last talked in part 2, we were discussing what TTL is not (it is not how long the name server takes to update) and what it is (it is how long a computer takes to update it's own, local, cache).

We ended with an object lesson where www.TheEmailAuthority.com has a 1 hour TTL on it's A record, and I change the A record and shut down the server at my old IP promptly at 3 PM.

User1 was just reading my site at 2:30, so when they go to load another page anytime after 3 and before their own, local, cached record expires at 3:30 they won't be able to load the page.

However, User2 was lucky enough to request my A record at 3:10, so they've got twenty full minutes of Email Authority goodness while User1 is still just reading an error screen.

We've now seen how two users each with their own cache and their own TTL respond to DNS changes.  This simple example only used one record type, and only one "hop" between the client and the server.  Let's take this to the next level and try a more complex, email related, change.

Domain1.com is changing their email server, and their online ordering system sends them emails every time someone places an order.  Because of this, they cannot afford to have any down time.  We'll presume that they're open 24/7/365 so late-night and weekend migrations are out of the question.  Take these three examples into consideration, presuming the following zone file:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com. 
domain1.com. 86400 IN NS ns2.domain1s-ISP.com. 

domain1.com. 14400 IN MX mail.domain1.com. 

mail.domain1.com. 3600  IN A  10.11.12.13.

Example 1: Change the MX record

Domain1.com's email administrator sets up mail2.domain1.com and places it into the MX records.  He sends a test email from his gmail account, and it routes correctly.  However, the online ordering emails are not coming through to mail2.domain1.com.

This is because the online ordering server still has his old MX records cached, and is sending the order emails to the old server until it updates it's cache sometime in the next 4 hours.  It could be less than that depending on when the online ordering server last cached the records, but there is still an outage and the admin loses his job.

Moral of this story: MX record changes aren't good enough.

Example 2: Set the TTL to a low value and change the IP of the A record

The next admin understands TTL, and logs in to his DNS interface to change the TTL in the morning before the migration that evening.  He sets the A record's TTL to 1 minute in the morning, and then changes the A record's IP address to point to the new server

This way the cached MX records include the same A record, and the short TTL means that the connecting mail servers *should* update the IP every minute.

Unfortunately, the owner's wife uses an AOL address, and they cache records for their own pre-set value (24+ hours), so since the owner's wife's email wasn't coming through the new admin got fired too.  He at least got a severance package...


Example 3: Set the original server to relay mail to the new server

The final admin understands that things don't always go according to plan, and leaves the old server online for several days after the migration.  He sets the old server to act as a mail relay though, so it's not delivering locally to any of the users (and thereby interrupting mail flow).  He waits until no legitimate email is seen in his logs for a day or so and then takes the server offline.

This admin also could have used a mail relay service such as a hosted anti-spam service to act as the relay for extra points :-)


Well, that was a much longer look into TTL than anyone would ever want to know.  Hopefully you enjoyed it, and can now impress your friends (or put them to sleep) with your new-found TTL skills.



-TEA

TTL in DNS part 2

In part 1 of this series, we discussed what TTL is, and provided a few exercises to help strengthen your understanding of TTL.  We'll pick up where we left off with our same DNS zone for domain1.com:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com. 
domain1.com. 86400 IN NS ns2.domain1s-ISP.com. 

domain1.com. 14400 IN MX mail.domain1.com. 

mail.domain1.com. 3600  IN A  10.11.12.13.

As we discussed, the NS records will expire 1 day after they were last cached, the MX records will expire 4 hours after they were last cached, and the A record will expire 60 minutes after it was last cached.  What does this mean for you and your planned migration?

A common misconception about TTL is that it's the amount of time the Name Server requires to update the record.  This is absolutely not the case.  Do you remember what RFC 1034 said a TTL is?  A TTL is:

...primarily used by resolvers when they cache RRs.  The TTL describes how long a RR can be cached before it should be* discarded.

Where in there does it say that a TTL is how long a name server takes to update?  What does it say?  It says that a DNS resolver (which is anything that resolves DNS, your computer has a DNS resolver in it) caches the records, and the TTL describes how long the DNS resolver should cache the record before discarding it (and then re-requesting it).

What this means is that (for the sake of simplicity) every computer connected to the internet has it's own local cache of websites it's visited, mail servers it's talked to, etc. and will not re-request those records until it's own, local, cache expires.  TTL is how long it should* keep the records until it discards them and re-requests them.

Object lesson time...

UserPC1.ValuedReader.com visits www.TheEmailAuthority.com at 2:30 PM.  www.TheEmailAuthority.com has a 1 hour TTL for it's A record.  If I move the website at 3 PM, and the old IP stops serving the site, how long would it be until UserPC1.ValuedReader.com can read my site again?

UserPC-2.ValuedReader.com visits www.TheEmailAuthority.com at 3:10 PM How long will it be until they can read the site?

The answers and explanations will be in part three.

TTL in DNS part 1

TTL, or Time To Live, is a confusing subject for many people. Due to DNS's distributed nature, there are many different scenarios to consider. In this post we will explain what TTL is, and what it is not, using a real world example.

First, I suppose we'll answer why TTL is important. TTL can impact your email infrastructure any time you make changes to a Name Server that hosts your DNS, the A record that your mail server resolves to (for instance, when you change to a new hosting provider), or when you change your MX records (for a new hosting provider, or a new mail relay). By no means is this a comprehensive list, but it will suffice for this example.  Since DNS records can be cached for days sometimes, it's important to understand how they work when planning a migration so you can account for several day's worth of traffic.

How it works:
According to RFC 1034, "DOMAIN NAMES - CONCEPTS AND FACILITIES", each DNS record (or "Resource Record", "RR" for short) has a TTL or Time To Live.   It describes TTL as:

TTL which is the time to live of the RR. This field is a 32 bit integer in units of seconds, an is primarily used by resolvers when they cache RRs. The TTL describes how long a RR can be cached before it should be* discarded. 

A "Resolver" is anything that resolves a DNS record. This could be your ISP's name servers, or your local computer. Anything that queries and resolves DNS records is a "DNS Resolver".

Let's take the below zone into consideration:

domain1.com. 86400 IN NS ns1.domain1s-ISP.com. 
domain1.com. 86400 IN NS ns2.domain1s-ISP.com. 

domain1.com. 14400 IN MX mail.domain1.com. 

mail.domain1.com. 3600  IN A  10.11.12.13. 

So, let's take our three examples above and do some "real world" exercises.  Remember, TTL is expressed in seconds.  You might want to open your calculator app now ;-)

If domain1.com wanted to switch their Name Server records to a different ISP, how many days would it be until the NS TTL expired?

If domain1.com wanted to add a hosted anti-spam service (which would require them to change the MX records), how many hours would it take for the MX TTL to expire?

If domain1.com wanted to upgrade their mail server and point the A record to a new IP, how many minutes would it take for the A record's TTL to expire?

Ok, so we've gotten through the math portion of this post.  Pencils on the table please. Now we know that records can take DAYS to update depending on their TTL.  How does this impact our migration?  I'll answer in part two...

Stopping spam

Albert Einstein said “We can't solve problems by using the same kind of thinking we used when we created them". This is certainly true of spam.

In the early days of email being widely adopted people were excited to use this new tool - and unaware of the threats that this new tool came with - so they handed out their email address willy-nilly. Every site that asked them for it, they gave it. If they wanted a personal response on a forum, they posted their email address for someone to respond to. In order to communicate with customers, they'd put their email address proudly on their "Contact us" page to show how "tech savvy" they were.

Then the unthinkable happened. People started emailing them.

Not about the thing they downloaded, not in response to their forum post, or even any customers from their "contact us" page. These emails were offers to buy prescription drugs or pornography. The senders had the unfortunate duty of informing them of the death of a rich loved one in Africa somewhere, or had good news of their winning a lottery Microsoft was running right now.

By the time they realized what was happening, their email address had already been abused, traded to other spammers, or sold in batches of "opt in" email addresses to the highest bidder. Their new email "toy" rapidly became much less fun to play with.

This is the "kind of thinking" that "we used when we created (the problem)". Email is a phenomenal tool, but it is a tool not a toy. You wouldn't let someone who had never used a chainsaw play with yours, and you wouldn't use a bandsaw to hammer a nail. Tools have purposes, and misuse can be dangerous.

So, what do we do? What's the "right" way of thinking we should start using?

1. Protect the email addresses you have.

Don't give it out to just anybody, and block unauthorized access to it. Use a spam blocker with not just a high block rate, but that also won't block your real email. This will save untold amounts of time (and therefore money) troubleshooting issues or even just searching through your junk looking for legit messages.

2. Use a unique email address.

This doesn't always mean creating new addresses every time you want to sign up for something. If your email address is YourUsername@YourDomain.com, try sending email to YourUsername+Whatever@YourDomain.com. If you receive it, you can do this when you sign up for things online and use something unique for each.

For instance, YourUsername+TheEmailAuthority@YourDomain.com if you sign up for a newsletter from me or YourUsername+TigerDirect@YourDomain.com if you sign up with Tiger Direct.

This will allow you to see WHO has compromised your email address, and also set up filtering rules if one of your "new addresses" gets compromised.

3. Use a "disposable" email address.

Ugh. We live in such a disposable culture. Disposable razors, disposable diapers, disposable this, that and something else. You mean to tell me there are disposable email addresses now too?!?! Yes. And they have a GREAT purpose.

When you sign up for a service, or a newsletter, coupons, etc you WANT to continue to receive emails from them. If you didn't, you wouldn't have signed up, right? Well, what about all of the myriad of websites that you need to give them your email address just to sign up and then you never want to receive another email from them, EVER? Use a disposable address, that's what.

Keep in mind, password resets, changes in TOSes and other vital email will go to this address as well, so don't use it for everything, but it's a great addition to your privacy protecting arsenal. Someone could guess to strip off everything after the "+" sign if you use the method above, but they'll never guess the real address associated to a completely fake address.

There are several services out there. I've used several, and I like Mailinator, but you can find a ton with this Google search.

So, here's the breakdown:

Give your real email address to friends, family and business partners.
Give your "+" address to sites you need to have ongoing communication with, but aren't in your immediate "circle of friends". Examples are stores, newsletters, blogs/websites, etc.
Use a disposable address for anyone you only need to communicate with once.

Using these simple steps will help keep your inbox clean and your privacy protected. To help protect against everything else, you need a good antispam service.


-TEA

Bounce message spam

There's a new spam in the wild claiming to be bounce messages to messages that you've sent, but that actually send you to a meds spammer's site. They've been mutating for several days, so the body and attachment change fairly rapidly, but here's what one looks like as of the date of this post:


--- begin spam ---

Delivery Status Notification (Failure) Note: Forwarded message is attached.

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

Delivery to the following recipient has been delayed:

user@domain.com

Message will be retried for 2 more day(s)

---end spam---

Then there is an HTML attachment which redirects to the spammer's web site.

It's really quite ingenious really. Who wouldn't open a bounce, even if they didn't remember sending the original email, or maybe especially if they didn't remember sending it?

Another one, this time stating that it's from "mailfilter@yourdomain.com":

---another spam---

Note: Forwarded message is attached.

An email you sent could not be delivered.
Subject: Hello

Delivery to the following address has failed...
user@RecipientDomain.com


Technical information about this permanent failure:


-- Transcript of session follows --

While talking to [some.server.com] :

>>> rcpt to: user@RecipientDomain.com
<<<>: Recipient address rejected: User unknown

---end spam---

Users are so accustomed to worrying thet they're "infected" every time the get a bounce (when in fact, often it's just "spoofing") that these fake spam bounces must have a huge click through rate.

Although it's difficult to block "backscatter", I have to imagine that these are relatively easy to block. All the HTML attachment includes is a refresh to their site, any content based scanner should pick them up pretty easily.

As for the user's perspective, the old rules (slightly modified) still pertain here.

If you do not know the sender, do not open the attachment.

With these new "bounce message" spams, you can modify the above rule to:

If you did not send the email, do not open any attachments.

If you DO know the sender, and it's not a file you requested, call them to see if they actually sent it. Especially if it's in the following formats: .exe, .pif, .zip, .rar, .bat, .pdf, .htm, .html.

Of course, these spam messages have .htm file attachments, so the above pertains to them as well. Although these are a simple spam, websites can be infected and spread malware through your browser. Always exercise extreme caution when opening .htm files.



-TEA

Spoofing

"Spoofing" someone's email address is really hard, right? Only the most elite hackers can do it, right? It's not something that I can try at home, right?

no, no and no.

In the post "Testing SMTP" we discussed all of the commands necessary to fully deliver an email. Nothing special, no encryption, just a plain text email (without MIME-types, etc), but an email nonetheless. Below is the list of commands again in case you missed that post:

---------------------------------------------------------------------------
HELO your hostname

MAIL FROM:

RCPT TO:

DATA

FROM:
TO:
SUBJECT: test

this is a test.
.

----------------------------------------------------------------

You'll notice something is missing in there. Something important for security. Something you do to many websites, to view your email, and often the first thing when you turn on your computer.

Don't scroll down, take a guess :-)






There's no login/password!

You could send yourself an email from GeorgeWashington@whitehouse.gov, or Bill@Microsoft.com saying whatever you like. Of course, you're still incredibly traceable, so don't get any bright ideas about sending an email "from" your boss giving yourself a raise or anything.

This is one way though that spammers and viruses hide their tracks or create a false sense of trust from their victim. SPF was implemented as a direct result of this flaw in SMTP's security.

Many times, spammers will "spoof" other users in their database of email addresses. Spam/virus filtering after the "Border-MTA" results then in backscatter, or bounces from email that was not sent by you. This is of course different than the article earlier about fake bounce message spam. Backscatter is legitimate bounces, but sent in response to an illegitimate use of the victim's email address.

Sometimes backscatter can inundate an inbox so thoroughly that it's hard to get any work done. In cases like this, ALL bounce messages should be blocked or moved to a folder keeping them out of the inbox. A common way to do this is to make a rule for these senders:

<>
postmaster@
mailer-daemon@

And that includes the following phrases:

"hi, this is the qmail send program"
"returned mail: see transcript for details"
"undeliverable mail returned to sender"
"delivery status notification (failure)"
"delivery status notification (delay)"
"out of office autoreply"

and any others that come in.

One of the reasons to not use a challenge-response anti-spam service is that they accept all email for your domain, then deliver "challenges" to which the sender must "respond". Since they accept the email, many deliver challenges to innocent users who are simply being abused by a spammer. Therefore you'll also want to include challenge response vendors in your filter.

Unfortunately, the lack of security in SMTP has been abused to no end. That's why it's imperative for your reputation, and a good "good neighbor" policy to implement SPF records. Although they're a little hard to grasp at first, They're the best way to be sure that nobody is using your email address without your permission. Unless you've been hacked, but that's a whole other post....


-TEA

Email and the OSI model.

All People Seem To Need Data Processing, and when it comes to networking, understanding the OSI model is imperative. However you remember this seven layer schema, it's vital when troubleshooting difficult issues to remember them. Use these handy dandy mnemonics to help:

Please Do Not Throw Sausage Pizza Away (layer 1-7)
All People Seem To Need Data Processing (backwards, from layer 7-1)
Please Do Not Trust Sales People Always (1-7 again)

More information can be found in A+ and Network+ cert books, and other study guides I'm sure. Wikipedia has more information here too.

It's important to understand that although rare, your server logs can lie. Your mail server runs on the Application layer, so it only "sees" what that layer sees. Although of course it relies on all layers on down to the Physical layer, one single server may have several daemons running on different layers intercepting the data at different points along the way.

For instance, on a *nix box, you may have Sendmail logging that it's connecting to mail.domain.com [1.2.3.4].

Sendmail made a call to the DNS resolver and pulled the MX records. Again it called the resolver to resolve the A records listed in the MX. Then it tries to connect, but IPtables has a static route for 1.2.3.4 to send the data to 5.6.7.8, which is now refusing the connection.

Since Sendmail is running on the application layer, it doesn't log (or even see) when IP it's ACTUALLY connecting to, only where it *should* be connecting to. In order to see the packets you're going to have to do some traffic dumps using tcpdump, tcpick or Wireshark.

Of course, this is a very unique situation, either intentionally done by a sophisticated admin as a workaround or as a temporary "fix" for DNS issues, etc. Either way it's a kludge, and this is why each server should keep a changelog for later reference including the exact lines entered for later grep-ability (yes, I just made that word up :-) )

Also, there obviously could be many other issues with routing or other types of packet interference from router/firewalls locally to ISP related routing issues, but this was one instance that boggled me for several hours, so it stands out as an excellent reference.

-TEA