All People Seem To Need Data Processing, and when it comes to networking, understanding the OSI model is imperative. However you remember this seven layer schema, it's vital when troubleshooting difficult issues to remember them. Use these handy dandy mnemonics to help:
Please Do Not Throw Sausage Pizza Away (layer 1-7)
All People Seem To Need Data Processing (backwards, from layer 7-1)
Please Do Not Trust Sales People Always (1-7 again)
More information can be found in A+ and Network+ cert books, and other study guides I'm sure. Wikipedia has more information here too.
It's important to understand that although rare, your server logs can lie. Your mail server runs on the Application layer, so it only "sees" what that layer sees. Although of course it relies on all layers on down to the Physical layer, one single server may have several daemons running on different layers intercepting the data at different points along the way.
For instance, on a *nix box, you may have Sendmail logging that it's connecting to mail.domain.com [1.2.3.4].
Sendmail made a call to the DNS resolver and pulled the MX records. Again it called the resolver to resolve the A records listed in the MX. Then it tries to connect, but IPtables has a static route for 1.2.3.4 to send the data to 5.6.7.8, which is now refusing the connection.
Since Sendmail is running on the application layer, it doesn't log (or even see) when IP it's ACTUALLY connecting to, only where it *should* be connecting to. In order to see the packets you're going to have to do some traffic dumps using tcpdump, tcpick or Wireshark.
Of course, this is a very unique situation, either intentionally done by a sophisticated admin as a workaround or as a temporary "fix" for DNS issues, etc. Either way it's a kludge, and this is why each server should keep a changelog for later reference including the exact lines entered for later grep-ability (yes, I just made that word up :-) )
Also, there obviously could be many other issues with routing or other types of packet interference from router/firewalls locally to ISP related routing issues, but this was one instance that boggled me for several hours, so it stands out as an excellent reference.
-TEA
Sunday, July 4, 2010
Monday, June 21, 2010
Testing SMTP
Sometimes it's important to be able to manually reproduce what your computer/server is doing manually so you can see exactly where it's failing. Troubleshooting "utilities" are great in that they can often diagnose hundreds of issues in less time than it would take you to troubleshoot one, but they lack the nuance sometimes necessary to accurately diagnose issues.
A simple pass/fail is nice, but if you don't know WHY it's failing, what good is it?
When troubleshooting email, there are websites, utilities, diagnostic messages and more designed to help figure out what's going wrong. I have never found a single one that has been more informative than a simple SMTP session.
If you have telnet on your computer, that'll work fine. Netcat or any other command line tool that you can specify the ports to connect on will work too.
On your command/shell prompt type: telnet <the server's IP> 25
Then, when prompted, type the following commands:
---------------------------------------------------------------------------
HELO your hostname
<wait for response>
MAIL FROM: <your email address>
RCPT TO:
<wait for response>
DATA
FROM:
TO:
SUBJECT: test
this is a test.
.
----------------------------------------------------------------
Note the additional CR/LF after the subject. That separates the headers from the body (visible part) of the message. It's very important.
Also important is the line with a "." by itself. This tells the recipient's server that the message is complete and to process it.
On to troubleshooting....
In each part above where it says "wait for a response" you're expecing a response from the server, either something like "ok, go ahead", an error, or sometimes nothing. Here are some examples:
HELO
If there's no response, this can indicate that the server is extremely busy or that sometimes there's packet filtration between you and the server that's intercepting and silently dropping the packets. Either way, if there's no response, there's no transmission, and therefore no email.
Sometimes you'll get an error here referencing that it does not like your IP (if you're blacklisted), or that it's imposing some type of lookup on your HELO argument. It's best to be sure thet you give your FQDN here to prevent issues.
MAIL FROM:
You might get SPF errors here if your SPF record is wrong, or if your email address is blacklisted you might get a response here.
RCPT TO:
"Relaying denied", "mailbox is full", or any number of other recipient level issues may result as a response to this line. Any server that has different rules per user or domain (like a shared spam filter with different settings per domain/user) might also process here.
DATA
You're now giving the actual data that will be shown to the user. Generally there won't be an issue here until you close the transmission channel (with the "." on a line by itself), but anywhere you wait for a response is a potential place for errors.
"." (or closing the transmission channel)
This is where the email in it's entirety can be scanned and processed. Sometimes nothing will be scanned until this point, so even though SPF *could* be looked up after the "mail from:" command, it *might* not be until the entire message has been transmitted. Any of the above errors can happen here as well as content based scanning of the body, etc.
Some mail servers will also accept the email here, and then scan it later before delivery. That's a horrible idea, but that doesn't stop anyone usually, so be sure to copy down any information the acceptance line gives you like the message ID, so you can reference it later if the test does not complete.
Generally, you'll only be able to run this test on a border MTA unless it's your own mail server or the recipient's firewall allows you through. This can definitely complicate troubleshooting, but if you can prove that the issue is with the border MTA, or that the border MTA accepts the email but it is not relayed it at least absolves you from any finger pointing.
Once you have these commands memorized (it doesn't take long), you'll often be able to troubleshoot and diagnose issues faster than submitting tickets to your vendors. There's nothing more frustrating than either waiting for a response only to find out you could have fixed it hours ago, or conversely that you know exactly where the error occurred, but the other admin requires "proof" that they should bother fixing the issue when the DSN or other logs obviously state the issue.
Send a couple SMTP sessions their way and copy the session to them. It's hard to argue with facts :-)
-TEA
A simple pass/fail is nice, but if you don't know WHY it's failing, what good is it?
When troubleshooting email, there are websites, utilities, diagnostic messages and more designed to help figure out what's going wrong. I have never found a single one that has been more informative than a simple SMTP session.
If you have telnet on your computer, that'll work fine. Netcat or any other command line tool that you can specify the ports to connect on will work too.
On your command/shell prompt type: telnet <the server's IP>
Then, when prompted, type the following commands:
---------------------------------------------------------------------------
HELO your hostname
<wait for response>
this is a test.
.
----------------------------------------------------------------
Note the additional CR/LF after the subject. That separates the headers from the body (visible part) of the message. It's very important.
Also important is the line with a "." by itself. This tells the recipient's server that the message is complete and to process it.
On to troubleshooting....
In each part above where it says "wait for a response" you're expecing a response from the server, either something like "ok, go ahead", an error, or sometimes nothing. Here are some examples:
HELO
If there's no response, this can indicate that the server is extremely busy or that sometimes there's packet filtration between you and the server that's intercepting and silently dropping the packets. Either way, if there's no response, there's no transmission, and therefore no email.
Sometimes you'll get an error here referencing that it does not like your IP (if you're blacklisted), or that it's imposing some type of lookup on your HELO argument. It's best to be sure thet you give your FQDN here to prevent issues.
MAIL FROM:
You might get SPF errors here if your SPF record is wrong, or if your email address is blacklisted you might get a response here.
RCPT TO:
"Relaying denied", "mailbox is full", or any number of other recipient level issues may result as a response to this line. Any server that has different rules per user or domain (like a shared spam filter with different settings per domain/user) might also process here.
DATA
You're now giving the actual data that will be shown to the user. Generally there won't be an issue here until you close the transmission channel (with the "." on a line by itself), but anywhere you wait for a response is a potential place for errors.
"." (or closing the transmission channel)
This is where the email in it's entirety can be scanned and processed. Sometimes nothing will be scanned until this point, so even though SPF *could* be looked up after the "mail from:" command, it *might* not be until the entire message has been transmitted. Any of the above errors can happen here as well as content based scanning of the body, etc.
Some mail servers will also accept the email here, and then scan it later before delivery. That's a horrible idea, but that doesn't stop anyone usually, so be sure to copy down any information the acceptance line gives you like the message ID, so you can reference it later if the test does not complete.
Generally, you'll only be able to run this test on a border MTA unless it's your own mail server or the recipient's firewall allows you through. This can definitely complicate troubleshooting, but if you can prove that the issue is with the border MTA, or that the border MTA accepts the email but it is not relayed it at least absolves you from any finger pointing.
Once you have these commands memorized (it doesn't take long), you'll often be able to troubleshoot and diagnose issues faster than submitting tickets to your vendors. There's nothing more frustrating than either waiting for a response only to find out you could have fixed it hours ago, or conversely that you know exactly where the error occurred, but the other admin requires "proof" that they should bother fixing the issue when the DSN or other logs obviously state the issue.
Send a couple SMTP sessions their way and copy the session to them. It's hard to argue with facts :-)
-TEA
Sunday, June 20, 2010
subdomain MX records
We've all used subdomains. The popular naming convention "www.domain.com" is in itself a subdomain (the subdomain "www" in the "domain.com" domain), but did you know each subdomain can also have it's own MX records?
The obvious questions are "why would anyone do that" and "how would anyone do that?". Here's some food for thought.
Way back in the day, things were simpler. Email specifically didn't require tons of MX and A record lookups, there weren't even DNS servers to query let alone DNS record types. Users had accounts on their local servers, and the servers had names that although understandable would be different from organization to organization. Email addresses weren't "bob@company.com", they were "bob@accounting.company.com" for Bob in accounting and "bob@engineering.company.com" for Bob the engineer. Each organization would have their own server names, and although often they'd have meaning within the organization they might be goofy to outsiders. Also, desktop PCs were uncommon then, so you would connect to the server for almost all of your daily tasks. Therefore, rocky@bullwinkle.company.com was literally addressing the email to the user account "rocky" on the server named bullwinkle.company.com. Some places still conform to these odd naming conventions. I knew someone who went to the University of Michigan's Engineering school who had an email address @engin.umich.edu. It took me months to figure out that "engin" was for " school of ENGINeering".
Nowadays it's slightly different. "company.com" generally will have all of it's users email addresses directly @company.com, but not always. For instance, let's say there's a domain BigCompany.com. BigCompany.com is all over the world, and they want their customers to know that they're dealing with their "local" office. So, they may have John@USA.BigCompany.com and John@Canada.BigCompany.com, but they're totally different Johns, in totally different regions.
Let's say they not only want to have email addresses at the country level, but also at the state level. They could also have John@California.USA.BigCompany.com and John@NewYork.USA.BigCompany.com.
Now, most companies wouldn't do this because they don't want to confuse their customers by having them talk to the wrong John all the time, but you commonly see this with schools for instance. They might be something like "Principal@SchoolName.NY.k12.us".
Ok, now we know why someone would have subdomain MX records. Either to conform to a legacy naming convention or to help routing between different regions/offices. Now the question is "how do we do that"? The answer shouldn't be too surprising, we make MX records for each subdomain.
Just like each domain name can have MX records (and should have them if email is addressed to it), subdomains can get in on the fun too with their own set of records. Let's say you have an email address webmaster@www.domain.com, but you don't want to run a mail server on the same box as your web server (which is a good idea btw!). You can simply have MX records for www.domain.com go to the same server as your regular domain like this:
Host Priority Record
domain.com 10 mail.domain.com
www.domain.com 10 mail.domain.com
Now, you may have to create a separate zone for this subdomain if you're running your own DNS server, or if you're using some provider's interface they may require additional steps to configure this but most of the time it's possible. Some services don't allow subdomain records to be made if you're using their DNS, but DNS hosting is cheap. If they don't allow this, and they have ridiculously bad support who won't discuss the finer side of RFCs with you (cough, yahoo.com, cough), just change your DNS host. Outside of this issue, you're likely to find other reasons to switch providers later if you're not already disgusted with their lack of support.
It's important to keep in mind, that this is not something you can do after the fact. Once your users have you in their address book as user@domain.com, they will ALWAYS query domain.com's MX records regardless of whether you add an alias for user@sub.domain.com unless of course you choose to change your email address and deal with all of the headaches associated with that.
It also should not be used after the border MTA using address rewriting without serious contemplation. Although there are some excellent reasons to have your border MTA rewrite the delivery addresses to "user@sub.domain.com" while the rest of the world still emails the user@domain.com" address, there are serious considerations to take into account at the very least the reply-to address, or sender rewriting back to user@domain.com on the way out.
This is not likely something that many admins have dealt with before, and could be conceptually confusing at first. Don't worry, it's nothing different than what you do on the main domain, it's just a second place to deal with. User@domain.com queries domain.com's MX and sends them there. User@Sub.Domain.com queries sub.domain.com's MX and sends them there. It's pretty simple actually once you get your head around it. :-)
-TEA
The obvious questions are "why would anyone do that" and "how would anyone do that?". Here's some food for thought.
Way back in the day, things were simpler. Email specifically didn't require tons of MX and A record lookups, there weren't even DNS servers to query let alone DNS record types. Users had accounts on their local servers, and the servers had names that although understandable would be different from organization to organization. Email addresses weren't "bob@company.com", they were "bob@accounting.company.com" for Bob in accounting and "bob@engineering.company.com" for Bob the engineer. Each organization would have their own server names, and although often they'd have meaning within the organization they might be goofy to outsiders. Also, desktop PCs were uncommon then, so you would connect to the server for almost all of your daily tasks. Therefore, rocky@bullwinkle.company.com was literally addressing the email to the user account "rocky" on the server named bullwinkle.company.com. Some places still conform to these odd naming conventions. I knew someone who went to the University of Michigan's Engineering school who had an email address @engin.umich.edu. It took me months to figure out that "engin" was for " school of ENGINeering".
Nowadays it's slightly different. "company.com" generally will have all of it's users email addresses directly @company.com, but not always. For instance, let's say there's a domain BigCompany.com. BigCompany.com is all over the world, and they want their customers to know that they're dealing with their "local" office. So, they may have John@USA.BigCompany.com and John@Canada.BigCompany.com, but they're totally different Johns, in totally different regions.
Let's say they not only want to have email addresses at the country level, but also at the state level. They could also have John@California.USA.BigCompany.com and John@NewYork.USA.BigCompany.com.
Now, most companies wouldn't do this because they don't want to confuse their customers by having them talk to the wrong John all the time, but you commonly see this with schools for instance. They might be something like "Principal@SchoolName.NY.k12.us".
Ok, now we know why someone would have subdomain MX records. Either to conform to a legacy naming convention or to help routing between different regions/offices. Now the question is "how do we do that"? The answer shouldn't be too surprising, we make MX records for each subdomain.
Just like each domain name can have MX records (and should have them if email is addressed to it), subdomains can get in on the fun too with their own set of records. Let's say you have an email address webmaster@www.domain.com, but you don't want to run a mail server on the same box as your web server (which is a good idea btw!). You can simply have MX records for www.domain.com go to the same server as your regular domain like this:
Host Priority Record
domain.com 10 mail.domain.com
www.domain.com 10 mail.domain.com
Now, you may have to create a separate zone for this subdomain if you're running your own DNS server, or if you're using some provider's interface they may require additional steps to configure this but most of the time it's possible. Some services don't allow subdomain records to be made if you're using their DNS, but DNS hosting is cheap. If they don't allow this, and they have ridiculously bad support who won't discuss the finer side of RFCs with you (cough, yahoo.com, cough), just change your DNS host. Outside of this issue, you're likely to find other reasons to switch providers later if you're not already disgusted with their lack of support.
It's important to keep in mind, that this is not something you can do after the fact. Once your users have you in their address book as user@domain.com, they will ALWAYS query domain.com's MX records regardless of whether you add an alias for user@sub.domain.com unless of course you choose to change your email address and deal with all of the headaches associated with that.
It also should not be used after the border MTA using address rewriting without serious contemplation. Although there are some excellent reasons to have your border MTA rewrite the delivery addresses to "user@sub.domain.com" while the rest of the world still emails the user@domain.com" address, there are serious considerations to take into account at the very least the reply-to address, or sender rewriting back to user@domain.com on the way out.
This is not likely something that many admins have dealt with before, and could be conceptually confusing at first. Don't worry, it's nothing different than what you do on the main domain, it's just a second place to deal with. User@domain.com queries domain.com's MX and sends them there. User@Sub.Domain.com queries sub.domain.com's MX and sends them there. It's pretty simple actually once you get your head around it. :-)
-TEA
DNS doesn't end at MX you know...
Ok, so you understand MX records, you're ready to take on the world, right?
Not quite yet. Patience young grasshopper.
So, you're sending email to user@domain.com, you queried dns1.domain.com for the MX records and now you've got these MX records:
Host Priority Record
domain.com 10 mail1.domain.com
domain.com 20 mail2.domain.com
domain.com 30 mail3.domain.com
Ummm, so, computers don't speak English, they don't know what a "mail1.domain.com" is. They want digits that can turn into nice 1s and 0s that they can compute . What's a poor mail server to do with this stuff?
While we're at it, how did we even know to talk to dns1.domain.com to get these records?
Also, once we knew to talk to dns1.domain.com, we're back to the first question of how does the mail server know WTH a "dns1.domain.com" is, and where are the 1s and 0s it needs?
I is so confus :-(
Don't worry, there's more to an email's life than just MX records. There's lots of types of records including type A, type SOA, and type CNAME.
I. Type A records
Type A records ("A" for "Address") turn "mail1.domain.com" into a nice pretty number (the "IP Address" or "Internet Protocol address") for our mail server (and all other computers) to compute. The A record might look like this:
Mail1.domain.com IN A 10.0.100.28
Now we can see that the first MX record for domain.com has a name of "mail1.domain.com", but that name actually resides at 10.0.100.28. This number means nothing to most of us silly humans, but to a computer it's a nice string of digits that it can compute and then connect to.
TCP/IP and all of it's routing mechanisms are beyond the scope of this article, so you'll just have to make do with "once you have the IP you can connect to the server". :-P pbbt!
II. SOA records
OK, so this is a little backwards since you need the SOA records before you can get the MX records or A records, but you should already understand MX records, and you need to understand A records before you can get SOA records.
SOA records are the "Start Of Authority" records. There are literally millions of DNS servers in the world. How do you know which ones are responsible for the records you're querying? Well, that's where SOA records come into play. SOA records are given by the "root" name servers to offload queries to other DNS servers, and to provide a distributed architecture that is easier to maintain and provides more fault tolerance.
Root name servers are the biggest and baddest name servers in the world. There are two sets of them for your .com, .net, and the like, one at root-servers.net and one at gtld-servers.net. If queried, they will either reference the other one (like a father who says "go ask your mother", or vice-versa) or they will give the DNS server registered with them as authoritative.
You can play with this yourself using nslookup on Windows or dig on *nix.
Once the root servers give the name of the DNS server, that server's A record must be resolved then it can be queried for the MX, A, CNAME, or any other type of records you wish.
III. CNAME records
CNAME records are basically aliases to A records. For instance, "www.domain.com" might be the same server as "ftp.domain.com", so you could make ftp.domain.com a CNAME for www.domain.com, then an additional query would happen for the A record "www.domain.com" which when resolved would then allow transmission.
Typically, CNAMEs should not be in MX records. There's a ton of debate over whether they're allowed in the MX at all according to RFCs. I'm not going to enter that debate, but I will say that there are enough mail servers on the planet that don't respect CNAMEs in the MX that you should do everything in your power not to put them there. At the very least it adds an additional query to your DNS server (to get the IP of the A record in the CNAME), so for simplicity's sake I'd recommend against it as well.
One time a CNAME might be used is if you have a server that accepts email that does not have MX records. Originally, user@domain.com was actually literally user@some.server.com. You could even address email to an IP address like "user@10.0.100.28" for example and it would work. Most mail servers will fall back to this convention if MX records cannot be resolved, so you could for instance have a CNAME errors.domain.com pointing to "mail1.domain.com", then you could email "www-server@errors.domain.com", "email-server@errors.domain.com", etc which would then actually send the email to mail1.domain.com.
NOTE: this does not alias other record types, only A records. For instance, setting domain2.com as a CNAME for domain1.com DOES NOT make domain2.com's MX records the same as domain1.com's. CNAME records are not copies of the zone, they are address aliases and will only route traffic to the A record they point to. Additionally, it's technically against RFCs to have both MX and CNAMEs for the same zone. I.e. if you have www.domain.com as a CNAME, you can't also have MX for www.domain.com.
So, there you have it. MX record resolution requires SOA lookups, which require their own A record resolution. MX records include A records typically, which require their own resolution by the SOA. CNAMEs shouldn't be used in the MX, and can't have their own MX, but they can still be used to route email in a pinch.
Try using nslookup or dig to do your own DNS recursion. Once you realize how many DNS queries each email requires, be sure to give your DNS server a big pat on the back for so many jobs well done.
-TEA
Not quite yet. Patience young grasshopper.
So, you're sending email to user@domain.com, you queried dns1.domain.com for the MX records and now you've got these MX records:
Host Priority Record
domain.com 10 mail1.domain.com
domain.com 20 mail2.domain.com
domain.com 30 mail3.domain.com
Ummm, so, computers don't speak English, they don't know what a "mail1.domain.com" is. They want digits that can turn into nice 1s and 0s that they can compute . What's a poor mail server to do with this stuff?
While we're at it, how did we even know to talk to dns1.domain.com to get these records?
Also, once we knew to talk to dns1.domain.com, we're back to the first question of how does the mail server know WTH a "dns1.domain.com" is, and where are the 1s and 0s it needs?
I is so confus :-(
Don't worry, there's more to an email's life than just MX records. There's lots of types of records including type A, type SOA, and type CNAME.
I. Type A records
Type A records ("A" for "Address") turn "mail1.domain.com" into a nice pretty number (the "IP Address" or "Internet Protocol address") for our mail server (and all other computers) to compute. The A record might look like this:
Mail1.domain.com IN A 10.0.100.28
Now we can see that the first MX record for domain.com has a name of "mail1.domain.com", but that name actually resides at 10.0.100.28. This number means nothing to most of us silly humans, but to a computer it's a nice string of digits that it can compute and then connect to.
TCP/IP and all of it's routing mechanisms are beyond the scope of this article, so you'll just have to make do with "once you have the IP you can connect to the server". :-P pbbt!
II. SOA records
OK, so this is a little backwards since you need the SOA records before you can get the MX records or A records, but you should already understand MX records, and you need to understand A records before you can get SOA records.
SOA records are the "Start Of Authority" records. There are literally millions of DNS servers in the world. How do you know which ones are responsible for the records you're querying? Well, that's where SOA records come into play. SOA records are given by the "root" name servers to offload queries to other DNS servers, and to provide a distributed architecture that is easier to maintain and provides more fault tolerance.
Root name servers are the biggest and baddest name servers in the world. There are two sets of them for your .com, .net, and the like, one at root-servers.net and one at gtld-servers.net. If queried, they will either reference the other one (like a father who says "go ask your mother", or vice-versa) or they will give the DNS server registered with them as authoritative.
You can play with this yourself using nslookup on Windows or dig on *nix.
Once the root servers give the name of the DNS server, that server's A record must be resolved then it can be queried for the MX, A, CNAME, or any other type of records you wish.
III. CNAME records
CNAME records are basically aliases to A records. For instance, "www.domain.com" might be the same server as "ftp.domain.com", so you could make ftp.domain.com a CNAME for www.domain.com, then an additional query would happen for the A record "www.domain.com" which when resolved would then allow transmission.
Typically, CNAMEs should not be in MX records. There's a ton of debate over whether they're allowed in the MX at all according to RFCs. I'm not going to enter that debate, but I will say that there are enough mail servers on the planet that don't respect CNAMEs in the MX that you should do everything in your power not to put them there. At the very least it adds an additional query to your DNS server (to get the IP of the A record in the CNAME), so for simplicity's sake I'd recommend against it as well.
One time a CNAME might be used is if you have a server that accepts email that does not have MX records. Originally, user@domain.com was actually literally user@some.server.com. You could even address email to an IP address like "user@10.0.100.28" for example and it would work. Most mail servers will fall back to this convention if MX records cannot be resolved, so you could for instance have a CNAME errors.domain.com pointing to "mail1.domain.com", then you could email "www-server@errors.domain.com", "email-server@errors.domain.com", etc which would then actually send the email to mail1.domain.com.
NOTE: this does not alias other record types, only A records. For instance, setting domain2.com as a CNAME for domain1.com DOES NOT make domain2.com's MX records the same as domain1.com's. CNAME records are not copies of the zone, they are address aliases and will only route traffic to the A record they point to. Additionally, it's technically against RFCs to have both MX and CNAMEs for the same zone. I.e. if you have www.domain.com as a CNAME, you can't also have MX for www.domain.com.
So, there you have it. MX record resolution requires SOA lookups, which require their own A record resolution. MX records include A records typically, which require their own resolution by the SOA. CNAMEs shouldn't be used in the MX, and can't have their own MX, but they can still be used to route email in a pinch.
Try using nslookup or dig to do your own DNS recursion. Once you realize how many DNS queries each email requires, be sure to give your DNS server a big pat on the back for so many jobs well done.
-TEA
Thursday, June 10, 2010
Email Feedback Report for IP
Really AOL, really?
Over the years I have seen remarkable feedback reports from AOL. Everything from obvious user error like newsletters, etc. to bills that the user did not want to receive (but were completely legit), to those "chain letters" telling you that you and your entire family will die painfully unless you forward the message to 200 of your closest friends.
Recently, I saw a feedback report for a read receipt. A READ RECEIPT. Someone sent an email, got the read receipt back and then reported it as spam. That part doesn't surprise me (nothing much does any more), but a provider that has been in business as long as AOL (remember when they "were" the WWW for many Americans?) not having processes in place to deal with a user error this remarkably dumb amazes me.
First, their entire concept of blacklisting IPs is so 1999. It's 2010, use a content scanner!
Second, relying on their users to create an automated IP blacklist for them misplaces the burden onto their customer base, rather than trained techs (who can tell the difference between spam and a read receipt) and simultaneously creates a worse experience for all of their other users when "spam button" happy users wreck the IPBL for everyone else.
Third, if you're going to have an IP blacklist, and it's going to automatically block based on user reports, AT LEAST try to filter out obvious user error. A READ RECEIPT! Their logs show the message leaving, and the message clearly states it's a read receipt, the headers even had the appropriate "notification" message type. AOL, do you really have to make it this easy to poke fun at you?
AOL, I would like to offer my services to you. I'll even do it for half my hourly rate. Just please, please, please revisit this horribly planned out mechanism and get the correct processes in place to handle your spam problems.
While you're at it, could you please fix your outbound spam problems too? How many of your customers are really in Nigeria that know dead rich people with my last name?
-TEA
Over the years I have seen remarkable feedback reports from AOL. Everything from obvious user error like newsletters, etc. to bills that the user did not want to receive (but were completely legit), to those "chain letters" telling you that you and your entire family will die painfully unless you forward the message to 200 of your closest friends.
Recently, I saw a feedback report for a read receipt. A READ RECEIPT. Someone sent an email, got the read receipt back and then reported it as spam. That part doesn't surprise me (nothing much does any more), but a provider that has been in business as long as AOL (remember when they "were" the WWW for many Americans?) not having processes in place to deal with a user error this remarkably dumb amazes me.
First, their entire concept of blacklisting IPs is so 1999. It's 2010, use a content scanner!
Second, relying on their users to create an automated IP blacklist for them misplaces the burden onto their customer base, rather than trained techs (who can tell the difference between spam and a read receipt) and simultaneously creates a worse experience for all of their other users when "spam button" happy users wreck the IPBL for everyone else.
Third, if you're going to have an IP blacklist, and it's going to automatically block based on user reports, AT LEAST try to filter out obvious user error. A READ RECEIPT! Their logs show the message leaving, and the message clearly states it's a read receipt, the headers even had the appropriate "notification" message type. AOL, do you really have to make it this easy to poke fun at you?
AOL, I would like to offer my services to you. I'll even do it for half my hourly rate. Just please, please, please revisit this horribly planned out mechanism and get the correct processes in place to handle your spam problems.
While you're at it, could you please fix your outbound spam problems too? How many of your customers are really in Nigeria that know dead rich people with my last name?
-TEA
Wednesday, June 9, 2010
What's an "MX Record"?
Unfortunately, MX records are one of the most misunderstood points of email transmission. Padawan mail admins have many misconceptions. Here are a few:
1. That the MX records have a specific naming convention like mail.domain.com or smtp.domain.com
2. That every server that handles email must be in the MX record
3. That the destination MTA must be in the MX record
4. That an "MX server" is another name for a mail server
5. That a "reverse DNS" lookup queries the sending domain's MX
Now that we know a few of the things an MX record is NOT, let's take a look at what it IS.
An MX record is simply an entry in a domains DNS zone. Typically they consist of one or more "A" type records (which then require resolution) with priorities assigned to them. Also, as with all DNS record a TTL value, or Time To Live, is assigned to them as a whole (and to each type A record as well).
They are simply references to the border MTA(s) responsible for relaying incoming email for a domain. When an email server is asked to email user@domain.com, it queries the DNS servers responsible for the domain (often by recursing to the root name servers looking for SOA records) and asks for the MX records. Typically, the MX records consist of type A records so then another query is given to find the IP addresses associated to them.
Let's take this story as an example. Let's say you're at a friends house discussing a long lost friend.
Friend: Hey I bumped into John last week at the mall
You: Really, I haven't seen him in years! Did you get his phone number?
Friend: Yes, but I don't remember it. You'll have to look in my phone.
You: Where's your phone? (recursive DNS query to the root NS)
Friend: Over on the table
You: What's his entry? (MX record query)
Friend: It's "John Doe"
You open the phone and look at "John Doe's" entry (type A record lookup) and call him on your phone. (transmission)
This simple story, that has probably never actually happened to any of us but should be able to be understood by most of us, is as simple as MX gets. Of course there's more to it (TTL values, "border MTAs", lots more DNS lookups, and any number of issues routing), but those are really more relegated to the realm of troubleshooting, and not something to worry our pretty little heads with right now :-)
Once it's been determined where to send the email, the transmission ensues and sooner or later after being relayed, archived, scanned for spam/viruses, and many more operations the email will eventually find it's home at the destination server and be picked up by the client to be read by the user.
Just think of all of the servers across the globe that played a part in getting your email to it's destination sometime when you're sending email. It really is amazing how it all comes together so seamlessly.
The next time you get the call from your user wondering why it took five seconds to get an email try explaining this to them. Boggle their mind a little bit ;-)
-TEA
1. That the MX records have a specific naming convention like mail.domain.com or smtp.domain.com
2. That every server that handles email must be in the MX record
3. That the destination MTA must be in the MX record
4. That an "MX server" is another name for a mail server
5. That a "reverse DNS" lookup queries the sending domain's MX
Now that we know a few of the things an MX record is NOT, let's take a look at what it IS.
An MX record is simply an entry in a domains DNS zone. Typically they consist of one or more "A" type records (which then require resolution) with priorities assigned to them. Also, as with all DNS record a TTL value, or Time To Live, is assigned to them as a whole (and to each type A record as well).
They are simply references to the border MTA(s) responsible for relaying incoming email for a domain. When an email server is asked to email user@domain.com, it queries the DNS servers responsible for the domain (often by recursing to the root name servers looking for SOA records) and asks for the MX records. Typically, the MX records consist of type A records so then another query is given to find the IP addresses associated to them.
Let's take this story as an example. Let's say you're at a friends house discussing a long lost friend.
Friend: Hey I bumped into John last week at the mall
You: Really, I haven't seen him in years! Did you get his phone number?
Friend: Yes, but I don't remember it. You'll have to look in my phone.
You: Where's your phone? (recursive DNS query to the root NS)
Friend: Over on the table
You: What's his entry? (MX record query)
Friend: It's "John Doe"
You open the phone and look at "John Doe's" entry (type A record lookup) and call him on your phone. (transmission)
This simple story, that has probably never actually happened to any of us but should be able to be understood by most of us, is as simple as MX gets. Of course there's more to it (TTL values, "border MTAs", lots more DNS lookups, and any number of issues routing), but those are really more relegated to the realm of troubleshooting, and not something to worry our pretty little heads with right now :-)
Once it's been determined where to send the email, the transmission ensues and sooner or later after being relayed, archived, scanned for spam/viruses, and many more operations the email will eventually find it's home at the destination server and be picked up by the client to be read by the user.
Just think of all of the servers across the globe that played a part in getting your email to it's destination sometime when you're sending email. It really is amazing how it all comes together so seamlessly.
The next time you get the call from your user wondering why it took five seconds to get an email try explaining this to them. Boggle their mind a little bit ;-)
-TEA
Thursday, June 3, 2010
Temporary and permanent errors
Typically, you'll only see two types of errors "Temporary" and "Permanent" errors. Standard codes can be found in RFC 2821 section 4.2.3, but there are many different servers that reply with alternate codes as well.
Temporary errors are expressed through 4.x.x error codes. These might be due to the recipient server being busy, the recipient server greylisting the sender, or closing the connection prematurely. The sending server can also generate 4.x.x errors if they can't resolve the recipient's DNS, the recipient's server is offline, the recipient server is connected to and responds but does not fully accept the email or any other issue that does not result in the recipient server accepting the email (with a 2.x.x status) or outright rejecting the message (with a 5.x.x message). Sometimes firewalls intercept the packets and mangle them or drop them, this can also cause a 4.x.x error since either the recipient's server does not receive the correct packets to accept the message and times out the connection eventually, or the sending server does not know that the recipient's server accepted the message and also times out the connection eventually.
While temporary errors occur due to transmission issues or issues that are expected to resolve themselves typically, Permanent errors mean that the server was connected to, delivered a response, and that that response was that it refused to accept the message... EVER.
Typical reasons for this are that the mailbox is full or nonexistent, the message was blocked as spam/virus, the mail server has vital errors or a policy violation such as non-local delivery for the domain, too many hops per message, or others. This error means that the server is online, listening for connections, accepted your connection but then actively refused to accept the message. It's the difference between sending someone you don't want to talk to to voicemail versus picking up the phone and telling them to stop calling. There is no confusion.
A sending server can also generate a permanent error as a result of timing out the message after repeated temporary errors, but this will typically be hours or days later, whereas a recipient server's rejection is almost instant usually. Obviously there will also be repeated 4.x.x errors in the logs to reference.
Understanding the difference between these errors is also important to determine where to troubleshoot first. A permanent error is almost always the recipient server, whereas a temporary error could be one of many things.
Senders side:
DNS issues (queries)
ISP/routing issues
Firewall issues (packet filtering)
Addressing issues
Recipient's side:
DNS issues (serving)
ISP/routing issues
Firewall issues
Greylisting
Rate Limiting
Hardware issues
NAT issues
and many more....
As always, the bounce itself will have a lot of good information, the "How to read a bounceback" post should be the first place to check if a NDR/DSN has been generated, but otherwise use of your troubleshooting tools in conjunction with the log data should provide plenty of information to determine the source of the issue.
TEA
Temporary errors are expressed through 4.x.x error codes. These might be due to the recipient server being busy, the recipient server greylisting the sender, or closing the connection prematurely. The sending server can also generate 4.x.x errors if they can't resolve the recipient's DNS, the recipient's server is offline, the recipient server is connected to and responds but does not fully accept the email or any other issue that does not result in the recipient server accepting the email (with a 2.x.x status) or outright rejecting the message (with a 5.x.x message). Sometimes firewalls intercept the packets and mangle them or drop them, this can also cause a 4.x.x error since either the recipient's server does not receive the correct packets to accept the message and times out the connection eventually, or the sending server does not know that the recipient's server accepted the message and also times out the connection eventually.
While temporary errors occur due to transmission issues or issues that are expected to resolve themselves typically, Permanent errors mean that the server was connected to, delivered a response, and that that response was that it refused to accept the message... EVER.
Typical reasons for this are that the mailbox is full or nonexistent, the message was blocked as spam/virus, the mail server has vital errors or a policy violation such as non-local delivery for the domain, too many hops per message, or others. This error means that the server is online, listening for connections, accepted your connection but then actively refused to accept the message. It's the difference between sending someone you don't want to talk to to voicemail versus picking up the phone and telling them to stop calling. There is no confusion.
A sending server can also generate a permanent error as a result of timing out the message after repeated temporary errors, but this will typically be hours or days later, whereas a recipient server's rejection is almost instant usually. Obviously there will also be repeated 4.x.x errors in the logs to reference.
Understanding the difference between these errors is also important to determine where to troubleshoot first. A permanent error is almost always the recipient server, whereas a temporary error could be one of many things.
Senders side:
DNS issues (queries)
ISP/routing issues
Firewall issues (packet filtering)
Addressing issues
Recipient's side:
DNS issues (serving)
ISP/routing issues
Firewall issues
Greylisting
Rate Limiting
Hardware issues
NAT issues
and many more....
As always, the bounce itself will have a lot of good information, the "How to read a bounceback" post should be the first place to check if a NDR/DSN has been generated, but otherwise use of your troubleshooting tools in conjunction with the log data should provide plenty of information to determine the source of the issue.
TEA
Subscribe to:
Posts
(
Atom
)