NL1040630C2 - Method and system for email spam elimination and classification, using recipient defined codes and sender response. - Google Patents

Description

Method and system for email spam elimination and classification, using recipient defined codes and sender response

Field of the Invention [0001] The invention relates to the field of email spam detection and elimination. More specifically the invention relates to a method and system for detecting legitimate emails based on address books of the recipient (contacts, whitelists, blacklists) and codes specifically defined for each recipient.

Background of the Invention [0001] The following discussion builds on the discussion of SPAM Mitigation Systems (SMS) in EP2665230A1.

[0002] In the recent years a vast majority of the email traffic could be considered spam. Spam emails are mostly sent from the so-called automatic botnets that tend to use dynamic IPs, so that the traditional black listing updating methods could not keep up. By using botnets, spammers create and exploit free webmail accounts or deliver Spam emails directly to victim mailboxes by exploiting computational power and network bandwidth of their hosts and sometimes even user credentials. Many spam mitigation methods are used by email service providers and organizations to protect mail boxes of their customers and employees respectively. There are three main approaches for Spam mitigation: content-based filtering, real-time blacklisting (RBL), and sender reputation mechanisms (SRM).

[0003] Content-based filtering (CBF) refers to techniques in which emails body, attached executables, pictures or other files are analyzed and processed for producing some features upon which email classification is made. The emails content is related to the application-level, which is the highest level of the Open Systems Interconnection (OSI) model. Content-based features have a lot of useful information for classification, however, in the perspective of Internet Service Providers (ISP), there are some disadvantages; first, in order to classify incoming emails, each email must pass through a relatively heavy-weighted content-based filter. This means that significant computational resources are wasted on the filtering process, thus making it fairly costly, compared to other approaches, such as real-time blacklisting or sender reputation systems, which will be discussed later. A second disadvantage of CBF evolves from the fact that spammers continuously improve their CBF evading techniques. For example, instead of sending plain textual Spam emails, spammers typically send Spam-images or smarter textual content that obfuscate the unwanted content in normal textual content.

[0004] Real-time blacklisting (RBL), is another technique for mitigating the spamming problem. The RBL are IP-based lists that contain IP prefixes, of spamming Mail Transfer Agents (MTA) and are regarded as network-level-based filters. Using the RBL, large firms such as Internet Service Providers (ISP) can filter out the emails that are detected as originated from spamming IPs. The filtering is very fast, since the decision to accept or reject the email neither requires receiving the full email (therefore saving network resources) nor requires processing its content (therefore saving computational resources). In order to avoid misclassification, RBLs must be updated systematically. For example, Spamhaus, Spam-Cop, and SORBS are some initiatives that keep RBL systems updated by tracking and reporting spammers' IPs. RBL methods, however, cannot entirely solve the Spam email problem, as spammers can escape them, for example, by repeatedly changing their IPs by stealing local network IPs, or by temporarily stealing IPs using BGP hijacking attacks [ A. Ramachandran and N. Feamster. Understanding the network-level behavior of spammers. In ACM SIGCOMM, Pisa, Italy, 2006 ]. Another shortcoming of RBL is that whenever an IP prefix is blacklisted, both spammers and benign senders who share the same prefix might be rejected. Benign senders can also be blocked because of inaccurate blacklisting heuristics. In order to lower the false-positive rates blacklisting, heuristics limit their true positive rates by allowing many spam-mails to pass the filter, while blocking mainly repeated spammers. RBL usually has lower accuracy than CBF; this is an acceptable tradeoff given the real-time nature and the low utilization of computational resources of the RBL.

[0005] Sender reputation mechanisms (SRM) for Spam mitigation are a collection of methods for computing liability scores of email senders. The computation is usually based on information extracted from the network or from the transport level, from social network information or from other useful identifiers. Sender reputation systems should react quickly to changes in senders' behavioral patterns. More specifically, when a sending pattern of a sender takes a shape of Spammer, his reputation typically decreases. If the reputation of a sender goes below a predefined threshold, the system typically rejects the sender’s mails, at least until he gains up some reputation, by changing its sending properties. One of the advantages of sender reputation systems is that they complement and improve RBLs both in terms of detection accuracy and faster response to changes in the sending behavior of potential spammers.

[0006] Machine learning algorithms for computing sender reputation from data sets, that are not based on email content analysis, were used in several works in the past. For example, SNARE by Hao et al. [ S. Hao, N. A. Syed, N. Feamster, A. G. Gray, and S. Krasser. Detecting spammers with snare: Spatiotemporal network-level automated reputation engine. In 18th USENIX Security Symposium, 2009 ] is solely based on the network-level and geodesic features, such as distance in IP space to other email senders or the geographic distance between the sender and receiver. Ramachandran et al. [ A. Ramachandran, N. Feamster, and S. Vempala. Filtering spam with behavioral blacklisting. In ACM CCS, pages 342-351, 2007 ], research a different sender reputation and blacklisting approach. They present a new blacklisting system, SpamTracker, to classify email senders based on their sending behavior rather than the MTA's IP addresses. The authors assume that spammers abuse multiple benign MTAs at different times, but their sending patterns tend to remain mostly similar even when shifting from one abused IP to another. SpamTracker extracts network-level features, such as received time, remote IP, targeted domain, whether rejected and uses spectrum clustering algorithms to cluster email servers. The clustering is made on the email destination domains. The authors reported 10.4% True Positive Rate when using SpamTracker on a data set of 620 Spam mails that were missed by the organization filter and eventually where blacklisted in the following weeks.

[0007] Sender reputation mechanisms are not limited only to network-level features; reputation can also be learned from a much higher communication level, such as the social network level. The social-network-based filtering approach takes advantage on the natural trust system of social networks. For example if the sender and the receiver belong to the same social group, the communication is assumed to be legitimate. On the other hand, if the sender does not belong to any trustworthy social group, it is more likely to be blacklisted. There are many methods that make a good use of social networks to create both black and white lists. For example J. Balthrop et al. [ J. Balthrop, S. Forrest, Μ. E. J. Newman, and Μ. M. Williamson. Technological networks and the spread of computer viruses. Science, 304(5670):527-529,2004 ], have used email address books to compute a sender trust-degree.

[0008] Boykin et al. [ P. Boykin and V. Roychowdhury. Leveraging social networks to fight spam. IEEE Computer, 38(4):61-68,2005 1 uses email-network-based spam-filtering algorithm to automatically create both white and black lists. The authors present an automatic tool for extraction of social networks' features from the email header fields such as From, To, and CC. Then, they constructed a social graph, as can be observed from a single user's perspective. Later, they found a cluster of users which can be regarded as trusted. Finally, they trained a Bayesian classifier on the email addresses in the white and black lists, labeled as normal and Spam-senders respectively. The authors showed that their algorithm has 56% True Positive Rate with the black list and 34% True Positive Rate with the white list. Their method, empirically tested on three data sets of several thousands of emails, did not have any false positives. The downside of the proposed algorithm is that both the black and white lists can be made erroneous. For example, the black list can be fooled by attackers that use spyware to learn the user's frequently used address list, and have one or more of them added to the Spam mail, so that the co-recipients (the spam victims) will look like they belong to the user's social network. The white list can also be fooled by sending spam mail using one or more of the user's friends' accounts. This can be done, for example, if the friend's computer had been compromised by a bot which selectively send spam mails. Golbeck and Hendler [ J. Golbeck and J. Hendler. Reputation network analysis for email filtering. In First Conference on Email and Anti-Spam, Mountain View, California, USA, 2004 ] present a supervised collaborative mechanism, TrustMail, for learning reputation networks. The proposed mechanism is aided by users’ own scores for email senders. For example, a user can assign a high reputation score to his closest friends, and they in their turn may assign a high reputation rank to their friends. In this way, a reputation network is created. The reputation network may be used as a white-list, namely as a recommendation system with very low false positive rate, that may allow users to sort their emails by a reputation score.

[0009] The Above spam mitigation solutions are mainly focused on the application-layer and the network layer (i.e. content based, sender reputation mechanism, social-network-based, and realtime black-listing (RBL)). Beverly and Sollins [ R. Beverly and K. Sollins. Exploiting the transport-level characteristics of am. In 5th Conference on Email and Anti-Spam (CEAS), 2008 ] investigated the effectiveness of using transport-level features, i.e. round trip time, FIN count, Jitter and several more. The best features were selected using forward fitting to train SVM-based classifier. They reported 90% accuracy on 60 training examples. However, one of the weaknesses of their method compared to RBL and other reputation mechanisms, is that the emails (both Spam and Ham) must be fully received in order to extract their features, thus making spam mitigation process less effective.

[0010] In EP2665230A1 prior art methods are enhanced with an Historical Data Set (HDS) system. EP2665230A1 claims that detection of SPAM is increased to 98% using this approach. Given the enormous amount of SPAM, the remaining 2% still poses a problem. As the discussion shows State of the Art SPAM detection mechanisms are not perfect, which results in SPAM still being delivered into an email address and benign emails still being filtered out, sometimes with very negative effects on business or personal relationships [0010a] In US2008270540 priori art is described that determines the legitimacy of an incoming email based on the senders' email address being in an approved list and otherwise sending a reply mail to the sender to which the sender must respond in order for senders' emails to be accepted. The disadvantage of such a method is that SPAMmers can fulfil such a requirement in an automated fashion and without prohibitive cost.

[0010b] In KR20030074122 a prior art method is described that determines the legitimacy of an incoming email based on the senders' email address being in an approved list and otherwise sending a manual response quiz to the sender. The disadvantage of such a method is that SPAMmers can fulfil such a requirement in an automated fashion and without prohibitive cost while bona fide senders who have no automated response implemented are too much burdened by the requirements.

[0011] It is an object of the present invention to provide a new and innovative method and software for eliminating SPAM out of the users main email inbox, while at the same time informing benign Senders to this email address that their email has not passed SPAM filtering and providing them with an option to have their email delivered with the least possible effort and time delay, an option which is unattractive to SPAMmers. The proposed method is best used in addition to prior art methods.

[0012] It is another object of the present invention to allow prior art pre-processing SPAM filtering methods to relax their parameter setting in such a way that the erroneous classification of incoming emails as illegitimate is reduced, thus reducing the erroneous rejection of emails. The present method allows this because the related increase in SPAM passed through the preprocessing prior art filters is dealt with through the method and system proposed in the present invention.

[0013] It is another object of the present invention to provide options for email receivers to have senders pre-classify their email addressed to the receiver, according to the receiver's settings.

[0014] Other objects and advantages of the invention will become apparent as the description proceeds.

Summary of the Invention [0015] The invention relates to an email categorizing and spam elimination system, which comprises: a mail server or client which comprises: (a.l) one or more "legitimate or malicious" address or address domain lists, each address list categorizes originating addresses of emails as either legitimate or malicious; (a.2) one or more code lists, each code characterizing an incoming email that contains such code as benign and/or as belonging to a receiver defined category; and (a.3) a Content Based Filter and analyzer for content analyzing each incoming email based on its content, for subdividing into benign and malicious and into different subcategories emails to respective addressees and blocking and/or placing into dedicated folders at the addressees email receiving system incoming emails based on the codes they contain; and (b) a website service where actors wishing to send a user of this system an email can obtain codes specific for this user for either characterizing the email(s) to be sent as benign and/or characterizing the email as belonging to a subcategory defined and/or recognized by the receiver. Such codes can be derived from the email address they belong to by an algorithm not known to SPAMmers, or the user can be given free choice of his codes.

[0016] Preferably, said website service is unattractive for use by SPAMmers.

[0017] Preferably, said website service requires a fee to be paid per code obtained. Such a fee would be small for occasional use, such as the equivalent cost of a postmark, but high for SPAMmers who send large numbers of emails to accounts with whom they have no relationship.

[0018] Preferably, said legitimate address lists include one or more of the receivers usual address lists or parts thereof.

[0019] Preferably, said legitimate address lists can be updated with email addresses present in receivers sent emails and genuine received emails by an automated process provided in the system.

[0020] The invention also relates to a spam detection method, which comprises the steps of: (a) at a mail server or client: (a.l) establishing one or more "legitimate or malicious" address or address domain lists, each address list categorizes originating addresses of emails as either legitimate or malicious; (a.2) establishing one or more code lists, each code characterizing an incoming email that contains such code as benign and/or as belonging to a receiver defined category; and (a.3) performing a Content Based Filter and analyzer for content analyzing each incoming email based on its content, for subdividing into benign and malicious and into different subcategories emails to respective addressees and blocking and/or placing into dedicated folders at the addressee's email receiving system incoming emails based on the codes they contain; and (b) providing a website service where actors wishing to send a user of this system an email can obtain codes specific for this user for either characterizing the email(s) to be sent as benign and/or characterizing the email as belonging to a subcategory defined and/or recognized by the receiver.

[0021] Preferably, said website service is unattractive for use by SPAMmers.

[0022] Preferably, said website service requires a fee to be paid per code obtained. Such a fee would be small for occasional use, such as the equivalent cost of a postmark, but high for SPAMmers who send large numbers of emails to accounts with whom they have no relationship.

[0023] Preferably, said legitimate address lists include one or more of the receivers usual address lists or parts thereof.

[0024] Preferably, said legitimate address lists can be updated with receivers sent emails and genuine received emails by an automated process provided in the system.

Brief Description of the Drawings [0025] In the drawings:

Fig. 1 schematically shows the present invention's SES;

Fig. 2 schematically shows a general system of the prior art;

Fig. 3 schematically shows one embodiment of the present invention;

Detailed Description of Embodiments of the invention [0026] The present invention provides means for extending the spamming elimination capabilities of a typical email provider or email user mail mitigation system. Mail mitigation systems are well known in the art, and are used at mail providers and users for detecting specific emails that are suspected to come from spammers, and prevent them from arriving their destination, as defined by the sender. As noted above, such systems of the prior art are not capable of filtering all the spam emails, and their successful rate is usually about 90%, with advanced methods such as HDS up to 98%. However, in view of the vast amount of spam emails, even the remaining 2% is very upsetting. Moreover, systems of the prior art also block a small percentage of legitimate emails, which can be even more upsetting and damaging to relationships with customers or friends. As will be demonstrated, the method and system of the present invention very significantly increases the rate of spam elimination, and it performs this task in efficient manner. Moreover it alerts senders to emails blocked, which will eliminate the problems associated with lost legitimate emails. Moreover using the invention, the parameters of a prior art SMS can be relaxed, thus further reducing the occurrence of lost legitimate emails. More specifically, the system of the invention fulfils two tasks, as follows: (a) Determining whether an incoming email is legitimate according to the user settings; and (b) Providing an alert to legitimate senders, whose email is nevertheless not classified as legitimate by the system, and providing such senders with a means to obtain a code for subsequently reaching the user unhindered.

[0027] In an embodiment of the invention, the system of the invention is provided as an extension to existing typical spam mitigation system of a mail provider.

[0028] Fig. 1 is a general block diagram which illustrates a typical spam mitigation system 100 of the prior art, as used by mail providers. A stream of a vast number of emails 124 coming from various IP senders is received at mail server 181. The incoming emails are subjected to two types of filters: an RBL filter 111 which comprises a black list of senders IPs, and a content based filter (CBF). The RBL filter 111 (hereinafter RBLF) is prepared in various manners well known in the art, and is updated from time to time. Those emails that pass said RBL filter 111 are subjected to CBF 121. The output emails 139 that pass both the RBLF and the CBF proceed to the respective addressees. Mail server 181 further comprises a control unit 165 that coordinates the various tasks of mail server 181, and it further creates an email log that summarizes selected details that relate to the email traffic passing through the server. Typically, prior art spam mitigation systems such as of Fig. 1 are capable of filtering about 90% of the spam emails. Yet, since a very significant amount of the email traffic is spam, the remaining 10% is still very disturbing.

[0029] The prior art HDS presented in EP2665230A1 increases the rate of spam filtering up to approximately 98%.

[0030] Fig 2. Illustrates in block diagram form the structure of the TSS spam elimination system 203, according to an embodiment of the present invention. A stream of emails 210 addressed to the receiver / user under consideration is received through internet at email system 201. This system utilizes prior art methods to reduce the number of SPAM emails in the stream, however with settings that emphasize not to erroneously reject any legitimate emails so that in the outgoing stream of emails 211 as much as possible all legitimate emails addressed to the user are contained, and a certain amount of malicious emails. The TSS SPAM elimination system 203 utilises the Code repository 202 comprising the particular user's TSS codes, the user's and system's address books, black lists and whitelists 205 described below to determine through the Content Based Filter and

Analyser 211 whether an email is legitimate, possibly malicious, or to be rejected based on black list considerations. If the email is classified legitimate, it is or remains placed in the user's inbox 212. If the code is specific for a specific inbox, the email is placed in that particular inbox. If the email is not classified as legitimate it is placed in a reject folder 213. If the email's sender's address or domain is recognised as on the black list a black list reply 204 or no reply is sent to the sender, as determined by the user on beforehand,. If the email is classified as possibly malicious, a bounce notification reply 206, determined by the user on beforehand, is sent to the sender.

[0031] Fig. 3. Illustrates in block diagram form the decision tree of the TSS spam elimination system, according to an embodiment of the present invention. For each email 301 that enters the system all predetermined user codes are searched for in the email body and header. If one of the codes 302 is present in the email (decision 303) the email is classified as legitimate and then placed in the inbox 304 indicated by the particular code.

If none of the codes 302 is present one or more of the user's address books 305 are searched for the email address of the sender of the email. If (decision 306) the sender's email address is present the email is placed in an inbox 307, usually the main inbox of the user.

If decision 306 is NO then one or more of the user's white list address books 8 is searched for the presence of the sender's email address. If (decision 309) the sender's email address is present the email is placed in an inbox 10, usually the main inbox of the user.

If decision 309 is NO then one or more of the user's white lists 311, containing email domains and possibly valid IP addresses for such domains, is searched for the presence of the sender's email domain or email domain/IP address combination. If (decision 312) the sender's email address domain or domain and IP adress are present the email is placed in an inbox 13, usually the main inbox of the user.

If decision 312 is NO then one or more of the user's black lists 314, containing email addresses and/or domains, is searched for the presence of the sender's email address or email domain. If (decision 315) the sender's email address or domain are present the email is discarded placed in a reject email box and optionally a reply 316 is automatically sent to the sender, such as for example: "No emails can be sent to this address from your email address or address-domain".

If decision 315 is NO the email is classified as possibly malicious and the email is optionally discarded or placed in a special email box for less often inspection and a notification message 317 is sent to the sender, such as for example: "Unfortunately your email has been classified as possible SPAM. You can obtain an access code for sending emails to this address at the website www.TotalSDamStoD.com or from our secretary. Your email will most likely not be read by the addressee.".

[0032] In an alternative embodiment of the invention the Users'White and Black Lists are combined with Lists made available from a General Repository made available through the internet.

[0033] ln an alternative embodiment of the invention the Spam Elimination System is an additional function in an Email Client Program running on the Recipients' computer (for example an Add-in in Outlook).

[0034] In an alternative embodiment of the invention the Spam Elimination System is executed on the Email Server of an ICT provider, which has access to the users' codes and address lists.

[0035] Although embodiments of the invention have been described by way of illustration, it will be understood that the invention may be carried out with many variations, modifications, and adaptations, without exceeding the scope of the claims.

Claims (10)

A SPAM elimination system, comprising: an email server and / or email client comprising: - A Code Repository comprising at least one code linked to the receiver, whereby an incoming email is classified as legitimate for this recipient and passed through in one of the recipients' email in folders if this email contains the code; - A Content Based Filter and Analyzer that is used to analyze the content (including header) of each incoming email based on the information contained therein relative to the Repository, and to only those emails in the recipient's main inbox places that are classified as legitimate by the analysis; and to send appropriate notification emails to senders of other emails. The system of claim 1, wherein the system also comprises a Repository containing email addresses of known contacts of the recipient that can be used in the analysis of incoming emails to determine the legitimacy of these emails; A system according to claim 1 or 2, wherein the system also comprises a Repository containing Blacklists and Whitelists with email addresses and email address domains used in the analysis of incoming emails to determine the legitimacy and illegitimacy of emails with a sender address or sender address domain in the Repository. The system of claim 3, wherein the Repository for Whitelist email address domains also contains a list of IP addresses of legitimate servers for those email address domains used in the analysis of incoming emails to classify an email with an email as legitimate Limit address domain on the Whitelist to those emails that come from a domain / IP address combination. A system according to claim 1, 2, 3 or 4, wherein the Code Repository also contains codes for classifying the email into categories that are standardized or determined by the recipient, and with which the email can be placed in a separate email inbox , depending on the associated code found. 6. Method for SPAM detection and handling of emails which comprises the steps of: in a mail client or server: - Creating a Code Repository comprising at least one code linked to the recipient - Performing an analysis on the content of incoming emails to determine if they contain the code in subject or body - Classify as an legitimate e-mail for this recipient if this e-mail contains the code; - If and insofar as desired by the user, send appropriate notification replies to senders whose e-mail is classified as illegal. A method according to claim 6, further comprising the steps of: - Creating a Repository comprising email addresses of known contacts of the recipient - classifying incoming emails as legitimate from a sender whose email address occurs in the Repository; A method according to claim 6 or 7, further comprising the steps of: - Creating a Repository comprising Blacklists and Whitelists with email addresses and email address domains - analyzing incoming emails to determine the legitimacy and illegitimacy of emails with a sender address or sender address domain in the Repository. The method of claim 8, further comprising the steps of: - Creating a Repository for Whitelist email address domains that also contains a list of IP addresses of legitimate servers for that email address domains - Analysis of incoming emails that classify it as legitimate an email with an email address domain on the Whitelist limited to those emails that come from a domain / IP address combination in the Repository. Method according to claim 6, 7, 8 or 9, further comprising the steps of: - Creating a Code Repository that also contains codes for classifying the email into categories that are standardized or determined by the recipient - Placing the email in a separate email inbox, depending on the associated code found.