BYR Achieve · 镜像论坛

Linux netfilter Hacking HOWTO Rusty Russell and Harald Welte, mailing list netfilter@lists.samba.org $Revision: 1.14 $ $Date: 2002/07/02 04:07:19 $ This document describes the netfilter architecture for Linux, how to hack it, and some of the major systems which sit on top of it, such as packet filtering, connection tracking and Network Address Translation. ______________________________________________________________________ Table of Contents 1. Introduction 1.1 What is netfilter? 1.2 What's wrong with what we had in 2.0 and 2.2? 1.3 Who are you? 1.4 Why does it crash? 2. Where Can I Get The Latest? 3. Netfilter Architecture 3.1 Netfilter Base 3.2 Packet Selection: IP Tables 3.2.1 Packet Filtering 3.2.2 NAT 3.2.2.1 Masquerading, Port Forwarding, Transparent Proxying 3.2.3 Packet Mangling 3.3 Connection Tracking 3.4 Other Additions 4. Information for Programmers 4.1 Understanding ip_tables 4.1.1 ip_tables Data Structures 4.1.2 ip_tables From Userspace 4.1.3 ip_tables Use And Traversal 4.2 Extending iptables 4.2.1 The Kernel 4.2.1.1 New Match Functions 4.2.1.2 New Targets 4.2.1.3 New Tables 4.2.2 Userspace Tool 4.2.2.1 New Match Functions 4.2.2.2 New Targets 4.2.3 Using `libiptc' 4.3 Understanding NAT 4.3.1 Connection Tracking 4.4 Extending Connection Tracking/NAT 4.4.1 Standard NAT Targets 4.4.2 New Protocols 4.4.2.1 Inside The Kernel 4.4.3 New NAT Targets 4.4.4 Protocol Helpers 4.4.5 Connection Tracking Helper Modules 4.4.5.1 Description 4.4.5.2 Structures and Functions Available 4.4.5.3 Example skeleton of a conntrack helper module 4.4.6 NAT helper modules 4.4.6.1 Description 4.4.6.2 Structures and Functions Available 4.4.6.3 Example NAT helper module 4.5 Understanding Netfilter 4.6 Writing New Netfilter Modules 4.6.1 Plugging Into Netfilter Hooks 4.6.2 Processing Queued Packets 4.6.3 Receiving Commands From Userspace 4.7 Packet Handling in Userspace 5. Translating 2.0 and 2.2 Packet Filter Modules 6. Netfilter Hooks for Tunnel Writers 7. The Test Suite 7.1 Writing a Test 7.2 Variables And Environment 7.3 Useful Tools 7.3.1 gen_ip 7.3.2 rcv_ip 7.3.3 gen_err 7.3.4 local_ip 7.4 Random Advice 8. Motivation 9. Thanks ______________________________________________________________________ 1. Introduction Hi guys. This document is a journey; some parts are well-traveled, and in other areas you will find yourself almost alone. The best advice I can give you is to grab a large, cozy mug of coffee or hot chocolate, get into a comfortable chair, and absorb the contents before venturing out into the sometimes dangerous world of network hacking. For more understanding of the use of the infrastructure on top of the netfilter framework, I recommend reading the Packet Filtering HOWTO and the NAT HOWTO. For information on kernel programming I suggest Rusty's Unreliable Guide to Kernel Hacking and Rusty's Unreliable Guide to Kernel Locking. (C) 2000 Paul `Rusty' Russell. Licenced under the GNU GPL. 1.1. What is netfilter? netfilter is a framework for packet mangling, outside the normal Berkeley socket interface. It has four parts. Firstly, each protocol defines "hooks" (IPv4 defines 5) which are well-defined points in a packet's traversal of that protocol stack. At each of these points, the protocol will call the netfilter framework with the packet and the hook number. Secondly, parts of the kernel can register to listen to the different hooks for each protocol. So when a packet is passed to the netfilter framework, it checks to see if anyone has registered for that protocol and hook; if so, they each get a chance to examine (and possibly alter) the packet in order, then discard the packet (NF_DROP), allow it to pass (NF_ACCEPT), tell netfilter to forget about the packet (NF_STOLEN), or ask netfilter to queue the packet for userspace (NF_QUEUE). The third part is that packets that have been queued are collected (by the ip_queue driver) for sending to userspace; these packets are handled asynchronously. The final part consists of cool comments in the code and documentation. This is instrumental for any experimental project. The netfilter motto is (stolen shamelessly from Cort Dougan): ``So... how is this better than KDE?'' (This motto narrowly edged out `Whip me, beat me, make me use ipchains'). In addition to this raw framework, various modules have been written which provide functionality similar to previous (pre-netfilter) kernels, in particular, an extensible NAT system, and an extensible packet filtering system (iptables). 1.2. What's wrong with what we had in 2.0 and 2.2? 1. No infrastructure established for passing packet to userspace: o Kernel coding is hard o Kernel coding must be done in C/C++ o Dynamic filtering policies do not belong in kernel o 2.2 introduced copying packets to userspace via netlink, but reinjecting packets is slow, and subject to `sanity' checks. For example, reinjecting packet claiming to come from an existing interface is not possible. 2. Transparent proxying is a crock: o We look up every packet to see if there is a socket bound to that address o Root is allowed to bind to foreign addresses o Can't redirect locally-generated packets o REDIRECT doesn't handle UDP replies: redirecting UDP named packets to 1153 doesn't work because some clients don't like replies coming from anything other than port 53. o REDIRECT doesn't coordinate with tcp/udp port allocation: a user may get a port shadowed by a REDIRECT rule. o Has been broken at least twice during 2.1 series. o Code is extremely intrusive. Consider the stats on the number of #ifdef CONFIG_IP_TRANSPARENT_PROXY in 2.2.1: 34 occurrences in 11 files. Compare this with CONFIG_IP_FIREWALL, which has 10 occurrences in 5 files. 3. Creating packet filter rules independent of interface addresses is not possible: o Must know local interface addresses to distinguish locally- generated or locally-terminating packets from through packets. o Even that is not enough in cases of redirection or masquerading. o Forward chain only has information on outgoing interface, meaning you have to figure where a packet came from using knowledge of the network topography. 4. Masquerading is tacked onto packet filtering: Interactions between packet filtering and masquerading make firewalling complex: o At input filtering, reply packets appear to be destined for box itself o At forward filtering, demasqueraded packets are not seen at all o At output filtering, packets appear to come from local box 5. TOS manipulation, redirect, ICMP unreachable and mark (which can effect port forwarding, routing, and QoS) are tacked onto packet filter code as well. 6. ipchains code is neither modular, nor extensible (eg. MAC address filtering, options filtering, etc). 7. Lack of sufficient infrastructure has led to a profusion of different techniques: o Masquerading, plus per-protocol modules o Fast static NAT by routing code (doesn't have per-protocol handling) o Port forwarding, redirect, auto forwarding o The Linux NAT and Virtual Server Projects. 8. Incompatibility between CONFIG_NET_FASTROUTE and packet filtering: o Forwarded packets traverse three chains anyway o No way to tell if these chains can be bypassed 9. Inspection of packets dropped due to routing protection (eg. Source Address Verification) not possible. 10. No way of atomically reading counters on packet filter rules. 11. CONFIG_IP_ALWAYS_DEFRAG is a compile-time option, making life difficult for distributions who want one general-purpose kernel. 1.3. Who are you? I'm the only one foolish enough to do this. As ipchains co-author and current Linux Kernel IP Firewall maintainer, I see many of the problems that people have with the current system, as well as getting exposure to what they are trying to do. 1.4. Why does it crash? Woah! You should have seen it last week! Because I'm not as great a programmer as we might all wish, and I certainly haven't tested all scenarios, because of lack of time, equipment and/or inspiration. I do have a testsuite, which I encourage you to contribute to. 2. Where Can I Get The Latest? There is a CVS server on netfilter.org which contains the latest HOWTOs, userspace tools and testsuite. For casual browsing, you can use the Web Interface <http://cvs.netfilter.org/>. To grab the latest sources, you can do the following: 1. Log in to the netfilter CVS server anonymously: cvs -d :pserver:cvs@pserver.netfilter.org:/cvspublic login 2. When it asks you for a password type `cvs'. 3. Check out the code using: # cvs -d :pserver:cvs@pserver.netfilter.org:/cvspublic co netfilter/userspace 4. To update to the latest version, use cvs update -d -P 3. Netfilter Architecture Netfilter is merely a series of hooks in various points in a protocol stack (at this stage, IPv4, IPv6 and DECnet). The (idealized) IPv4 traversal diagram looks like the following: A Packet Traversing the Netfilter System: --->[1]--->[ROUTE]--->[3]--->[4]---> | ^ | | | [ROUTE] v | [2] [5] | ^ | | v | On the left is where packets come in: having passed the simple sanity checks (i.e., not truncated, IP checksum OK, not a promiscuous receive), they are passed to the netfilter framework's NF_IP_PRE_ROUTING [1] hook. Next they enter the routing code, which decides whether the packet is destined for another interface, or a local process. The routing code may drop packets that are unroutable. If it's destined for the box itself, the netfilter framework is called again for the NF_IP_LOCAL_IN [2] hook, before being passed to the process (if any). If it's destined to pass to another interface instead, the netfilter framework is called for the NF_IP_FORWARD [3] hook. The packet then passes a final netfilter hook, the NF_IP_POST_ROUTING [4] hook, before being put on the wire again. The NF_IP_LOCAL_OUT [5] hook is called for packets that are created locally. Here you can see that routing occurs after this hook is called: in fact, the routing code is called first (to figure out the source IP address and some IP options): if you want to alter the routing, you must alter the `skb->dst' field yourself, as is done in the NAT code. 3.1. Netfilter Base Now we have an example of netfilter for IPv4, you can see when each hook is activated. This is the essence of netfilter. Kernel modules can register to listen at any of these hooks. A module that registers a function must specify the priority of the function within the hook; then when that netfilter hook is called from the core networking code, each module registered at that point is called in the order of priorites, and is free to manipulate the packet. The module can then tell netfilter to do one of five things: 1. NF_ACCEPT: continue traversal as normal. 2. NF_DROP: drop the packet; don't continue traversal. 3. NF_STOLEN: I've taken over the packet; don't continue traversal. 4. NF_QUEUE: queue the packet (usually for userspace handling). 5. NF_REPEAT: call this hook again. The other parts of netfilter (handling queued packets, cool comments) will be covered in the kernel section later. Upon this foundation, we can build fairly complex packet manipulations, as shown in the next two sections. 3.2. Packet Selection: IP Tables A packet selection system called IP Tables has been built over the netfilter framework. It is a direct descendent of ipchains (that came from ipfwadm, that came from BSD's ipfw IIRC), with extensibility. Kernel modules can register a new table, and ask for a packet to traverse a given table. This packet selection method is used for packet filtering (the `filter' table), Network Address Translation (the `nat' table) and general pre-route packet mangling (the `mangle' table). The hooks that are registered with netfilter are as follows (with the functions in each hook in the order that they are actually called): --->PRE------>[ROUTE]--->FWD---------->POST------> Conntrack | Mangle ^ Mangle Mangle | Filter | NAT (Src) NAT (Dst) | | Conntrack (QDisc) | [ROUTE] v | IN Filter OUT Conntrack | Conntrack ^ Mangle | Mangle | NAT (Dst) v | Filter 3.2.1. Packet Filtering This table, `filter', should never alter packets: only filter them. One of the advantages of iptables filter over ipchains is that it is small and fast, and it hooks into netfilter at the NF_IP_LOCAL_IN, NF_IP_FORWARD and NF_IP_LOCAL_OUT points. This means that for any given packet, there is one (and only one) possible place to filter it. This makes things much simpler for users than ipchains was. Also, the fact that the netfilter framework provides both the input and output interfaces for the NF_IP_FORWARD hook means that many kinds of filtering are far simpler. Note: I have ported the kernel portions of both ipchains and ipfwadm as modules on top of netfilter, enabling the use of the old ipfwadm and ipchains userspace tools without requiring an upgrade. 3.2.2. NAT This is the realm of the `nat' table, which is fed packets from two netfilter hooks: for non-local packets, the NF_IP_PRE_ROUTING and NF_IP_POST_ROUTING hooks are perfect for destination and source alterations respectively. If CONFIG_IP_NF_NAT_LOCAL is defined, the hooks NF_IP_LOCAL_OUT and NF_IP_LOCAL_IN are used for altering the destination of local packets. This table is slightly different from the `filter' table, in that only the first packet of a new connection will traverse the table: the result of this traversal is then applied to all future packets in the same connection. 3.2.2.1. Masquerading, Port Forwarding, Transparent Proxying I divide NAT into Source NAT (where the first packet has its source altered), and Destination NAT (the first packet has its destination altered). Masquerading is a special form of Source NAT: port forwarding and transparent proxying are special forms of Destination NAT. These are now all done using the NAT framework, rather than being independent entities. 3.2.3. Packet Mangling The packet mangling table (the `mangle' table) is used for actual changing of packet information. Example applications are the TOS and TCPMSS targets. The mangle table hooks into all five netfilter hooks. (please note this changed with kernel 2.4.18. Previous kernels didn't have mangle attached to all hooks) 3.3. Connection Tracking Connection tracking is fundamental to NAT, but it is implemented as a separate module; this allows an extension to the packet filtering code to simply and cleanly use connection tracking (the `state' module). 3.4. Other Additions The new flexibility provides both the opportunity to do really funky things, but for people to write enhancements or complete replacements that can be mixed and matched. 4. Information for Programmers I'll let you in on a secret: my pet hamster did all the coding. I was just a channel, a `front' if you will, in my pet's grand plan. So, don't blame me if there are bugs. Blame the cute, furry one. 4.1. Understanding ip_tables iptables simply provides a named array of rules in memory (hence the name `iptables'), and such information as where packets from each hook should begin traversal. After a table is registered, userspace can read and replace its contents using getsockopt() and setsockopt(). iptables does not register with any netfilter hooks: it relies on other modules to do that and feed it the packets as appropriate; a module must register the netfilter hooks and ip_tables separately, and provide the mechanism to call ip_tables when the hook is reached. 4.1.1. ip_tables Data Structures For convenience, the same data structure is used to represent a rule by userspace and within the kernel, although a few fields are only used inside the kernel. Each rule consists of the following parts: 1. A `struct ipt_entry'. 2. Zero or more `struct ipt_entry_match' structures, each with a variable amount (0 or more bytes) of data appended to it. 3. A `struct ipt_entry_target' structure, with a variable amount (0 or more bytes) of data appended to it. The variable nature of the rule gives a huge amount of flexibility for extensions, as we'll see, especially as each match or target can carry an arbitrary amount of data. This does create a few traps, however: we have to watch out for alignment. We do this by ensuring that the `ipt_entry', `ipt_entry_match' and `ipt_entry_target' structures are conveniently sized, and that all data is rounded up to the maximal alignment of the machine using the IPT_ALIGN() macro. The `struct ipt_entry' has the following fields: 1. A `struct ipt_ip' part, containing the specifications for the IP header that it is to match. 2. An `nf_cache' bitfield showing what parts of the packet this rule examined. 3. A `target_offset' field indicating the offset from the beginning of this rule where the ipt_entry_target structure begins. This should always be aligned correctly (with the IPT_ALIGN macro). 4. A `next_offset' field indicating the total size of this rule, including the matches and target. This should also be aligned correctly using the IPT_ALIGN macro. 5. A `comefrom' field used by the kernel to track packet traversal. 6. A `struct ipt_counters' field containing the packet and byte counters for packets which matched this rule. The `struct ipt_entry_match' and `struct ipt_entry_target' are very similar, in that they contain a total (IPT_ALIGN'ed) length field (`match_size' and `target_size' respectively) and a union holding the name of the match or target (for userspace), and a pointer (for the kernel). Because of the tricky nature of the rule data structure, some helper routines are provided: ipt_get_target() This inline function returns a pointer to the target of a rule. IPT_MATCH_ITERATE() This macro calls the given function for every match in the given rule. The function's first argument is the `struct ipt_match_entry', and other arguments (if any) are those supplied to the IPT_MATCH_ITERATE() macro. The function must return either zero for the iteration to continue, or a non-zero value to stop. IPT_ENTRY_ITERATE() This function takes a pointer to an entry, the total size of the table of entries, and a function to call. The functions first argument is the `struct ipt_entry', and other arguments (if any) are those supplied to the IPT_ENTRY_ITERATE() macro. The function must return either zero for the iteration to continue, or a non-zero value to stop. 4.1.2. ip_tables From Userspace Userspace has four operations: it can read the current table, read the info (hook positions and size of table), replace the table (and grab the old counters), and add in new counters. This allows any atomic operation to be simulated by userspace: this is done by the libiptc library, which provides convenience "add/delete/replace" semantics for programs. Because these tables are transferred into kernel space, alignment becomes an issue for machines which have different userspace and kernelspace type rules (eg. Sparc64 with 32-bit userland). These cases are handled by overriding the definition of IPT_ALIGN for these platforms in `libiptc.h'. 4.1.3. ip_tables Use And Traversal The kernel starts traversing at the location indicated by the particular hook. That rule is examined, if the `struct ipt_ip' elements match, each `struct ipt_entry_match' is checked in turn (the match function associated with that match is called). If the match function returns 0, iteration stops on that rule. If it sets the `hotdrop' parameter to 1, the packet will also be immediately dropped (this is used for some suspicious packets, such as in the tcp match function). If the iteration continues to the end, the counters are incremented, the `struct ipt_entry_target' is examined: if it's a standard target, the `verdict' field is read (negative means a packet verdict, positive means an offset to jump to). If the answer is positive and the offset is not that of the next rule, the `back' variable is set, and the previous `back' value is placed in that rule's `comefrom' field. For non-standard targets, the target function is called: it returns a verdict (non-standard targets can't jump, as this would break the static loop-detection code). The verdict can be IPT_CONTINUE, to continue on to the next rule. 4.2. Extending iptables Because I'm lazy, iptables is fairly extensible. This is basically a scam to palm off work onto other people, which is what Open Source is all about (cf. Free Software, which as RMS would say, is about freedom, and I was sitting in one of his talks when I wrote this). Extending iptables potentially involves two parts: extending the kernel, by writing a new module, and possibly extending the userspace program iptables, by writing a new shared library. 4.2.1. The Kernel Writing a kernel module itself is fairly simple, as you can see from the examples. One thing to be aware of is that your code must be re- entrant: there can be one packet coming in from userspace, while another arrives on an interrupt. In fact in SMP there can be one packet on an interrupt per CPU in 2.3.4 and above. The functions you need to know about are: init_module() This is the entry-point of the module. It returns a negative error number, or 0 if it successfully registers itself with netfilter. cleanup_module() This is the exit point of the module; it should unregister itself with netfilter. ipt_register_match() This is used to register a new match type. You hand it a `struct ipt_match', which is usually declared as a static (file- scope) variable. ipt_register_target() This is used to register a new type. You hand it a `struct ipt_target', which is usually declared as a static (file-scope) variable. ipt_unregister_target() Used to unregister your target. ipt_unregister_match() Used to unregister your match. One warning about doing tricky things (such as providing counters) in the extra space in your new match or target. On SMP machines, the entire table is duplicated using memcpy for each CPU: if you really want to keep central information, you should see the method used in the `limit' match. 4.2.1.1. New Match Functions New match functions are usually written as a standalone module. It's possible to have these modules extensible in turn, although it's usually not necessary. One way would be to use the netfilter framework's `nf_register_sockopt' function to allows users to talk to your module directly. Another way would be to export symbols for other modules to register themselves, the same way netfilter and ip_tables do. The core of your new match function is the struct ipt_match which it passes to `ipt_register_match()'. This structure has the following fields: list This field is set to any junk, say `{ NULL, NULL }'. name This field is the name of the match function, as referred to by userspace. The name should match the name of the module (i.e., if the name is "mac", the module must be "ipt_mac.o") for auto- loading to work. match This field is a pointer to a match function, which takes the skb, the in and out device pointers (one of which may be NULL, depending on the hook), a pointer to the match data in the rule that is worked on (the structure that was prepared in userspace), the IP offset (non-zero means a non-head fragment), a pointer to the protocol header (i.e., just past the IP header), the length of the data (ie. the packet length minus the IP header length) and finally a pointer to a `hotdrop' variable. It should return non-zero if the packet matches, and can set `hotdrop' to 1 if it returns 0, to indicate that the packet must be dropped immediately. checkentry This field is a pointer to a function which checks the specifications for a rule; if this returns 0, then the rule will not be accepted from the user. For example, the "tcp" match type will only accept tcp packets, and so if the `struct ipt_ip' part of the rule does not specify that the protocol must be tcp, a zero is returned. The tablename argument allows your match to control what tables it can be used in, and the `hook_mask' is a bitmask of hooks this rule may be called from: if your match does not make sense from some netfilter hooks, you can avoid that here. destroy This field is a pointer to a function which is called when an entry using this match is deleted. This allows you to dynamically allocate resources in checkentry and clean them up here. me This field is set to `THIS_MODULE', which gives a pointer to your module. It causes the usage-count to go up and down as rules of that type are created and destroyed. This prevents a user removing the module (and hence cleanup_module() being called) if a rule refers to it. 4.2.1.2. New Targets If your target alters the packet (ie. the headers or the body), it must call skb_unshare() to copy the packet in case it is cloned: otherwise any raw sockets which have a clone of the skbuff will see the alterations (ie. people will see wierd stuff happening in tcpdump). New targets are also usually written as a standalone module. The discussions under the above section on `New Match Functions' apply equally here. The core of your new target is the struct ipt_target that it passes to ipt_register_target(). This structure has the following fields: list This field is set to any junk, say `{ NULL, NULL }'. name This field is the name of the target function, as referred to by userspace. The name should match the name of the module (i.e., if the name is "REJECT", the module must be "ipt_REJECT.o") for auto-loading to work. target This is a pointer to the target function, which takes the skbuff, the hook number, the input and output device pointers (either of which may be NULL), a pointer to the target data, and the position of the rule in the table. The target function may return either IPT_CONTINUE (-1) if traversing should continue, or a netfilter verdict (NF_DROP, NF_ACCEPT, NF_STOLEN etc.). checkentry This field is a pointer to a function which checks the specifications for a rule; if this returns 0, then the rule will not be accepted from the user. destroy This field is a pointer to a function which is called when an entry using this target is deleted. This allows you to dynamically allocate resources in checkentry and clean them up here. me This field is set to `THIS_MODULE', which gives a pointer to your module. It causes the usage-count to go up and down as rules with this as a target are created and destroyed. This prevents a user removing the module (and hence cleanup_module() being called) if a rule refers to it. 4.2.1.3. New Tables You can create a new table for your specific purpose if you wish. To do this, you call `ipt_register_table()', with a `struct ipt_table', which has the following fields: list This field is set to any junk, say `{ NULL, NULL }'. name This field is the name of the table function, as referred to by userspace. The name should match the name of the module (i.e., if the name is "nat", the module must be "iptable_nat.o") for auto-loading to work. table This is a fully-populated `struct ipt_replace', as used by userspace to replace a table. The `counters' pointer should be set to NULL. This data structure can be declared `__initdata' so it is discarded after boot. valid_hooks This is a bitmask of the IPv4 netfilter hooks you will enter the table with: this is used to check that those entry points are valid, and to calculate the possible hooks for ipt_match and ipt_target `checkentry()' functions. lock This is the read-write spinlock for the entire table; initialize it to RW_LOCK_UNLOCKED. private This is used internally by the ip_tables code. 4.2.2. Userspace Tool Now you've written your nice shiny kernel module, you may want to control the options on it from userspace. Rather than have a branched version of iptables for each extension, I use the very latest 90's technology: furbies. Sorry, I mean shared libraries. New tables generally don't require any extension to iptables: the user just uses the `-t' option to make it use the new table. The shared library should have an `_init()' function, which will automatically be called upon loading: the moral equivalent of the kernel module's `init_module()' function. This should call `register_match()' or `register_target()', depending on whether your shared library provides a new match or a new target. You need to provide a shared library: this can be used to initialize part of the structure, or provide additional options. I now insist on a shared library even if it doesn't do anything, to reduce problem reports where the shares libraries are missing. There are useful functions described in the `iptables.h' header, especially: check_inverse() checks if an argument is actually a `!', and if so, sets the `invert' flag if not already set. If it returns true, you should increment optind, as done in the examples. string_to_number() converts a string into a number in the given range, returning -1 if it is malformed or out of range. `string_to_number' rely on `strtol' (see the manpage), meaning that a leading "0x" would make the number be in Hexadecimal base, a leading "0" would make it be in Octal base. exit_error() should be called if an error is found. Usually the first argument is `PARAMETER_PROBLEM', meaning the user didn't use the command line correctly. 4.2.2.1. New Match Functions Your shared library's _init() function hands `register_match()' a pointer to a static `struct iptables_match', which has the following fields: next This pointer is used to make a linked list of matches (such as used for listing rules). It should be set to NULL initially. name The name of the match function. This should match the library name (eg "tcp" for `libipt_tcp.so'). version Usually set to the IPTABLES_VERSION macro: this is used to ensure that the iptables binary doesn't pick up the wrong shared libraries by mistake. size The size of the match data for this match; you should use the IPT_ALIGN() macro to ensure it is correctly aligned. userspacesize For some matches, the kernel changes some fields internally (the `limit' target is a case of this). This means that a simple `memcmp()' is insufficient to compare two rules (required for delete-matching-rule functionality). If this is the case, place all the fields which do not change at the start of the structure, and put the size of the unchanging fields here. Usually, however, this will be identical to the `size' field. help A function which prints out the option synopsis. init This can be used to initialize the extra space (if any) in the ipt_entry_match structure, and set any nfcache bits; if you are examining something not expressible using the contents of `linux/include/netfilter_ipv4.h', then simply OR in the NFC_UNKNOWN bit. It will be called before `parse()'. parse This is called when an unrecognized option is seen on the command line: it should return non-zero if the option was indeed for your library. `invert' is true if a `!' has already been seen. The `flags' pointer is for the exclusive use of your match library, and is usually used to store a bitmask of options which have been specified. Make sure you adjust the nfcache field. You may extend the size of the `ipt_entry_match' structure by reallocating if necessary, but then you must ensure that the size is passed through the IPT_ALIGN macro. final_check This is called after the command line has been parsed, and is handed the `flags' integer reserved for your library. This gives you a chance to check that any compulsory options have been specified, for example: call `exit_error()' if this is the case. print This is used by the chain listing code to print (to standard output) the extra match information (if any) for a rule. The numeric flag is set if the user specified the `-n' flag. save This is the reverse of parse: it is used by `iptables-save' to reproduce the options which created the rule. extra_opts This is a NULL-terminated list of extra options which your library offers. This is merged with the current options and handed to getopt_long; see the man page for details. The return code for getopt_long becomes the first argument (`c') to your `parse()' function. There are extra elements at the end of this structure for use internally by iptables: you don't need to set them. 4.2.2.2. New Targets Your shared library's _init() function hands `register_target()' it a pointer to a static `struct iptables_target', which has similar fields to the iptables_match structure detailed above. 4.2.3. Using `libiptc' libiptc is the iptables control library, designed for listing and manipulating rules in the iptables kernel module. While its current use is for the iptables program, it makes writing other tools fairly easy. You need to be root to use these functions. The kernel tables themselves are simply a table of rules, and a set of numbers representing entry points. Chain names ("INPUT", etc) are provided as an abstraction by the library. User defined chains are labelled by inserting an error node before the head of the user- defined chain, which contains the chain name in the extra data section of the target (the builtin chain positions are defined by the three table entry points). The following standard targets are supported: ACCEPT, DROP, QUEUE (which are translated to NF_ACCEPT, NF_DROP, and NF_QUEUE, respectively), RETURN (which is translated to a special IPT_RETURN value handled by ip_tables), and JUMP (which is translated from the chain name to an actual offset within the table). When `iptc_init()' is called, the table, including the counters, is read. This table is manipulated by the `iptc_insert_entry()', `iptc_replace_entry()', `iptc_append_entry()', `iptc_delete_entry()', `iptc_delete_num_entry()', `iptc_flush_entries()', `iptc_zero_entries()', `iptc_create_chain()' `iptc_delete_chain()', and `iptc_set_policy()' functions. The table changes are not written back until the `iptc_commit()' function is called. This means it is possible for two library users operating on the same chain to race each other; locking would be required to prevent this, and it is not currently done. There is no race with counters, however; counters are added back in to the kernel in such a way that counter increments between the reading and writing of the table still show up in the new table. There are various helper functions: iptc_first_chain() This function returns the first chain name in the table. iptc_next_chain() This function returns the next chain name in the table: NULL means no more chains. iptc_builtin() Returns true if the given chain name is the name of a builtin chain. iptc_first_rule() This returns a pointer to the first rule in the given chain name: NULL for an empty chain. iptc_next_rule() This returns a pointer to the next rule in the chain: NULL means the end of the chain. iptc_get_target() This gets the target of the given rule. If it's an extended target, the name of that target is returned. If it's a jump to another chain, the name of that chain is returned. If it's a verdict (eg. DROP), that name is returned. If it has no target (an accounting-style rule), then the empty string is returned. Note that this function should be used instead of using the value of the `verdict' field of the ipt_entry structure directly, as it offers the above further interpretations of the standard verdict. iptc_get_policy() This gets the policy of a builtin chain, and fills in the `counters' argument with the hit statistics on that policy. iptc_strerror() This function returns a more meaningful explanation of a failure code in the iptc library. If a function fails, it will always set errno: this value can be passed to iptc_strerror() to yield an error message. 4.3. Understanding NAT Welcome to Network Address Translation in the kernel. Note that the infrastructure offered is designed more for completeness than raw efficiency, and that future tweaks may increase the efficiency markedly. For the moment I'm happy that it works at all. NAT is separated into connection tracking (which doesn't manipulate packets at all), and the NAT code itself. Connection tracking is also designed to be used by an iptables modules, so it makes subtle distinctions in states which NAT doesn't care about. 4.3.1. Connection Tracking Connection tracking hooks into high-priority NF_IP_LOCAL_OUT and NF_IP_PRE_ROUTING hooks, in order to see packets before they enter the system. The nfct field in the skb is a pointer to inside the struct ip_conntrack, at one of the infos[] array. Hence we can tell the state of the skb by which element in this array it is pointing to: this pointer encodes both the state structure and the relationship of this skb to that state. The best way to extract the `nfct' field is to call `ip_conntrack_get()', which returns NULL if it's not set, or the connection pointer, and fills in ctinfo which describes the relationship of the packet to that connection. This enumerated type has several values: IP_CT_ESTABLISHED The packet is part of an established connection, in the original direction. IP_CT_RELATED The packet is related to the connection, and is passing in the original direction. IP_CT_NEW The packet is trying to create a new connection (obviously, it is in the original direction). IP_CT_ESTABLISHED + IP_CT_IS_REPLY The packet is part of an established connection, in the reply direction. IP_CT_RELATED + IP_CT_IS_REPLY The packet is related to the connection, and is passing in the reply direction. Hence a reply packet can be identified by testing for >= IP_CT_IS_REPLY. 4.4. Extending Connection Tracking/NAT These frameworks are designed to accommodate any number of protocols and different mapping types. Some of these mapping types might be quite specific, such as a load-balancing/fail-over mapping type. Internally, connection tracking converts a packet to a "tuple", representing the interesting parts of the packet, before searching for bindings or rules which match it. This tuple has a manipulatable part, and a non-manipulatable part; called "src" and "dst", as this is the view for the first packet in the Source NAT world (it'd be a reply packet in the Destination NAT world). The tuple for every packet in the same packet stream in that direction is the same. For example, a TCP packet's tuple contains the manipulatable part: source IP and source port, the non-manipulatable part: destination IP and the destination port. The manipulatable and non-manipulatable parts do not need to be the same type though; for example, an ICMP packet's tuple contains the manipulatable part: source IP and the ICMP id, and the non-manipulatable part: the destination IP and the ICMP type and code. Every tuple has an inverse, which is the tuple of the reply packets in the stream. For example, the inverse of an ICMP ping packet, icmp id 12345, from 192.168.1.1 to 1.2.3.4, is a ping-reply packet, icmp id 12345, from 1.2.3.4 to 192.168.1.1. These tuples, represented by the `struct ip_conntrack_tuple', are used widely. In fact, together with the hook the packet came in on (which has an effect on the type of manipulation expected), and the device involved, this is the complete information on the packet. Most tuples are contained within a `struct ip_conntrack_tuple_hash', which adds a doubly linked list entry, and a pointer to the connection that the tuple belongs to. A connection is represented by the `struct ip_conntrack': it has two `struct ip_conntrack_tuple_hash' fields: one referring to the direction of the original packet (tuplehash[IP_CT_DIR_ORIGINAL]), and one referring to packets in the reply direction (tuplehash[IP_CT_DIR_REPLY]). Anyway, the first thing the NAT code does is to see if the connection tracking code managed to extract a tuple and find an existing connection, by looking at the skbuff's nfct field; this tells us if it's an attempt on a new connection, or if not, which direction it is in; in the latter case, then the manipulations determined previously for that connection are done. If it was the start of a new connection, we look for a rule for that tuple, using the standard iptables traversal mechanism, on the `nat' table. If a rule matches, it is used to initialize the manipulations for both that direction and the reply; the connection-tracking code is told that the reply it should expect has changed. Then, it's manipulated as above. If there is no rule, a `null' binding is created: this usually does not map the packet, but exists to ensure we don't map another stream over an existing one. Sometimes, the null binding cannot be created, because we have already mapped an existing stream over it, in which case the per-protocol manipulation may try to remap it, even though it's nominally a `null' binding. 4.4.1. Standard NAT Targets NAT targets are like any other iptables target extensions, except they insist on being used only in the `nat' table. Both the SNAT and DNAT targets take a `struct ip_nat_multi_range' as their extra data; this is used to specify the range of addresses a mapping is allowed to bind into. A range element, `struct ip_nat_range' consists of an inclusive minimum and maximum IP address, and an inclusive maximum and minimum protocol-specific value (eg. TCP ports). There is also room for flags, which say whether the IP address can be mapped (sometimes we only want to map the protocol-specific part of a tuple, not the IP), and another to say that the protocol-specific part of the range is valid. A multi-range is an array of these `struct ip_nat_range' elements; this means that a range could be "1.1.1.1-1.1.1.2 ports 50-55 AND 1.1.1.3 port 80". Each range element adds to the range (a union, for those who like set theory). 4.4.2. New Protocols 4.4.2.1. Inside The Kernel Implementing a new protocol first means deciding what the manipulatable and non-manipulatable parts of the tuple should be. Everything in the tuple has the property that it identifies the stream uniquely. The manipulatable part of the tuple is the part you can do NAT with: for TCP this is the source port, for ICMP it's the icmp ID; something to use as a "stream identifier". The non-manipulatable part is the rest of the packet that uniquely identifies the stream, but we can't play with (eg. TCP destination port, ICMP type). Once you've decided this, you can write an extension to the connection-tracking code in the directory, and go about populating the `ip_conntrack_protocol' structure which you need to pass to `ip_conntrack_register_protocol()'. The fields of `struct ip_conntrack_protocol' are: list Set it to '{ NULL, NULL }'; used to sew you into the list. proto Your protocol number; see `/etc/protocols'. name The name of your protocol. This is the name the user will see; it's usually best if it's the canonical name in `/etc/protocols'. pkt_to_tuple The function which fills out the protocol specific parts of the tuple, given the packet. The `datah' pointer points to the start of your header (just past the IP header), and the datalen is the length of the packet. If the packet isn't long enough to contain the header information, return 0; datalen will always be at least 8 bytes though (enforced by framework). invert_tuple This function is simply used to change the protocol-specific part of the tuple into the way a reply to that packet would look. print_tuple This function is used to print out the protocol-specific part of a tuple; usually it's sprintf()'d into the buffer provided. The number of buffer characters used is returned. This is used to print the states for the /proc entry. print_conntrack This function is used to print the private part of the conntrack structure, if any, also used for printing the states in /proc. packet This function is called when a packet is seen which is part of an established connection. You get a pointer to the conntrack structure, the IP header, the length, and the ctinfo. You return a verdict for the packet (usually NF_ACCEPT), or -1 if the packet is not a valid part of the connection. You can delete the connection inside this function if you wish, but you must use the following idiom to avoid races (see ip_conntrack_proto_icmp.c): if (del_timer(&ct->timeout)) ct->timeout.function((unsigned long)ct); new This function is called when a packet creates a connection for the first time; there is no ctinfo arg, since the first packet is of ctinfo IP_CT_NEW by definition. It returns 0 to fail to create the connection, or a connection timeout in jiffies. Once you've written and tested that you can track your new protocol, it's time to teach NAT how to translate it. This means writing a new module; an extension to the NAT code and go about populating the `ip_nat_protocol' structure which you need to pass to `ip_nat_protocol_register()'. list Set it to '{ NULL, NULL }'; used to sew you into the list. name The name of your protocol. This is the name the user will see; it's best if it's the canonical name in `/etc/protocols' for userspace auto-loading, as we'll see later. protonum Your protocol number; see `/etc/protocols'. manip_pkt This is the other half of connection tracking's pkt_to_tuple function: you can think of it as "tuple_to_pkt". There are some differences though: you get a pointer to the start of the IP header, and the total packet length. This is because some protocols (UDP, TCP) need to know the IP header. You're given the ip_nat_tuple_manip field from the tuple (i.e., the "src" field), rather than the entire tuple, and the type of manipulation you are to perform. in_range This function is used to tell if manipulatable part of the given tuple is in the given range. This function is a bit tricky: we're given the manipulation type which has been applied to the tuple, which tells us how to interpret the range (is it a source range or a destination range we're aiming for?). This function is used to check if an existing mapping puts us in the right range, and also to check if no manipulation is necessary at all. unique_tuple This function is the core of NAT: given a tuple and a range, we're to alter the per-protocol part of the tuple to place it within the range, and make it unique. If we can't find an unused tuple in the range, return 0. We also get a pointer to the conntrack structure, which is required for ip_nat_used_tuple(). The usual approach is to simply iterate the per-protocol part of the tuple through the range, checking `ip_nat_used_tuple()' on it, until one returns false. Note that the null-mapping case has already been checked: it's either outside the range given, or already taken. If IP_NAT_RANGE_PROTO_SPECIFIED isn't set, it means that the user is doing NAT, not NAPT: do something sensible with the range. If no mapping is desirable (for example, within TCP, a destination mapping should not change the TCP port unless ordered to), return 0. print Given a character buffer, a match tuple and a mask, write out the per-protocol parts and return the length of the buffer used. print_range Given a character buffer and a range, write out the per-protocol part of the range, and return the length of the buffer used. This won't be called if the IP_NAT_RANGE_PROTO_SPECIFIED flag wasn't set for the range. 4.4.3. New NAT Targets This is the really interesting part. You can write new NAT targets which provide a new mapping type: two extra targets are provided in the default package: MASQUERADE and REDIRECT. These are fairly simple to illustrate the potential and power of writing a new NAT target. These are written just like any other iptables targets, but internally they will extract the connection and call `ip_nat_setup_info()'. 4.4.4. Protocol Helpers Protocol helpers for connection tracking allow the connection tracking code to understand protocols which use multiple network connections (eg. FTP) and mark the `child' connections as being related to the initial connection, usually by reading the related address out of the data stream. Protocol helpers for NAT do two things: firstly allow the NAT code to manipulate the data stream to change the address contained within it, and secondly to perform NAT on the related connection when it comes in, based on the original connection. 4.4.5. Connection Tracking Helper Modules 4.4.5.1. Description The duty of a connection tracking module is to specify which packets belong to an already established connection. The module has the following means to do that: o Tell netfilter which packets our module is interested in (most helpers operate on a particular port). o Register a function with netfilter. This function is called for every packet which matches the criteria above. o An `ip_conntrack_expect_related()' function which can be called from there to tell netfilter to expect related connections. If there is some additional work to be done at the time the first packet of the expected connection arrives, the module can register a callback function which is called at that time. 4.4.5.2. Structures and Functions Available Your kernel module's init function has to call `ip_conntrack_helper_register()' with a pointer to a `struct ip_conntrack_helper'. This struct has the following fields: list This is the header for the linked list. Netfilter handles this list internally. Just initialize it with `{ NULL, NULL }'. name This is a pointer to a string constant specifying the name of the protocol. ("ftp", "irc", ...) flags A set of flags with one or more out of the following flgs: o IP_CT_HELPER_F_REUSE_EXPECTReuse expectations if the limit (see `max_expected` below) is reached. me A pointer to the module structure of the helper. Intitialize this with the `THIS_MODULE' macro. max_expected Maximum number of unconfirmed (outstanding) expectations. timeout Timeout (in seconds) for each unconfirmed expectation. An expectation is deleted `timeout' seconds after the expectation was issued with the `ip_conntrack_expect_related()' function. tuple This is a `struct ip_conntrack_tuple' which specifies the packets our conntrack helper module is interested in. mask Again a `struct ip_conntrack_tuple'. This mask specifies which bits of tuple are valid. help The function which netfilter should call for each packet matching tuple+mask 4.4.5.3. Example skeleton of a conntrack helper module ______________________________________________________________________ #define FOO_PORT 111 static int foo_expectfn(struct ip_conntrack *new) { /* called when the first packet of an expected connection arrives */ return 0; } static int foo_help(const struct iphdr *iph, size_t len, struct ip_conntrack *ct, enum ip_conntrack_info ctinfo) { /* analyze the data passed on this connection and decide how related packets will look like */ /* update per master-connection private data (session state, ...) */ ct->help.ct_foo_info = ... if (there_will_be_new_packets_related_to_this_connection) { struct ip_conntrack_expect exp; memset(&exp, 0, sizeof(exp)); exp.t = tuple_specifying_related_packets; exp.mask = mask_for_above_tuple; exp.expectfn = foo_expectfn; exp.seq = tcp_sequence_number_of_expectation_cause; /* per slave-connection private data */ exp.help.exp_foo_info = ... ip_conntrack_expect_related(ct, &exp); } return NF_ACCEPT; } static struct ip_conntrack_helper foo; static int __init init(void) { memset(&foo, 0, sizeof(struct ip_conntrack_helper); foo.name = "foo"; foo.flags = IP_CT_HELPER_F_REUSE_EXPECT; foo.me = THIS_MODULE; foo.max_expected = 1; /* one expectation at a time */ foo.timeout = 0; /* expectation never expires */ /* we are interested in all TCP packets with destport 111 */ foo.tuple.dst.protonum = IPPROTO_TCP; foo.tuple.dst.u.tcp.port = htons(FOO_PORT); foo.mask.dst.protonum = 0xFFFF; foo.mask.dst.u.tcp.port = 0xFFFF; foo.help = foo_help; return ip_conntrack_helper_register(&foo); } static void __exit fini(void) { ip_conntrack_helper_unregister(&foo); } ______________________________________________________________________ 4.4.6. NAT helper modules 4.4.6.1. Description NAT helper modules do some application specific NAT handling. Usually this includes on-the-fly manipulation of data: think about the PORT command in FTP, where the client tells the server which IP/port to connect to. Therefor an FTP helper module must replace the IP/port after the PORT command in the FTP control connection. If we are dealing with TCP, things get slightly more complicated. The reason is a possible change of the packet size (FTP example: the length of the string representing an IP/port tuple after the PORT command has changed). If we change the packet size, we have a syn/ack difference between left and right side of the NAT box. (i.e. if we had extended one packet by 4 octets, we have to add this offset to the TCP sequence number of each following packet). Special NAT handling of all related packets is required, too. Take as example again FTP, where all incoming packets of the DATA connection have to be NATed to the IP/port given by the client with the PORT command on the control connection, rather than going through the normal table lookup. o callback for the packet causing the related connection (foo_help) o callback for all related packets (foo_nat_expected) 4.4.6.2. Structures and Functions Available Your nat helper module's `init()' function calls `ip_nat_helper_register()' with a pointer to a `struct ip_nat_helper'. This struct has the following members: list Just again the list header for netfilters internal use. Initialize this with { NULL, NULL }. name A pointer to a string constant with the protocol's name flags A set out of zero, one or more of the following flags: o IP_NAT_HELPER_F_ALWAYSCall the NAT helper for every packet, not only for packets where conntrack has detected an expectation-cause. o IP_NAT_HELPER_F_STANDALONETell the NAT core that this protocol doesn't have a conntrack helper, only a NAT helper. me A pointer to the module structure of the helper. Initialize this using the `THIS_MODULE' macro. tuple a `struct ip_conntrack_tuple' describing which packets our NAT helper is interested in. mask a `struct ip_conntrack_tuple', telling netfilter which bits of tuple are valid. help The help function which is called for each packet matching tuple+mask. expect The expect function which is called for every first packet of an expected connection. This is very similar to writing a connection tracking helper. 4.4.6.3. Example NAT helper module ______________________________________________________________________ #define FOO_PORT 111 static int foo_nat_expected(struct sk_buff **pksb, unsigned int hooknum, struct ip_conntrack *ct, struct ip_nat_info *info) /* called whenever the first packet of a related connection arrives. params: pksb packet buffer hooknum HOOK the call comes from (POST_ROUTING, PRE_ROUTING) ct information about this (the related) connection info &ct->nat.info return value: Verdict (NF_ACCEPT, ...) { /* Change ip/port of the packet to the masqueraded values (read from master->tuplehash), to map it the same way, call ip_nat_setup_info, return NF_ACCEPT. */ } static int foo_help(struct ip_conntrack *ct, struct ip_conntrack_expect *exp, struct ip_nat_info *info, enum ip_conntrack_info ctinfo, unsigned int hooknum, struct sk_buff **pksb) /* called for every packet where conntrack detected an expectation-cause params: ct struct ip_conntrack of the master connection exp struct ip_conntrack_expect of the expectation caused by the conntrack helper for this protocol info (STATE: related, new, established, ... ) hooknum HOOK the call comes from (POST_ROUTING, PRE_ROUTING) pksb packet buffer */ { /* extract information about future related packets (you can share information with the connection tracking's foo_help). Exchange address/port with masqueraded values, insert tuple about related packets */ } static struct ip_nat_helper hlpr; static int __init(void) { int ret; memset(&hlpr, 0, sizeof(struct ip_nat_helper)); hlpr.list = { NULL, NULL }; hlpr.tuple.dst.protonum = IPPROTO_TCP; hlpr.tuple.dst.u.tcp.port = htons(FOO_PORT); hlpr.mask.dst.protonum = 0xFFFF; hlpr.mask.dst.u.tcp.port = 0xFFFF; hlpr.help = foo_help; hlpr.expect = foo_nat_expect; ret = ip_nat_helper_register(hlpr); return ret; } static void __exit(void) { ip_nat_helper_unregister(&hlpr); } ______________________________________________________________________ 4.5. Understanding Netfilter Netfilter is pretty simple, and is described fairly thoroughly in the previous sections. However, sometimes it's necessary to go beyond what the NAT or ip_tables infrastructure offers, or you may want to replace them entirely. One important issue for netfilter (well, in the future) is caching. Each skb has an `nfcache' field: a bitmask of what fields in the header were examined, and whether the packet was altered or not. The idea is that each hook off netfilter OR's in the bits relevant to it, so that we can later write a cache system which will be clever enough to realize when packets do not need to be passed through netfilter at all. The most important bits are NFC_ALTERED, meaning the packet was altered (this is already used for IPv4's NF_IP_LOCAL_OUT hook, to reroute altered packets), and NFC_UNKNOWN, which means caching should not be done because some property which cannot be expressed was examined. If in doubt, simply set the NFC_UNKNOWN flag on the skb's nfcache field inside your hook. 4.6. Writing New Netfilter Modules 4.6.1. Plugging Into Netfilter Hooks To receive/mangle packets inside the kernel, you can simply write a module which registers a "netfilter hook". This is basically an expression of interest at some given point; the actual points are protocol-specific, and defined in protocol-specific netfilter headers, such as "netfilter_ipv4.h". To register and unregister netfilter hooks, you use the functions `nf_register_hook' and `nf_unregister_hook'. These each take a pointer to a `struct nf_hook_ops', which you populate as follows: list Used to sew you into the linked list: set to '{ NULL, NULL }' hook The function which is called when a packet hits this hook point. Your function must return NF_ACCEPT, NF_DROP or NF_QUEUE. If NF_ACCEPT, the next hook attached to that point will be called. If NF_DROP, the packet is dropped. If NF_QUEUE, it's queued. You receive a pointer to an skb pointer, so you can entirely replace the skb if you wish. flush Currently unused: designed to pass on packet hits when the cache is flushed. May never be implemented: set it to NULL. pf The protocol family, eg, `PF_INET' for IPv4. hooknum The number of the hook you are interested in, eg `NF_IP_LOCAL_OUT'. 4.6.2. Processing Queued Packets This interface is currently used by ip_queue; you can register to handle queued packets for a given protocol. This has similar semantics to registering for a hook, except you can block processing the packet, and you only see packets for which a hook has replied `NF_QUEUE'. The two functions used to register interest in queued packets are `nf_register_queue_handler()' and `nf_unregister_queue_handler()'. The function you register will be called with the `void *' pointer you handed it to `nf_register_queue_handler()'. If no-one is registered to handle a protocol, then returning NF_QUEUE is equivalent to returning NF_DROP. Once you have registered interest in queued packets, they begin queueing. You can do whatever you want with them, but you must call `nf_reinject()' when you are finished with them (don't simply kfree_skb() them). When you reinject an skb, you hand it the skb, the `struct nf_info' which your queue handler was given, and a verdict: NF_DROP causes them to be dropped, NF_ACCEPT causes them to continue to iterate through the hooks, NF_QUEUE causes them to be queued again, and NF_REPEAT causes the hook which queued the packet to be consulted again (beware infinite loops). You can look inside the `struct nf_info' to get auxiliary information about the packet, such as the interfaces and hook it was on. 4.6.3. Receiving Commands From Userspace It is common for netfilter components to want to interact with userspace. The method for doing this is by using the setsockopt mechanism. Note that each protocol must be modified to call nf_setsockopt() for setsockopt numbers it doesn't understand (and nf_getsockopt() for getsockopt numbers), and so far only IPv4, IPv6 and DECnet have been modified. Using a now-familiar technique, we register a `struct nf_sockopt_ops' using the nf_register_sockopt() call. The fields of this structure are as follows: list Used to sew it into the linked list: set to '{ NULL, NULL }'. pf The protocol family you handle, eg. PF_INET. set_optmin and set_optmax These specify the (exclusive) range of setsockopt numbers handled. Hence using 0 and 0 means you have no setsockopt numbers. set This is the function called when the user calls one of your setsockopts. You should check that they have NET_ADMIN capability within this function. get_optmin and get_optmax These specify the (exclusive) range of getsockopt numbers handled. Hence using 0 and 0 means you have no getsockopt numbers. get This is the function called when the user calls one of your getsockopts. You should check that they have NET_ADMIN capability within this function. The final two fields are used internally. 4.7. Packet Handling in Userspace Using the libipq library and the `ip_queue' module, almost anything which can be done inside the kernel can now be done in userspace. This means that, with some speed penalty, you can develop your code entirely in userspace. Unless you are trying to filter large bandwidths, you should find this approach superior to in-kernel packet mangling. In the very early days of netfilter, I proved this by porting an embryonic version of iptables to userspace. Netfilter opens the doors for more people to write their own, fairly efficient netmangling modules, in whatever language they want. 5. Translating 2.0 and 2.2 Packet Filter Modules Look at the ip_fw_compat.c file for a simple layer which should make porting quite simple. 6. Netfilter Hooks for Tunnel Writers Authors of tunnel (or encapsulation) drivers should follow two simple rules for the 2.4 kernel (as do the drivers inside the kernel, like net/ipv4/ipip.c): o Release skb->nfct if you're going to make the packet unrecognisable (ie. decapsulating/encapsulating). You don't need to do this if you unwrap it into a *new* skb, but if you're going to do it in place, you must do this. Otherwise: the NAT code will use the old connection tracking information to mangle the packet, with bad consequences. o Make sure the encapsulated packets go through the LOCAL_OUT hook, and decapsulated packets go through the PRE_ROUTING hook (most tunnels use ip_rcv(), which does this for you). Otherwise: the user will not be able to filter as they expect to with tunnels. The canonical way to do the first is to insert code like the following before you wrap or unwrap the packet: /* Tell the netfilter framework that this packet is not the same as the one before! */ #ifdef CONFIG_NETFILTER nf_conntrack_put(skb->nfct); skb->nfct = NULL; #ifdef CONFIG_NETFILTER_DEBUG skb->nf_debug = 0; #endif #endif Usually, all you need to do for the second, is to find where the newly encapsulated packet goes into "ip_send()", and replace it with something like: /* Send "new" packet from local host */ NF_HOOK(PF_INET, NF_IP_LOCAL_OUT, skb, NULL, rt->u.dst.dev, ip_send); Following these rules means that the person setting up the packet filtering rules on the tunnel box will see something like the following sequence for a packet being tunnelled: 1. FORWARD hook: normal packet (from eth0 -> tunl0) 2. LOCAL_OUT hook: encapsulated packet (to eth1). And for the reply packet: 1. LOCAL_IN hook: encapsulated reply packet (from eth1) 2. FORWARD hook: reply packet (from eth1 -> eth0). 7. The Test Suite Within the CVS repository lives a test suite: the more the test suite covers, the greater confidence you can have that changes to the code hasn't quietly broken something. Trivial tests are at least as important as tricky tests: it's the trivial tests which simplify the complex tests (since you know the basics work fine before the complex test gets run). The tests are simple: they are just shell scripts under the testsuite/ subdirectory which are supposed to succeed. The scripts are run in alphabetical order, so `01test' is run before `02test'. Currently there are 5 test directories: 00netfilter/ General netfilter framework tests. 01iptables/ iptables tests. 02conntrack/ connection tracking tests. 03NAT/ NAT tests 04ipchains-compat/ ipchains/ipfwadm compatibility tests Inside the testsuite/ directory is a script called `test.sh'. It configures two dummy interfaces (tap0 and tap1), turns forwarding on, and removes all netfilter modules. Then it runs through the directories above and runs each of their test.sh scripts until one fails. This script takes two optional arguments: `-v' meaning to print out each test as it proceeds, and an optional test name: if this is given, it will skip over all tests until this one is found. 7.1. Writing a Test Create a new file in the appropriate directory: try to number your test so that it gets run at the right time. For example, in order to test ICMP reply tracking (02conntrack/02reply.sh), we need to first check that outgoing ICMPs are tracked properly (02conntrack/01simple.sh). It's usually better to create many small files, each of which covers one area, because it helps to isolate problems immediately for people running the testsuite. If something goes wrong in the test, simply do an `exit 1', which causes failure; if it's something you expect may fail, you should print a unique message. Your test should end with `exit 0' if everything goes OK. You should check the success of every command, either using `set -e' at the top of the script, or appending `|| exit 1' to the end of each command. The helper functions `load_module' and `remove_module' can be used to load modules: you should never rely on autoloading in the testsuite unless that is what you are specifically testing. 7.2. Variables And Environment You have two play interfaces: tap0 and tap1. Their interface addresses are in variables $TAP0 and $TAP1 respectively. They both have netmasks of 255.255.255.0; their networks are in $TAP0NET and $TAP1NET respectively. There is an empty temporary file in $TMPFILE. It is deleted at the end of your test. Your script will be run from the testsuite/ directory, wherever it is. Hence you should access tools (such as iptables) using path starting with `../userspace'. Your script can print out more information if $VERBOSE is set (meaning that the user specified `-v' on the command line). 7.3. Useful Tools There are several useful testsuite tools in the "tools" subdirectory: each one exits with a non-zero exit status if there is a problem. 7.3.1. gen_ip You can generate IP packets using `gen_ip', which outputs an IP packet to standard input. You can feed packets in the tap0 and tap1 by sending standard output to /dev/tap0 and /dev/tap1 (these are created upon first running the testsuite if they don't exist). gen_ip is a simplistic program which is currently very fussy about its argument order. First are the general optional arguments: FRAG=offset,length Generate the packet, then turn it into a fragment at the following offset and length. MF Set the `More Fragments' bit on the packet. MAC=xx:xx:xx:xx:xx:xx Set the source MAC address on the packet. TOS=tos Set the TOS field on the packet (0 to 255). Next come the compulsory arguments: source ip Source IP address of the packet. dest ip Destination IP address of the packet. length Total length of the packet, including headers. protocol Protocol number of the packet, eg 17 = UDP. Then the arguments depend on the protocol: for UDP (17), they are the source and destination port numbers. For ICMP (1), they are the type and code of the ICMP message: if the type is 0 or 8 (ping-reply or ping), then two additional arguments (the ID and sequence fields) are required. For TCP, the source and destination ports, and flags ("SYN", "SYN/ACK", "ACK", "RST" or "FIN") are required. There are three optional arguments: "OPT=" followed by a comma-separated list of options, "SYN=" followed by a sequence number, and "ACK=" followed by a sequence number. Finally, the optional argument "DATA" indicates that the payload of the TCP packet is to be filled with the contents of standard input. 7.3.2. rcv_ip You can see IP packets using `rcv_ip', which prints out the command line as close as possible to the original value fed to gen_ip (fragments are the exception). This is extremely useful for analyzing packets. It takes two compulsory arguments: wait time The maximum time in seconds to wait for a packet from standard input. iterations The number of packets to receive. There is one optional argument, "DATA", which causes the payload of a TCP packet to be printed on standard output after the packet header. The standard way to use `rcv_ip' in a shell script is as follows: # Set up job control, so we can use & in shell scripts. set -m # Wait two seconds for one packet from tap0 ../tools/rcv_ip 2 1 < /dev/tap0 > $TMPFILE & # Make sure that rcv_ip has started running. sleep 1 # Send a ping packet ../tools/gen_ip $TAP1NET.2 $TAP0NET.2 100 1 8 0 55 57 > /dev/tap1 || exit 1 # Wait for rcv_ip, if wait %../tools/rcv_ip; then : else echo rcv_ip failed: cat $TMPFILE exit 1 fi 7.3.3. gen_err This program takes a packet (as generated by gen_ip, for example) on standard input, and turns it into an ICMP error. It takes three arguments: a source IP address, a type and a code. The destination IP address will be set to the source IP address of the packet fed in standard input. 7.3.4. local_ip This takes a packet from standard input and injects it into the system from a raw socket. This give the appearance of a locally-generated packet (as separate from feeding a packet in one of the ethertap devices, which looks like a remotely-generated packet). 7.4. Random Advice All the tools assume they can do everything in one read or write: this is true for the ethertap devices, but might not be true if you're doing something tricky with pipes. dd can be used to cut packets: dd has an obs (output block size) option which can be used to make it output the packet in a single write. Test for success first: eg. testing that packets are successfully blocked. First test that packets pass through normally, then test that some packets are blocked. Otherwise an unrelated failure could be stopping the packets... Try to write exact tests, not `throw random stuff and see what happens' tests. If an exact test goes wrong, it's a useful thing to know. If a random test goes wrong once, it doesn't help much. If a test fails without a message, you can add `-x' to the top line of the script (ie. `#! /bin/sh -x') to see what commands it's running. If a test fails randomly, check for random network traffic interfering (try downing all your external interfaces). Sitting on the same network as Andrew Tridgell, I tend to get plagued by Windows broadcasts, for example. 8. Motivation As I was developing ipchains, I realized (in one of those blinding- flash-while-waiting-for-entree moments in a Chinese restaurant in Sydney) that packet filtering was being done in the wrong place. I can't find it now, but I remember sending mail to Alan Cox, who kind of said `why don't you finish what you're doing, first, even though you're probably right'. In the short term, pragmatism was to win over The Right Thing. After I finished ipchains, which was initially going to be a minor modification of the kernel part of ipfwadm, and turned into a larger rewrite, and wrote the HOWTO, I became aware of just how much confusion there is in the wider Linux community about issues like packet filtering, masquerading, port forwarding and the like. This is the joy of doing your own support: you get a closer feel for what the users are trying to do, and what they are struggling with. Free software is most rewarding when it's in the hands of the most users (that's the point, right?), and that means making it easy. The architecture, not the documentation, was the key flaw. So I had the experience, with the ipchains code, and a good idea of what people out there were doing. There were only two problems. Firstly, I didn't want to get back into security. Being a security consultant is a constant moral tug-of-war between your conscience and your wallet. At a fundamental level, you are selling the feeling of security, which is at odds with actual security. Maybe working in a military setting, where they understand security, it'd be different. The second problem is that newbie users aren't the only concern; an increasing number of large companies and ISPs are using this stuff. I needed reliable input from that class of users if it was to scale to tomorrow's home users. These problems were resolved, when I ran into David Bonn, of WatchGuard fame, at Usenix in July 1998. They were looking for a Linux kernel coder; in the end we agreed that I'd head across to their Seattle offices for a month and we'd see if we could hammer out an agreement whereby they'd sponsor my new code, and my current support efforts. The rate we agreed on was more than I asked, so I didn't take a pay cut. This means I don't have to even think about external conslutting for a while. Exposure to WatchGuard gave me exposure to the large clients I need, and being independent from them allowed me to support all users (eg. WatchGuard competitors) equally. So I could have simply written netfilter, ported ipchains over the top, and been done with it. Unfortunately, that would leave all the masquerading code in the kernel: making masquerading independent from filtering is the one of the major wins point of moving the packet filtering points, but to do that masquerading also needed to be moved over to the netfilter framework as well. Also, my experience with ipfwadm's `interface-address' feature (the one I removed in ipchains) had taught me that there was no hope of simply ripping out the masquerading code and expecting someone who needed it to do the work of porting it onto netfilter for me. So I needed to have at least as many features as the current code; preferably a few more, to encourage niche users to become early adopters. This means replacing transparent proxying (gladly!), masquerading and port forwarding. In other words, a complete NAT layer. Even if I had decided to port the existing masquerading layer, instead of writing a generic NAT system, the masquerading code was showing its age, and lack of maintenance. See, there was no masquerading maintainer, and it shows. It seems that serious users generally don't use masquerading, and there aren't many home users up to the task of doing maintenance. Brave people like Juan Ciarlante were doing fixes, but it had reached to the stage (being extended over and over) that a rewrite was needed. Please note that I wasn't the person to do a NAT rewrite: I didn't use masquerading any more, and I'd not studied the existing code at the time. That's probably why it took me longer than it should have. But the result is fairly good, in my opinion, and I sure as hell learned a lot. No doubt the second version will be even better, once we see how people use it. 9. Thanks Thanks to those who helped, expecially Harald Welte for writing the Protocol Helpers section.