ChangeLog of IP Virtual Server Netfilter module for Linux kernel 2.4

IP Virtual Server Netfilter module - Version 0.8.2 - October 23, 2001

Changes:
	* Julian added copy-on-write of skb before mangling it in the
	ip_vs_nat_transmitter.

	* Julian added more strict tcp header checking of making sure the
	doff >= sizeof(struct tcphdr) in the function ip_vs_proto_doff.

	* Added ip_vs_csum_partial as the warpper of csum_partial to
	return the sum directly if the buffer length is zero. It can
	bypass that csum_patial/i386 return incorrect result when the
	buffer length is zero and the buffer is not double word aligned,
	which may cause VS/NAT drop some packet because of checksum
	failure.

	* Added ip_defrag() to reassemble fragments before mangling
	packets in the ip_vs_out(). It is needed when no other netfilter
	modules are loaded in the kernel, because ip_vs_out is hooked at
	the FORWARD chain and may receive fragments.


IP Virtual Server Netfilter module - Version 0.8.1 - June 22, 2001

Changes:
	* Fixed the missing break statement bug in ip_vs_set_state. Thank
	Herve Masson <herve@mindstep.com> for reporting it.

	* Fixed the missing atomic_inc(&cp->in_pkts) bug in
	ip_vs_in/ip_vs_in_icmp/ip_vs_leave.

	* Julian changed to use OR operation on skb->nfcache for socket
	buffer belongs to ipvs, instead of assignment operation. Then, it
	will not conflict with other netfilter modules.

	* ipvsadm updates

	fixed the unknown option bug of the first command with long name
	in the popt version.


IP Virtual Server Netfilter module - Version 0.8.0 - May 9, 2001

Changes:
	* Removed some unused function prototypes in the ip_vs.h

	* Removed the use_dst sysctl variable.


IP Virtual Server Netfilter module - Version 0.2.12 - May 5, 2001

Changes:
	* Adapted the ip_select_ident call in the ip_vs_tunnel_xmit to the
	change of kernel 2.4.4, and set skb->ip_summed CHECKSUM_NONE,
	which might be useful.


IP Virtual Server Netfilter module - Version 0.2.11 - April 20, 2001

Changes:
	* Fixed an old bug in the ip_vs_unbind_dest. Its
	atomic_dec_and_test may have race with __ip_vs_del_dest. Changed
	atomic_dec_and_test to atomic_dec in the ip_vs_unbind_dest,
	because the destination will be either in a service's destination
	list or in the destination trash and there is no need to release
	it here. Thank Julian for pointing it out.

	* Fixed the possible memory leakage that the destinations in the
	trash haven't been released after ipvs module exit,
	ip_vs_trash_cleanup was added to clean up the destinations in the
	trash before module exit.


IP Virtual Server Netfilter module - Version 0.2.10 - April 19, 2001

Changes:
	* Fixed the possible race that the service is referred in the
	meanwhile it may be released by ip_vs_flush. The ip_vs_flush need
	to wait until all the service users leave, before releasing the
	service.

	* Fixed the missing __ip_vs_dst_reset bug in the
	ip_vs_unbind_dest.  Before the destination server is released in
	the ip_vs_unbind_dest, the __refcnt of dst_cache need to be
	decreased.


IP Virtual Server Netfilter module - Version 0.2.9 - April 12, 2001

Changes:
	* Fixed the negative connection counting problem. The possible
	race in connection state transition on SMP would lead to the
	negative counting problem, a spinlock was added to make core state
	transition atomic.  Thank Jason Steenblik for reporting the
	problem and testing the fix.

	* Changed to declare the flags, state and timeout members of
	struct ip_vs_conn volatile.

	* Changed to binding application helper and setting state of the
	connection entry before hashing it in the connection table.


IP Virtual Server Netfilter module - Version 0.2.8 - April 4, 2001

Changes:
	* Julian fixed the wrong ip_vs_conn_put in icmp handling code.

	* Fixed the wrong setting expiration of ipvs connection entries
	which would lead to the wrong add_sltimer/del_sltimer calls in the
	SMP environment.

	* Removed the unnecessary del_sltimer in the ip_vs_sched_persist.

	* Code clarification changing ip_vs_table to ip_vs_conn_tab.
	adding "static" to local variables.  removing the used variables
	in the ip_vs.h removing the unnecessary "#define EXPORT_SYMTAB" in
	the code.

	* Julian added the NO_CPORT checking in the ip_vs_random_dropentry
	and the ip_vs_unbind_dest.

	* Julian and Wensong revisited ip_vs_conn_put, __ip_vs_conn_put
	and ip_vs_conn_expire in the SMP environment, hope the the
	possible race has been removed. Thank Jason Steenblik
	<jsteenblik@zantaz.com> for some testings.


IP Virtual Server Netfilter module - Version 0.2.7 - March 22, 2001

Changes:
	* Removing the unnecessary strict buffer boundary checking,
	because the kernel proc buffer already leave some slack (1024
	bytes) for overruns. Thank Julian for pointing it out for me.

	* Changed to use 64-bit variables for byte counters instead of
	32-bit. Thank Julian and Ratz for suggesting this feature and the
	help.

	* Fixed the wrong printing in /proc/net/ip_vs, which was
	introduced in 0.2.6, and removed all the compiling warnings when
	ipvs is built in the kernel. Thank Julian and Ratz for the help
	too.

	* Added many checking that if schedulers or application helpers
	are not modules, don't increase/decrease their use_count. Thank
	Bruno Cornec <Bruno_Cornec@hp.com> for reporting the kernel crashs
	when everything is built into kernel (no modules).

	* Clarified some declarations of ip_vs_stats and __ip_vs_svc_lock
	so that ipvs can be built into kernel with the SMP option, and
	claried some inconsistent declarations of fwmark too.

	* ipvsadm updates fixed some warnings while compiling
	config_stream.c and  dynamic_array.c.  fixed the error result of
	"ipvsadm -R" and "ipvsadm -S".  added ipvsadm-save and
	ipvsadm-restore man pages.  added the "-n" option into
	ipvsadm-save.  changed Makefile so that it can install ipvsadm.sh,
	ipvsadm-save and ipvsadm-restore to the right places.  added
	trying to request the ip_vs module when the ipvs service isn't
	available. Thank Joe for suggesting this feature.


IP Virtual Server Netfilter module - Version 0.2.6 - March 18, 2001

Changes:
	* ipvsadm updates Horms fixed that ipvsadm cannot read
	/etc/sysconfig/ipvsadm correctly on Red Hat 7. It is caused by
	scanf under the glibc that ships with Red Hat 7 not returning EOF
	as the man page documents it will once input stops. Added some
	code in the config_stream.c so that it can detect ab EOF.  Horms
	tidys up ipvsadm-restore, ipvsadm-save and ipvsadm.sh a little too.

	* Julian fixed the dst_cache usage, i.e. __ip_vs_dst_check() needs
	doing dst_hold(dst) before returning an available dst. Then, the
	__refcnt of dst_cache is correct.

	* Changed ip_vs_ftp.c to handles established sessions only, which
	can speed up ftp handling in the LVS/NAT a little bit.

	* Changed to use records of fixed size and added more boundary
	checking in all the proc info handlers (of /proc/net/ip_vs,
	/proc/net/ip_vs_conn, /proc/net/ip_vs_app), so that possible
	buffer overflow can be avoided. Thanks to Julian for discussions
	on this issues.

	* Changed svc->refcnt updates a little bit. Tidy up some stucts in
	the ip_vs.h and some comments in the code.

	* Added some statistics functionalities in the ipvs. The
	statistics information can be accessed by /proc/net/ip_vs_stats.
	And, thank Julian for a lot discussions on this too.


IP Virtual Server Netfilter module - Version 0.2.5 - February 23, 2001

Changes:
	* Fixed the bug of missing unregistering forward_icmp hook in the
	module exit, which would make system crash after the ip_vs module
	is removed and inserted back.

	* Fixed the wrong alignment of IPVS tcp state transition table by
	changing to put IP_VS_S_SYNACK after IP_VS_S_LISTEN in the
	enumeration. Thank Radu-Adrian Feurdean <raf@chez.com> for
	reporting this bug.

	* Fixed the bug of forgetting binding ftp data connection with its
	destination in ip_vs_ftp.c. It would make ftp service of LVS/NAT
	not working properly. Thank Radu-Adrian Feurdean for the report
	again.

	* Fixed to use (char *)th+(th->doff<<2) to get the starting
	address of DATA in the tcp packet, instead of (char*)&th[1].
	Because there may be tcp options in the tcp packet.


IP Virtual Server Netfilter module - Version 0.2.4 - February 15, 2001

Changes:
	* Changed ip_vs_ctl.c to call scheduler's update-service function
	after a new server is added.

	* Added the Destination Hashing scheduler. The DH scheduler
	assigns connections to servers through looking up a statically
	assigned hash table by their destination IP addresses.

	* Added the Source Hashing scheduler. The SH scheduler assigns
	connections to servers through looking up a statically assigned
	hash table by their source IP addresses.

	* Alexander Atanasov fixed the lblcr scheduler that when when all
	servers in the desination set are of weight 0 or are unavailable,
	falls back to wlc method, instead of return NULL.

	* IPVSADM upgrade to a new version added --connection -c in
	command syntax and man page.

	revisited the SYNOPSIS of ipvsadm command, this one should be more
	correct, thank Joe and Horms for telling me there is some
	inconsistency.

	added the VERSION and SCHEDULERS files.

	changed "-M [netmask]" to "-M netmask".


IP Virtual Server Netfilter module - Version 0.2.3 - January 29, 2001

Changes:
	* Julian fixed a bug in the LBLCR scheduler by adding the missing
	(dest->weight>0) condition in the ip_vs_dest_set_max.

	* Fixed the wrong dep_tristate states in the Config.in of IPVS

	* Alexander Atanasov <alex@ssi.bg> has done many testings on the
	LBLC and LBLCR scheduler, added selecting another server if the
	assigned dest->weight<=0 in the LBLC scheduler, added the checking
	on the ip_vs_dest_set_min and ip_vs_dest_set_max return in the
	LBLCR, otherwise the system would crash.

	* Changed to when the packet of new connections is marked with
	non-zero value, lookup the fwmark-based virtual service first,
	then check normal virtual service if it is not for fwmark-based
	virtual service.

	* Added the ip_vs_forward_icmp hook at the NF_IP_FORWARD chain. It
	is to catch incoming ICMP packets destined for 0.0.0.0/0 but
	belonging to IPVS connections, and forward them to right servers.


IP Virtual Server Netfilter module - Version 0.2.2 - January 26, 2001

Changes:
	* Julian added the ip_vs_post_routing in the POST_ROUTING hook,
	which detects outgoing packets for IPVS connections and let them
	skip the ip_nat_out. Then, VS/NAT works with iptable_nat now.

	* Added several lines of code in ip_fw_compact.c, which can also
	identify outgoing packets for IPVS connections and call ip_vs_out
	to mangle and send them out. Now, VS/NAT works with ipchains too.

	* Added several lines of instructions about how to compile ipvs
	code with the kernel in the README.

IP Virtual Server Netfilter module - Version 0.2.1 - January 15, 2001

Changes:
	* Removed some wrong "static" declarations in ip_vs_ctl.c and
	ip_vs_conn.c.

	* Ported new stuff of ipvsadm for kernel 2.2 to ipvsadm for kernel
	2.4, and  upgrade ipvsadm-1.12 for kernel 2.4 to version 1.14.

	* Added the related Config.in and Makefiles. You can compile ipvs
	within the kernel after copying those files manually.


IP Virtual Server Netfilter module - Version 0.2.0 - January 10, 2001

Changes:
	* Added a "packet_xmit" function in the ip_vs_conn entry, and
	remove the  original ip_vs_forward, which can save several CPU
	cycles in the packet forwarding and is more flexible.

	* Added the "cache_bypass" feature. when no cache server is
	available, cache_bypass sysctl is up and iph->daddr is RTN_UNICAST
	(not local), forward packets to the original destination directly.

	* Clarified all the debug levels in the code.

	* Tidy up the code, such as adding new functions, changing the
	function names, updating the comments.


IP Virtual Server Netfilter module - Version 0.1.2 - December 19, 2000

Changes:
	* Julian added using destination cache to speed up route lookup
	for outgoing packets. The feature can be controlled by the
	following sysctl: /proc/sys/net/ipv4/vs/use_dst It is enabled by
	the default.

	* The bug of repeated sysctl ID for NET_IPV4_VS_LBLC_EXPIRE and
	NET_IPV4_VS_LBLCR_EXPIRE was fixed.

IP Virtual Server Netfilter module - Version 0.1.1 - November 30, 2000

Changes:
	* Added the Locality-Based Least Connection with Replication
	scheduler. In the LBLC scheduling, the load balancer maintains
	mappings from a target to a cache node, when the cache node is
	overloaded, the target may be remapped to the current least loaded
	node. The target will cause the node overloaded again, then will
	be remapped to other node, and so on. This is the "hot spot"
	problem, it may lead to a lot of replications in most nodes. So,
	the LBLC with Replication algorithm is introduced to solve this
	problem, it maintains mappings from a target to a set of nodes
	that can solve this target. So, the "hot spot" can be solved by a
	set of nodes, then "hot spot" can seldom overload one and another
	among all the nodes.

IP Virtual Server Netfilter module - Version 0.1.0 - November 26, 2000

Changes:
	* Julian added ECN support in IPVS tunnelling transmitter

	* Ported the Locality-Based Least Connection scheduler from IPVS
	code for kernel 2.2. Julian replaced del_timer function call with
	with del_timer_sync to avoid the possible race between timer
	handler and del_timer thread in SMP.

	* Julian and Wensong fixed the possible oops in the
	todrop_entry().  When the incoming packet counter of a very busy
	connection exceeds 2^32, accessing an array with a negative index
	leads to oops.

IP Virtual Server Netfilter module - Version 0.0.5 - September 6, 2000

Changes:
	* Rusty fixed the checksum bug in ip_vs_skb_replace of ip_vs_app.c
	iph->tot_len should be updated before the checksum is calculated.

	* Rusty made all local functions and variables static, and cleaned
	up the modules without #ifdef's.

	* Julian added the real service checking back in ip_vs_out.

	* Removed the ip_vs_register_scheduler_module() and the
	ip_vs_unregister_scheduler_module() in the ip_vs_sched.c

	* Julian and Wensong explored race of ipvs connection creation/
	deletion. The original wrong locking was fixed in the
	ip_vs_random_dropentry(). Changed to increase the reference
	counter of a new connection entry before hashing it, so that while
	one CPU is creating a new entry, the other CPU may run the
	ip_vs_random_dropentry() but cannot drop it.

	* Julian fixed the bug in the WRR scheduler, server with weight
	zero is selected when all server weights (>= 2 servers) are zero.
	Thank Ratz for reporting the bug.

	* Added the "-s tcp tcpfin udp" option in the ipvsadm.c, so that
	the timeout values for IPVS connections (tcp tcpfin udp) can be
	changed in the kernel.

	* Fixed the compiling error in sysctl_ip_vs_debug_level when the
	CONFIG_IP_VS_DEBUG is turned off.

IP Virtual Server Netfilter module - Version 0.0.4 - July 30, 2000

Changes:
	* Rusty tidy up the ipvs code during the Ottawa Linux Symposium 1)
	Fix `large ftp' bug: more precision in sequence delta.  2) Reorder
	struct ip_vs_conn to save some space.  3) Firewall marks must be
	`unsigned long' not __u32.  4) kfree_s changed to kfree

	* Julian changed kmalloc connection allocation to slab cache
	allocation. It's good for performance.

	* Added the decreasement of the ip_vs_conn_no_cport_cnt if the
	flag is set with IP_VS_CONN_F_NO_CPORT. It can avoid the wrong
	counting when IP_VS_CONN_F_NO_CPORT connection entries are expired
	or dropped.

IP Virtual Server Netfilter module - Version 0.0.3 - July 6, 2000

Changes:
	* Made IPVS application module and ip_vs_ftp module work.

	* Changed ip_vs_in hook from NF_IP_PRE_ROUTING to NF_IP_LOCAL_IN.
	Then, there is no need to call ip_defrag(), and no source routing
	problem. Firewalling and policy routing rules apply to IPVS. The
	structure looks better.

	* Splitted some code of ip_vs_in into ip_vs_nat_xmit and added MTU
	checking there. It may exist that application modules may  extend
	the sk_buff into a new one and the length of the new one is larger
	than MTU, but this problem should rarely happen.

	* Changed the checksum checking is only done for VS/NAT just
	before mangling pakcet. VS/TUN and VS/DR don't modify packets, so
	let real servers do checksum checking. It can speed up packet
	handling in VS/TUN and VS/DR.

	* Fixed many small bugs, and changed lots of debugging display
	nicer.

	The functionality of IPVS for kernel 2.4 is almost implemented.
	The rest things to do are bug fixes, performance tuning, and
	making it work with other netfilter modules harmonly. Currently,
	if ip_conntrack module is inserted, then there will be two
	tracking for each connection. The iptable_nat module will make the
	passive FTP of VS/NAT not working correctly.

	The above changes were done with Julian together. Thank Julian for
	good suggestions and many bug fixes.

	Note: The version 0.0.2 is for internal developemnt, so it is not
	released publicly.

IP Virtual Server Netfilter module - Version 0.0.1 - May 31, 2000

Changes:
	* Port the IPVS patch for kernel 2.2 as a Netfilter module for
	kernel 2.4. Lots of code were rewritten in order to adapt it to
	the Netfilter framework, and lots of code were taken from the ip
	masquerading code of kernel 2.2. Currently, this draft version can
	just work basically. There are still lots of work to do, such as
	more rewrite, bug fixes and performance improvements for kernel
	2.4.

	* Julian adapted locking for kernel 2.4

	* Julian adapted slow timer handling for kernel 2.4

	* Julian made the module reference correct

	===========================================================================
ChangeLog of Virtual Server patch for Linux 2.2

Virtual Server patch for Linux 2.2.15 - Version 0.9.13 - May 25, 2000

Changes:
	* Changed that the key of real server hash table is generated from
	<addr,port>, instead of <proto,addr,port>, and changed the check
	in __ip_vs_lookup_real_service(). It should work correctly for
	fwmark service.

	* Added the missing "INIT_LIST_HEAD(&dest->d_list);" in
	ip_vs_rs_unhash().

	* Changed that ip_vs_wrr_init_svc() returns -ENOMEM instead of
	ENOMEM if no memory is available, just in order to keep good
	format of return value.

	* Added the ftp service and port zero service counters, which can
	speed up service lookup.

	* Restrict real server ports covered from the ftp service.  The
	port number of ftp data connections on real servers can only be 20
	or >= PROT_SOCK(1024).

	* Split some duplicate code into __ip_vs_del_service()

	The above changes were made by Julian and Wensong.

Virtual Server patch for Linux 2.2.14 - Version 0.9.12 - May 2, 2000

Changes:
	* The condition matching order in __ip_vs_in_get & __ip_vs_out_get
	was optimized, it should speed up the entry lookup when the row is
	long. Thank Julian!

	* Fixed the silly bug that I forgot adding an sysctl ID for the
	/proc/sys/net/ipv4/vs/am_droprate.

	* Julian fixed the wrong return (EINVAL) to the correct one
	(-EINVAL), when the option length is less than the sizeof the
	ip_masq_ctl entry.

Virtual Server patch for Linux 2.2.14 - Version 0.9.11 - April 17, 2000

Changes:
	* Fixed icmp handling bug introduced in adding the fwmark feature

	* Made the dest lookup correctly in trash for both normal service
	and fwmark service.

	* Changed that the ICMP_DEST_UNREACH packet is sent to clients
	only if its entry doesn't exist and it is not RST or not of TCP,
	the ICMP_DEST_UNREACH packet is sent to real servers only if its
	entry doesn't exist and it is not RST or not of TCP.

	* Horms tidy up some description and grammar of ipvsadm man page.

	The above changes were made by Julian, Horms and Wensong together

Virtual Server patch for Linux 2.2.14 - Version 0.9.10 - April 9, 2000

Changes:
	* Julian added the droprate and secure_tcp defense strategies.

	* The dropentry defense strategy was revisited.

	* The fwmark service lookup was added by Horms, Julian and Wensong
	Use a firewall-marking to denote a virtual service instead of a
	triplet <protocol,addr,port>.  The marking of packets with a
	firewall-mark is done by firewalling code. This feature can be
	used to build a virtual service assoicated to different IP
	addresses or port numbers, but sharing the same real servers, such
	as multiple-homed LVS.

Virtual Server patch for Linux 2.2.14 - Version 0.9.9 - March 16, 2000

Changes:
	* Fixed the OUTPUT state transition table It entered from RS to ES
	while receiving SYN packet, which is  not right and will cause
	that randrom drop won't be effective for VS/NAT.

	* Added random drop of UDP entries too before out of memory Thank
	Julian for his cute comments.

Virtual Server patch for Linux 2.2.14 - Version 0.9.8 - March 13, 2000

Changes:
	* Added random drop of syn entries before running out of memory
	When available memory is less than 1024 pages, randomly scan 1/16
	of table to drop entries that are in SYN_RECV state. To  activate
	this random drop functionality, echo 1 >
	/proc/sys/net/ipv4/ip_vs_randomdrop it will be hard for
	distributed syn flooding attack tools to make the LVS box run out
	of memory.

	* Fixed a negative counter bug because of wrong invalid template
	The invalid template like <cip, 65535, vip, 65535, rip, 65535> is
	changed to <cip, 0, vip, 65535, rip, 65535>.

Virtual Server patch for Linux 2.2.14 - Version 0.9.7 - January 19, 2000

Changes:
	* Just resolve a patch rejection on Configure.help for 2.2.14

Virtual Server patch for Linux 2.2 - Version 0.9.7 - December 22, 1999

Changes:
	* Fixed the huge timeout entry bug when destinations are
	unavailable When the destination server of a packet is found
	unavailable, the packet is droped silently but the entry is
	forgotten to be added back to the slow timer table. It would
	generate the entries of huge timeout. Thank Julian for the bug.

	* Changed two IP_VS_ERR calls to IP_VS_DBG Since the ipvsadm would
	report the error information when deleting a nonexist destionation
	or adding an existing service, there is no need to report error
	message in kernel. Thank Julian again for the change.

	* Added the sysctl_ip_always_defrag counting in ip_masq_new_vs
	This is for the coming kernel patch 2.2.14, where the wrong
	sysctl_ip_always_defrag handling is fixed.

Virtual Server patch for Linux 2.2 - Version 0.9.6 - December 7, 1999

Changes:
	* Invalidate a persistent template when its dest is unavailable We
	define templates like <cip, 0, vip, vport, rip, rport>
	(persistence for a single service) or <cip, 0, vip, 0, rip, 0>
	(persistence for all services) are valid, and templates like <cip,
	65535, vip, 65535, rip, 65535> are invalid.  When new connection
	arrives and the destination of its template is not available,
	invalidate the template, then create a new template with new
	destination, and new connection is served.

	* Fixed the wrong debugging information in ip_vs_forward

Virtual Server patch for Linux 2.2 - Version 0.9.5 - November 28, 1999

Changes:
	* Fixed the undefined variable bug in the IP_VS_DBG Due to my
	carelessness, an undefined variable was left in the IP_VS_DBG
	statement of the ip_vs_dr_xmit function. Thank Roberto Nibali for
	reporting.

	* Changed ICMP_PROT_UNREACH to ICMP_PORT_UNREACH in ip_vs_leave
	When virtual service is available but no destination is available,
	The ICMP_PORT_UNREACH icmp packet is sent to notify the client
	that the service is not available. Since IPVS is in IP layer,  the
	TCP socket has been created, the TCP RST packet cannot be sent for
	TCP services, instead that ICMP_PORT_UNREACH is sent, no  matter
	it talks TCP/UDP. Thank Julian.

	* Added port zero support for persistent services For some
	applications, there are more than one service, once a client is
	assigned to a real server for the first service, requests for
	other services from the same clients must be sent to the same
	server. Port zero is added for this kind of persistent services.

	* Fixed the bug that virtual ftp service blocks other services
	When virtual ftp service is presented and packets destined for
	other services not listed in ipvs table arrives, wrong masq
	entries will be created and those services are blocked.

	* Fixed the (null) print for unknown services in ipvsadm Thank
	Julian for reporting.


Virtual Server patch for Linux 2.2 - Version 0.9.4 - November 10, 1999

Changes:
	* Julian fixed the fatal return bug of ip_vs_leave() Since some
	code of last version ipvs is changed, ip_vs_leave should return -2
	instead of -3 if no virtual service is  found.

	* Added the IPSKB_REDIRECTED flag The skb is set with the
	IPSKB_REDIRECTED and IPSKB_MASQUERADED flag, so that the system
	can detect infinite loop of TUNNELED/ DROUTED packets in the
	ip_local_deliver caused by misconfiguration.  For example, user
	might configure the following: ipvsadm -a -t VIP:http -r
	<non-local IP address> -i ifconfig <an interface> <the IP address
	above> up then packets for VIP:http is tunneled to its own
	interface, which will causes infinite loop.

	* Fixed the bug that freed skb may be used to masq_set_state In
	the original ip_fw_demasquerade function, masq_set_state was
	called after ip_vs_forward, and ip_vs_forward may free the skb, so
	masq_set_state may operate the already freed skb. The current
	solution is just to simply do masq_set_state before ip_vs_forward.
	No matter whether the packet is forwarded successfully or not, the
	masq state will be updated. Although it brokes the original
	sematics, it won't lead to serious errors. We look forward to
	fixing it under the Rusty's netfilter framework both for
	correctness and modularization. :-)

	Many thanks must go to Julian for his very cute comments to the
	ipvs 0.9.3 code. He also raised a question, could we simply use
	ip_route_output to skip IPv4 forwarding and firewall to tunnel/
	droute packets for a little bit performance, or should we be back
	to ip_route_input for correctness? I am still thinking about it.


Virtual Server patch for Linux 2.2 - Version 0.9.3 - November 7, 1999

Changes:
	* Adapted the patch for kernel 2.2.13 Since the ntohl and like
	were changed to unsigned int(because the unsigned long int is
	64-bit these days), some code in VS patch is modified for this
	change, and the compiling warnings and  unnecessary casting can be
	avoided.

	* Changed the masq timeout type and the maximum persistent timeout
	The type of masq timeout was changed from 'unsigned' to 'unsigned
	long', in order to keep it the same as the type of timer_struct
	expires, then masq timeout will be 64-bit on 64-bit platforms. The
	maximum persistent timeout was changed from one year to one month,
	because this is enough. Thank Julian for the suggestions.

	* Added ICMP handling for IPVS The incoming ICMP packets for
	virtual services will be forwarded to the right real servers, and
	outgoing ICMP packets from virtual services will be altered and
	send out correctly. This is important for error and control
	notification between clients and servers, such as the MTU
	discovery. Sorry for adding this stuff so late, because I used to
	stupidly think that it is not easy to add ICMP  handling for
	IPVS. After spending a couple of hours reading the textbooks and
	the masq code, I found that it was quite easy to add this
	stuff. Sorry!

	* Changed the tunnel/dr/local forwarding without doing
	masq_skb_cow Some orders in the ip_fw_demasquerade and
	ip_fw_demasq_icmp functions, so that the masq skbuff copy-on-write
	can be avoided in the tunnel/ dr/local forwarding methods. This
	improves performance for the  tunnel/dr/local forwarding methods.

	* Use vmalloc to allocate big hash table.  The big IPVS hash table
	of 256K entries or more can be allocated now.


Virtual Server patch for Linux 2.2 - Version 0.9.2 - October 17, 1999

Changes:
	* Added support for netmasks with persistence The client source
	address is masked with this netmask for the purpose of accessing
	the templates. Added a new port to the service structure and
	changed ipvsadm to support this. Defaults to a 255.255.255.255,
	which emulates the old behaviour. (Lars Marowsky-Bree
	<lmb@teuto.net>)

	* Fixed the bug that server status checking doesn't work for
	LVS/NAT, and changed some comestics things for debugging. Thank
	Julian for  the fix.


Virtual Server patch for Linux 2.2 - Version 0.9.1 - October 6, 1999

Changes:
	* Fixed the counting bug in ip_vs_unbind_masq again Don't touch
	counters for templates.

	* Removed extra read_unlock in __ip_vs_lookup_service

	* Changed not to restart template timers if dest is unavailable If
	the client actively send packets when the destination is
	unavailable, the masq template can expire.

	* Added the destination trash  The destination trash is used to
	hold the destinations that  are removed from the service table but
	are still referenced  by some masq entries. The reason to add the
	destination trash  is when the dest is temporary down (either by
	administrator  or by monitor program), the dest can be picked back
	from the  trash, the remaining connections to the dest can
	continue,  and the counting information of the dest is also useful
	for scheduling.

	* Added the ip_vs_leave function It is called by
	ip_fw_demasquerade when the matched service  is avaiable but no
	destination is available for a new  connection, to drop the
	packet. This should be a good behavior.

	* Changed drasticly removing the masq to silently dropping packets
	and keeping the masq in expire, when its destination is not
	available. It is a good behavior, when the destination is
	temporary down.

	The above fixes and changes won't be possible without  Julian
	Anastasov's fixes and suggestions. Thank Julian!

	* Added the handling of weight=0 in every scheduler The
	destination with weight=0 is "quiesced" and will not  receive any
	new connection, but will still serve the existing
	connections. This feature is useful to cool down the overloaded
	servers or to get some servers out of service for maintenance.

	* Added the update_service function in every scheduler When the
	destination list of a service is modified, the  update_service
	function is called to reset the scheduling pointer, so that the
	scheduling pointer won't point to the freed destination.

	* Changed some IP_VS_ERR to IP_VS_DBG in the ip_vs_tunnel_xmit

	* Added different timeout support for persistent service Users can
	specify different timeout values for their different persistent
	services.

	* Fixed the bug that persistent service cannot be edited

	* Changed the output of ip_vs_procinfo for the new version of
	ipvsadm.


Virtual Server patch for Linux 2.2 - Version 0.9.0 - September 24, 1999

Changes:
	*   Added the hash table for virtual services It will greatly
	speedup the lookup of services.

	*   Added new persistent service handling The template is looked
	up only if the service that the packet is destined to is
	persistent, so it is more efficient. For all the persistent
	services except FTP, we create a masq template like <daddr, 0,
	maddr, mport, saddr, sport>. So, the persistent services  won't
	disturb each other, and it fixes the wrong accounting bug for
	different persistent services.  FTP is a very complicated network
	protocol, and it uses control  connection and data
	connections. For active FTP, FTP server initilizes  data
	connection to the client, its source port is often 20. For passive
	FTP, FTP server tells the clients the port that it passively
	listens  to,  and the client issues the data connection. In the
	tunneling or  direct routing mode, the load balancer is on the
	client-to-server half  of connection, the port number is unknown
	to the load balancer. So, a  template masq like <daddr, 0, maddr,
	0, saddr, 0> is created for  persistent FTP service.

	*   Changed the destination lists to the d-linked lists

	*   Changed the scheduler list to the d-linked list

	*   Added back the least connection scheduling module.

	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.8.3 - September 8, 1999

Changes:
	*   Fixed the missing unlock bug in ip_vs_schedule.  If no virtual
	service is found in ip_vs_schedule, this missing unlock bug will
	make system crash.

	*   Fixed the uncounting bug in creating masqs by template.
	Missing to counter connections when creating masqs by template.

	*   Don't touch counters in ip_vs_unbind_masq for templates Thanks
	must go to Julian Anastasov for the three fixes above.

	*   Changed some condition orders for a bit performance

	*   Changed some cosmetic things for debugging

Virtual Server patch for Linux 2.2 - Version 0.8.2 - September 5, 1999

Changes:
	*   Fixed the the IP_MASQ_F_VS_INACTIVE cleared bug after editing
	dest.  Thank Julian Anastasov for the fix.

	*   Added the separate inactive connection counter for each dest
	The WLC sheduler can use this counter directly for scheduling.
	And, the masq template won't be counted in inactive connections.
	Thank Julian Anastasov for the suggestion.

	*   Changed all the schedulers modules to return server dest
	directly, and ip_vs_schedule creates new masq entry itself.

Virtual Server patch for Linux 2.2 - Version 0.8.1 - September 2, 1999

Changes:
	*   Uncomment a few statement to make virtual FTP via NAT really
	work.  Virtual FTP service via NAT really work well no matter it
	is in active or passive mode. But, remember to "insmod
	ip_masq_ftp" before using FTP service through VS-NAT.

	*   Remove some commented out block. The code looks nice. :)


Virtual Server patch for Linux 2.2 - Version 0.8 - September 1, 1999

Changes:
	*   Added the persistent port feature.  Users can specify whether
	the virtual service port is persistent or not. It is more
	flexible. The original PCC scheduling is removed.

	*   Added the dest server status checking.  The server status is
	checked before forwording a packet. If the server is not
	available(down or put out of service), the packet will be dropped
	and the client will be notified immediately.  The server status is
	also checked while generating a masq entry based on the masq
	template. If not available, the new entry won't be created.

	*   Added some code in ip_masq_ftp.c to handle virtual FTP service
	for VS-NAT. The passive handling code in ip_masq_ftp.c never works.

	*   Fixed stepping to mSR after SYN in INPUT_ONLY table.  Thank
	Julian Anastasov for doing it. It make much much harder that a
	LinuxDirector is synflooded to run out of memory.

	*   Fixed huge masq expire bug for after bad checksum.  Thank
	Julian Anastasov for fixing it.

	*   Added the IP_MASQ_F_VS_INACTIVE flag and fixed the connection
	counter Thank Julian Anastasov for the suggestion and fix example.

	*   Fixed the incorrect lookup in hash table.  The ms=NULL
	statement was forgot if no entry is found, this makes the
	incorrect lookup, which may lead to huge masq expire.  Stupid
	mistake, but the result is serious.

	*   Fixed the incorrect slow timer vector layout Correct layout
	and more efficient to use memory.

	*   Fixed the bug of slow timer being added twice for masq template

	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.7 - July 9, 1999

Changes:
	*   Added a separate masq hash table for IPVS.

	*   Added slow timers to expire masq entries.  Slow timers are
	checked in one second by default. Most overhead of cascading
	timers is avoided.

	With this new hash table and slow timers, the system can hold huge
	number of masq entries, but make sure that you have enough free
	memory. One masq entry costs 128 bytes memory effectively (Thank
	Alan Cox), if your box holds 1 million masq entries (it means that
	your box can receive 2000 connections per  second if masq expire
	time is 500 seconds in average.), make sure that you have 128M
	free memory. And, thank Alan for suggesting the early random drop
	algorithm for masq entries that prevents the system from running
	out of memory, I will design and implement this feature in the
	near future.

	*   Fixed the unlocking bug in the ip_vs_del_dest().  Thank Ted
	Pavlic <tpavlic@netwalk.com> for reporting it.

	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.6 - July 1, 1999

Changes:
	*   Fixed the overflow bug in the ip_vs_procinfo().  Thank Ted
	Pavlic <tpavlic@netwalk.com> for reporting it.

	*   Added the functionality to change weight and forwarding
	(dispatching) method of existing real server.  This is useful for
	load-informed scheduling.

	*   Added the functionality to change scheduler of virtual service
	on the fly.

	*   Reorganized some code and changed names of some functions.
	This make the code more readable.

	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.5 - June 22, 1999

Changes:
	*   Fixed the bug that LocalNode doesn't work in vs-0.4-2.2.9.
	Thank Changwon Kim <chwkim@samsung.co.kr> for reporting the bug
	and pointing me the checksum update problem in the code.

	*   some code of VS in the ip_fw_demasquerade was reorganized so
	that the packets for VS-Tunneling, VS-DRouting and LocalNode skip
	the checksum update. This make the code right and efficient


	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.4 - June 1, 1999

Most of the code was rewritten. The locking and refcnt was changed
The violation of "no floats in kernel mode" rule in the weighted
least-connection scheduling was fixed. This patch is more efficient,
and should be more stable.


	----------------------------------------------------------------------

Virtual Server patch for Linux 2.2 - Version 0.1~0.3 - May 1999

Peter Kese <peter.kese@ijs.si> ported the VS patch to kernel 2.2,
rewrote the code and loadable scheduling modules.


	===========================================================================

ChangeLog of Virtual Server patch for Linux 2.0
	----------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.9 - May 1, 1999

Differences with virtual server patch version 0.8:

	*  Add Virtual Server via Direct Routing This approach was first
	implemented in IBM's NetDispatcher. All real servers have their
	loopback alias interface configured with the virtual IP address,
	the load balancer and the real servers must have one of their
	interfaces physically linked by a HUB/Switch. When the packets
	destined for the virtual IP address arrives, the load balnacer
	directly route them to the real servers, the real servers
	processing the requests and return the reply packets directly to
	the clients. Compared to the virtual server via IP tunneling
	approach, this approach doesn't have tunneling overhead(In fact,
	this overhead is minimal in most situations), but requires that
	one of the load balancer's interfaces and the real servers'
	interfaces must be in physical segment.

	*  Add more satistics information The active connection counter
	and the total connection counter of each real server were added
	for all the scheduling algorithms.

	*  Add resetting(zeroing) counters The total connection counters
	of all real servers can be reset to zero.

	*  Change some statements in the masq_expire function and the
	ip_fw_demasquerade function, so that ip_masq_free_ports won't
	become abnormal number after the masquerading entries for virtual
	server are released.

	*  Fix the bug of "double unlock on device queue" Remove the
	unnecessary function call of skb_device_unlock(skb) in the
	ip_pfvs_encapsule function, which sometimes cause "kernel: double
	unlock on device queue" waring in the virtual server via tunneling.

	*  Many functions of virtual server patch was splitted into the
	linux/net/ipv4/ip_masq_pfvs.c.

	*  Upgrade ippfvsadm 1.0.2 to ippfvsadm 1.0.3 Zeroing counters is
	supported in the new version. The ippfvsadm 1.0.3 can be used for
	all kernel with different virtual server options without
	rebuilding the program.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.8 - March 6, 1999

Differences with virtual server patch version 0.7:

	*  Add virtual FTP server support The original ippfvs via IP
	tunneling could not be used to build a virtual FTP server, because
	the real servers could not establish data connections to
	clients. The code was added to parse the port number in the ftp
	control data and create the corresponding masquerading entry for
	the coming data connection.  Although the original ippfvs via NAT
	could be used to build a virtual server, the data connection was
	established in this way.  Real Server port:20  ----> ippfvs:
	allocate a free masq port ----->  the client port It is not
	elegent but time-consuming. Now it was changed as follows: Real
	Server port:20  ----> ippfvs port: 20  ----> the client port

	*  Change the port checking order in the ip_fw_demasquerade() If
	the size of masquerade hash table is well chosen, checking a
	masquerading entry in the hash table will just require one hit. It
	is much efficient than checking port for  virtual services, and
	there are at least 3 incoming packets for each connection, which
	require port checking. So, it is efficient to check the
	masquerading hash table first and then check port for virtual
	services.

	*  Remove a useless statement in the ip_masq_new_pfvs() The
	useless statement in the ip_masq_new_pfvs function is
	ip_masq_free_ports[masq_proto_num(proto)]++; which may disturb
	system.

	*  Change the header printing of the ip_pfvs_procinfo()

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.7 - Febuary 10, 1999

Differences with virtual server patch version 0.6:

	*  Fix a bug in detect the finish of connection for tunneling or
	NATing to the local node.  Since the server reply the client
	directly in tunneling or NATing to the local node, the load
	balancer (LinuxDirector) can only detect a FIN segment. It is
	mistake that the masq entry is removed only if both-side FIN
	segments are detected, and then the masq entry expires in 15
	minutes. For the situation above, the code was changed to set the
	masq entry expire in TCP_FIN_TIMEOUT (2min) when an incoming FIN
	segment is detecting.  *  Add the patch version printing in the
	ip_pfvs_procinfo() It would be easy for users and hackers to know
	which virtual server patch version they are running. Thank Peter
	Kese <peter.kese@ijs.si> for the suggestion.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.6 - Febuary 2, 1999

Differences with virtual server patch version 0.5:

	*  Add the local node feature in virtual server.  If the local
	node feature is enabled, the load balancer can  not only redirect
	the packets of the specified port to the  other servers (remote
	nodes) to process it, but also can process  the packets locally
	(local node). Which node is chosen depends on the scheduling
	algorithms.  This local node feature can be used to build a
	virtual server of a few nodes, for example, 2, 3 or more sites, in
	which it is a  resource waste if the load balancer is only used to
	redirect packets. It is wise to direct some packets to the local
	node to process. This feature can also be used to build
	distributed identical servers, in which one is too busy to handle
	requests locally, then it can seamlessly forward requests to other
	servers to process them.  This feature can be applied to both
	virtual server via NAT and virtual server via IP tunneling.  Thank
	Peter Kese <peter.kese@ijs.si> for idea of "Two node Virtual
	Server" and his single line patch for virtual server via IP
	tunneling.  *  Remove a useless function call ip_send_check in the
	virtual server via IP tunneling code.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.5 - November 25, 1998

Differences with virtual server patch version 0.4:

	*  Add the feature of virtual server via IP tunneling.  If the
	ippfvs is enabled using IP tunneling, the load balancer chooses a
	real server from a cluster based on a scheduling algorithm,
	encapsules the packet and forwards it to the chosen server. All
	real servers are configured with "ifconfig tunl0 <Virtual IP
	Address> up".  When the chosen server receives the encapsuled
	packet, it decapsules the packet, processes the request and
	returns the reply packets  directly to the client without passing
	the load balancer. This can  greatly increase the scalability of
	virtual server.  *  Fix a bug in the ip_portfw_del() for the
	weighted RR scheduling.  The bug in version 0.4 is when the
	weighted round-robin scheduling is used, deleting the last rule
	for a virtual server will report "setsockopt failed: Invalid
	argument" warning, in fact the last rule is deleted but the
	gen_scheduling_seq() works on a null list and causes that warning.
	*  Add and modify some description for virtual server options in
	the Linux kernel configuration help texts.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.4 - November 12, 1998

Differences with virtual server patch version 0.3:

	*  Fix a memory access error bug.  The set_serverpointer_null()
	function is added to scan all the existing ip masquerading records
	for its server pointer which points to the  server specified and
	set it null. It is useful when administrators  delete a real
	server or all real servers, those pointers pointing to  the server
	must be set null.  Otherwise, decreasing the connection  counter
	of the server may cause memory access error when the connection
	terminates or timeout.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.3 - November 10, 1998

Differences with virtual server patch version 0.2:

	*  Change the simple round-robin scheduling to the weighted
	round-robin scheduling. Simple is a special instance of the
	weighted round-robin scheduling when the weights of the servers
	are the same.  *  The scheduling algorithm, originally called the
	weighted round-robin scheduling in version 0.2, actually is the
	weighted least-connection scheduling. So the concept is clarified
	here.  *  Add the least-connection scheduling algorithm. Although
	it is a  special instance of the weighted least-connection
	scheduling algorithm, it is used to avoid dividing the weight in
	looking up servers when the weights of the servers are the same,
	so the overhead of scheduling can be minimized in this case.  *
	Change the type of the server load variables, curr_load and
	least_load, from integer to float in the weighted least-connection
	scheduling.  It can make a better load-balancing when the weights
	specified are high.  *  Merge the original two patches into
	one. Users have to specify which scheduling algorithm is used, the
	weighted round-robin scheduling, the least-connection scheduling,
	or the weighted least-connection scheduling, before rebuild the
	kernel.  *  Change the ip_pfvs_proc function to make the output of
	the port  forwarding & virtual server table more beautiful.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.2 - May 28, 1998

Differences with virtual server patch version 0.1:

	*  Add the weighted round-robin scheduling patch.

	--------------------------------------------------------------------

Virtual Server Patch for Linux - Version 0.1 - May 26, 1998

	*  Implement the infrastructure of virtual server.  *  Implement
	the simple round-robin scheduling algorithm.

	--------------------------------------------------------------------
